From fef191df5e681c3c89166fdf3b0caff8e75399c8 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 01/12] Add the RowFn scalar function framework Progress towards #9128. `RowFn` derives the whole of `ScalarFnVTable` from a row closure: name the element types, write the closure, and lifting supplies null propagation, constant folding, nullability, validity, and options serde. The input side is open. `InputElement::Elem` is a GAT, so an element can hand the closure borrowed variable-length data or drill through a wrapper to an extension array's storage. Covering a new type family is one impl. The output is always an `OutputSink`, allocated once per batch and handing the closure one row to write. `ElementSink` covers one owned `OutputElement` per row, and a custom sink carries runtime-shaped output such as a tensor whose width comes from its input dtype. Work that depends only on a batch-constant operand goes in `RowVisitor::visit_prepared_into`'s once-per-batch prepare step. Prepare must not be load-bearing for validation, because an empty batch decodes every operand as non-constant. Null-strategy selection is derived too. A nullable batch runs densely, by branch-and-skip, or by filtering, and the framework picks per batch. The one input an element controls is `InputElement::FILTERED_DECODE_COST`, set when decoding a column does expensive per-row work, so sparse batches keep the filter strategy's shrunken decode. Two things send a function to `ScalarFnVTable` instead, and no output sink covers either: a result that aliases an input, and a null result for a non-null row. The module docs on `scalar_fn` record the full choice between the two traits. Signed-off-by: Connor Tsui Co-authored-by: Claude --- vortex-array/src/scalar_fn/mod.rs | 65 ++ .../src/scalar_fn/row/element/bool.rs | 73 ++ .../src/scalar_fn/row/element/conformance.rs | 78 ++ vortex-array/src/scalar_fn/row/element/mod.rs | 167 +++++ .../src/scalar_fn/row/element/primitive.rs | 78 ++ .../src/scalar_fn/row/element/tuple.rs | 370 ++++++++++ vortex-array/src/scalar_fn/row/execute.rs | 195 +++++ vortex-array/src/scalar_fn/row/lift.rs | 688 ++++++++++++++++++ vortex-array/src/scalar_fn/row/mod.rs | 74 ++ vortex-array/src/scalar_fn/row/result.rs | 161 ++++ vortex-array/src/scalar_fn/row/row_fn.rs | 138 ++++ vortex-array/src/scalar_fn/row/sink.rs | 133 ++++ .../src/scalar_fn/row/tests/conformance.rs | 76 ++ .../scalar_fn/row/tests/constant_operands.rs | 187 +++++ .../scalar_fn/row/tests/decode_fallibility.rs | 98 +++ .../src/scalar_fn/row/tests/dispatched.rs | 77 ++ .../src/scalar_fn/row/tests/lifting.rs | 230 ++++++ vortex-array/src/scalar_fn/row/tests/mod.rs | 509 +++++++++++++ .../scalar_fn/row/tests/null_strategies.rs | 502 +++++++++++++ .../scalar_fn/row/tests/nullable_outputs.rs | 120 +++ .../src/scalar_fn/row/tests/prepared.rs | 144 ++++ vortex-array/src/scalar_fn/row/tests/sink.rs | 375 ++++++++++ vortex-array/src/scalar_fn/row/vtable.rs | 398 ++++++++++ vortex-array/src/scalar_fn/vtable.rs | 2 +- 24 files changed, 4937 insertions(+), 1 deletion(-) create mode 100644 vortex-array/src/scalar_fn/row/element/bool.rs create mode 100644 vortex-array/src/scalar_fn/row/element/conformance.rs create mode 100644 vortex-array/src/scalar_fn/row/element/mod.rs create mode 100644 vortex-array/src/scalar_fn/row/element/primitive.rs create mode 100644 vortex-array/src/scalar_fn/row/element/tuple.rs create mode 100644 vortex-array/src/scalar_fn/row/execute.rs create mode 100644 vortex-array/src/scalar_fn/row/lift.rs create mode 100644 vortex-array/src/scalar_fn/row/mod.rs create mode 100644 vortex-array/src/scalar_fn/row/result.rs create mode 100644 vortex-array/src/scalar_fn/row/row_fn.rs create mode 100644 vortex-array/src/scalar_fn/row/sink.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/conformance.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/constant_operands.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/dispatched.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/lifting.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/mod.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/null_strategies.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/prepared.rs create mode 100644 vortex-array/src/scalar_fn/row/tests/sink.rs create mode 100644 vortex-array/src/scalar_fn/row/vtable.rs diff --git a/vortex-array/src/scalar_fn/mod.rs b/vortex-array/src/scalar_fn/mod.rs index 003dbc2ca48..11bcefe0325 100644 --- a/vortex-array/src/scalar_fn/mod.rs +++ b/vortex-array/src/scalar_fn/mod.rs @@ -6,6 +6,68 @@ //! This module contains the [`ScalarFnVTable`] trait and all built-in scalar function //! implementations. Expressions ([`crate::expr::Expression`]) reference scalar functions //! at each node. +//! +//! # Choosing a trait +//! +//! Two traits reach this vtable, and [`RowFn`] derives the whole of [`ScalarFnVTable`] from a row +//! closure. Implement `RowFn` when the function fits it, and `ScalarFnVTable` when it does not. +//! +//! [`RowFn`] is for a kernel whose value at a row is determined by that row alone, and which has to +//! read every row anyway: the arithmetic operators over primitive columns, `vortex.tensor.l2_norm`, +//! `vortex.tensor.inner_product`, `vortex.tensor.cosine_similarity`, `vortex.geo.distance`, +//! `vortex.geo.contains`. Name the element types and write the row closure, and the rest is +//! derived, including which rows get visited. +//! +//! Its *input* side is open. [`InputElement::Elem`] is a GAT, so an element can hand the closure +//! borrowed variable-length data (a byte-string element yielding `&[u8]`) or drill through a wrapper +//! (`vortex-tensor`'s `TensorRow` yields a slice of an extension array's storage). Covering a new +//! type family, a list row included, is one impl. +//! +//! Its output is always an [`OutputSink`], allocated once per batch and handing the closure one row +//! to write. [`ElementSink`] is the standard sink for one owned [`OutputElement`] per row. A custom +//! sink carries runtime-shaped output, such as a tensor whose width comes from its input dtype, or a +//! future string transform appending every row into one shared byte buffer. +//! +//! When part of the kernel's work depends only on an operand that is constant for the batch (the +//! norm of a broadcast query vector, a prepared form of a constant geometry), do that work in +//! [`RowVisitor::visit_prepared_into`]'s once-per-batch prepare step. Pass `|_| ()` when there is +//! nothing to prepare. Prepare **must not** be load-bearing for validation: an empty batch decodes +//! every operand as non-constant, so a prepare that validated its constant would silently not run. +//! +//! Null handling is derived too, null-strategy selection included: a nullable batch runs densely +//! (compute every row, mask after), by branch-and-skip (decode full length, compute only the +//! conjoined-valid rows, mask after), or by filtering (shrink the inputs to the valid rows, +//! compute, scatter back), and the framework picks per batch. Function authors do nothing. The one +//! input to that choice an element controls is [`InputElement::FILTERED_DECODE_COST`]: set it when +//! decoding a column does expensive per-row work (parsing a geometry), so sparse batches keep the +//! filter strategy's shrunken decode. Costs from separate arguments are additive. +//! +//! Two things no output sink covers, and they are what actually send a function to +//! [`ScalarFnVTable`]: +//! +//! - **A result that aliases an input.** Sinks own their output bytes. Trimming strings is the +//! example, where the ideal kernel keeps the input's data buffer and writes new views over it, +//! copying no bytes, which only a columnar kernel can express. +//! - **A null result for a non-null row.** Sinks build an all-valid column, so +//! `vortex.list.sum` cannot be a row function: a valid empty list sums to null. +//! +//! [`ScalarFnVTable`] takes the whole column instead, and everything a row function gets derived is +//! then hand-written: null propagation, constant folding, nullability, validity, and options serde. +//! Besides the two cases above and the functions that are simply not strict (Kleene logic, or a +//! strictness that depends on the options), reach for it when a row loop *could* express the +//! function but would do avoidable work: +//! +//! - **The answer is already an array, or is one value for the whole column.** +//! `vortex.list.length` hands back a `ListViewArray`'s sizes child, and a single `ConstantArray` +//! for a `FixedSizeListArray`. A row loop would rebuild that one `u64` at a time, even given a +//! list-length element that reads the size out of the layout rather than the list. +//! - **A row is not the natural unit of work.** `vortex.not` is one `!` per 64-bit word, in place +//! when the bit buffer is unshared, against 64 loop iterations and 64 bit writes, and its +//! encoding-aware fallback pushes the inversion down instead of canonicalizing. +//! - **The row's value is cheaper to read than the row.** `vortex.byte_length` was tried as a row +//! function and measured 7.6x slower than its columnar implementation, because the length is a +//! field of the view and the row loop paid to resolve the bytes it never looked at. Being +//! row-determined is necessary but not sufficient. use vortex_session::registry::Id; @@ -35,6 +97,9 @@ pub use options::*; mod signature; pub use signature::*; +mod row; +pub use row::*; + pub mod fns; pub mod internal; pub mod session; diff --git a/vortex-array/src/scalar_fn/row/element/bool.rs b/vortex-array/src/scalar_fn/row/element/bool.rs new file mode 100644 index 00000000000..d4fc51c769c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/bool.rs @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_buffer::BitBuffer; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::BoolArray; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar_fn::InputElement; +use crate::scalar_fn::OutputElement; +use crate::validity::Validity; + +impl InputElement for bool { + type Column = BitBuffer; + type Varying<'a> = &'a BitBuffer; + type Elem<'a> = bool; + + // Every bit of the buffer is readable, valid or not. + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + matches!(dtype, DType::Bool(_)), + "expected a Bool column, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(array.execute::(ctx)?.into_bit_buffer()) + } + + fn get(column: &Self::Column, index: usize) -> bool { + column.value(index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> bool + where + Self: 'a, + { + column.value(index) + } +} + +impl OutputElement for bool { + fn element_dtype() -> DType { + DType::Bool(Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + // `From>` packs through the multiversioned SIMD path; `from_iter` would set one + // bit at a time, which measures 6.6-7.9x slower on the packing step alone. + BoolArray::new(BitBuffer::from(values), Validity::NonNullable).into_array() + } + + fn placeholder() -> Self { + false + } +} diff --git a/vortex-array/src/scalar_fn/row/element/conformance.rs b/vortex-array/src/scalar_fn/row/element/conformance.rs new file mode 100644 index 00000000000..45687ea9c1a --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/conformance.rs @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! A shared conformance check every [`InputElement`] should be run through. + +use std::hint::black_box; + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::InputElement; + +/// Assert that `E` honors its [`InputElement`] contract over `array`, and rejects `rejected_dtype`. +/// +/// The part worth checking mechanically is [`InputElement::DENSE_SAFE`]. An element claiming it will +/// be read at rows that are *null*, where an array guarantees nothing about the payload, and getting +/// the `const` wrong is either an out-of-bounds panic in production (the failure mode of +/// [#9090](https://github.com/vortex-data/vortex/issues/9090)) or an unnecessary valid-only +/// execution path. Nothing else verifies it, since the framework reads the `const` rather than +/// testing the claim. +/// +/// So `array` **must** contain at least one null row, and its payload behind those nulls **must** be +/// deliberately extreme rather than zeroed, or the check passes vacuously. Build that safely by +/// putting the extreme values in the array first and masking those rows afterwards, as the callers of +/// this function do. +/// +/// What this cannot check: [`DECODE_FALLIBLE`](InputElement::DECODE_FALLIBLE), which needs data that +/// is legal but malformed, and whether `validate` accepts everything it *should*, since only the +/// element knows its full dtype domain. Pass one representative rejection. +#[track_caller] +pub fn assert_element_conforms( + array: ArrayRef, + rejected_dtype: &DType, + ctx: &mut ExecutionCtx, +) -> VortexResult<()> { + let dtype = array.dtype().clone(); + E::validate(&dtype)?; + + assert!( + E::validate(rejected_dtype).is_err(), + "element accepted {rejected_dtype}, which it was expected to reject", + ); + + let len = array.len(); + let valid = array.validity()?.execute_mask(len, ctx)?; + assert!( + !valid.all_true(), + "conformance needs a null row to read behind, but every row of the {dtype} input is valid", + ); + + let column = E::decode(array, ctx)?; + let varying = E::varying(&column); + assert_eq!( + E::varying_len(&varying), + len, + "varying element view changed the decoded row count", + ); + + // The claim under test. Reading a null row may yield garbage, but it must not fault, so an + // element that secretly follows a per-row offset panics here instead of in production. + if E::DENSE_SAFE { + for index in 0..len { + black_box(E::get(&column, index)); + black_box(E::get_varying(&varying, index)); + } + } else { + for index in 0..len { + if valid.value(index) { + black_box(E::get(&column, index)); + black_box(E::get_varying(&varying, index)); + } + } + } + + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/element/mod.rs b/vortex-array/src/scalar_fn/row/element/mod.rs new file mode 100644 index 00000000000..fc88690b6b0 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/mod.rs @@ -0,0 +1,167 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The element types a row function can read and produce. +//! +//! Both traits are open, and this module holds one file per type family, so covering a new one is a +//! sibling file and every row function gains it. The families are not confined to this crate: +//! `vortex-tensor`'s `TensorRow` drills through an extension wrapper into its storage. +//! +//! The two directions are deliberately asymmetric. [`InputElement::Elem`] is a GAT, so an input row +//! can borrow out of the decoded column, while an [`OutputElement`] is one owned value written into +//! an [`ElementSink`](crate::scalar_fn::ElementSink). Runtime-shaped output uses a custom +//! [`OutputSink`](crate::scalar_fn::OutputSink) instead. + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; + +mod bool; + +#[cfg(any(test, feature = "_test-harness"))] +mod conformance; +#[cfg(any(test, feature = "_test-harness"))] +pub use conformance::assert_element_conforms; + +mod primitive; + +mod tuple; +pub use tuple::ElementTuple; +pub(super) use tuple::batch_constant; + +/// An element type that can be read row-wise out of an input column. +pub trait InputElement: 'static { + /// The decoded column representation supporting `O(1)` row access. + type Column; + + /// The view of a varying decoded column read by the hot row loop. + /// + /// This may borrow a cheaper representation than [`Column`](Self::Column). Primitive elements, + /// for example, expose a slice so its pointer and length are loop invariants rather than + /// re-reading a [`Buffer`](vortex_buffer::Buffer) descriptor for every row. + type Varying<'a>; + + /// The borrowed element value handed to the row closure a [`RowFn`](crate::scalar_fn::RowFn) + /// visits with. + type Elem<'a>; + + /// Whether [`decode`](Self::decode) and [`get`](Self::get) tolerate rows that are null in the + /// input. + /// + /// Arrays only guarantee their contents for _valid_ rows, so this is `false` for any element + /// that follows an offset or pointer stored in the array: behind a null row that value is + /// arbitrary and may not address anything. Reading a whole value out of a flat buffer is `true`, + /// since the value is garbage but the read cannot fault. + /// + /// Dense execution requires this of every argument; otherwise the row layer executes only + /// valid rows. + const DENSE_SAFE: bool = false; + + /// Whether [`decode`](Self::decode) can fail on _legal_ input data. + /// + /// `false` for an element read straight out of a buffer: decoding can still fail for + /// infrastructural reasons (IO, allocation), but never because of the values. `true` for an + /// element that parses its bytes, since a malformed WKB geometry in a _valid_ row is a domain + /// error, which makes a function over that element + /// [fallible](crate::scalar_fn::ScalarFnVTable::is_fallible) however infallible its own row + /// computation is. + const DECODE_FALLIBLE: bool = true; + + /// A relative unit count for per-row decode work avoided by filtering this argument first. + /// + /// Use `1` for an element whose decode _parses_ every row (a geometry built from coordinate + /// storage): decoding only the survivors of a sparse validity mask is genuinely cheaper than + /// decoding everyone. Keep the default `0` for a bulk canonicalization (bytes, bools, + /// primitives), whose decode is a memcpy-shaped pass that filtering barely shrinks. Larger + /// values may express a proportionally more expensive decode. + /// + /// The lifting reads this when it picks a null strategy for a batch with a mixed + /// validity mask: filtering the inputs first only pays off when it shrinks a per-row decode, + /// so elements that leave this at zero always take the cheaper branch-and-skip strategy. + /// Getting it wrong is a performance bug, never a correctness bug. + const FILTERED_DECODE_COST: usize = 0; + + /// Validate that `dtype` is an acceptable input column dtype for this element type. + fn validate(dtype: &DType) -> VortexResult<()>; + + /// Decode `array` into its column representation. Called once per batch. + /// + /// This is where every per-batch cost belongs: resolving the dtype, downcasting the buffer, + /// checking the ptype, and anything else that does not vary by row. [`Column`](Self::Column) is + /// the type to widen if that means carrying more, since it is chosen by the element. + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Decode `array` _without_ assuming every row is valid, or `Ok(None)` when this element + /// cannot for this particular array. + /// + /// An element with [`DENSE_SAFE`](Self::DENSE_SAFE) set **should not** override this: its + /// ordinary decode already tolerates null payloads, so the default is already correct and an + /// override just restates it. Overriding is for an element that is *not* dense-safe but can + /// still write an arbitrary placeholder into null slots; the caller guarantees + /// [`get`](Self::get) is never called for such a row. It is what the branch-and-skip null + /// strategy decodes with. + /// + /// Return `Ok(None)` rather than an error when an array has no null-tolerant decode; the lifting + /// falls back to the filter strategy. + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if Self::DENSE_SAFE { + Self::decode(array, ctx).map(Some) + } else { + Ok(None) + } + } + + /// Read the element at `index`, the one function called once per row. + /// + /// `O(1)` is necessary but **not sufficient**: this must not repeat work that is constant across + /// the batch, however cheap that work looks per call. An `O(1)` ptype check and buffer downcast + /// per row cost `l2_norm` 2x at width 2, invisible in the call because it read like a getter. Do + /// that work in [`decode`](Self::decode) and leave this an offset computation. + fn get(column: &Self::Column, index: usize) -> Self::Elem<'_>; + + /// Borrow the representation used when this argument varies within the batch. + /// + /// Called once before the hot loop. Constants do not use this view because the tuple adapter + /// keeps their one-row decoded representation separate. + fn varying(column: &Self::Column) -> Self::Varying<'_>; + + /// Number of rows addressable through a [`Varying`](Self::Varying) view. + fn varying_len(column: &Self::Varying<'_>) -> usize; + + /// Read one row from a [`Varying`](Self::Varying) view. + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> Self::Elem<'a> + where + Self: 'a; +} + +/// An element type that a row computation can produce, buildable into an all-valid column. +/// +/// [`Clone`] is required so [`ElementSink`](crate::scalar_fn::ElementSink) can allocate through +/// `vec![placeholder; rows]`, which is what lets a zero placeholder reach the allocator's zeroed +/// path instead of costing a write pass over the output. +pub trait OutputElement: 'static + Sized + Clone { + /// The dtype of columns built from this element type. Must be non-nullable: nullability is + /// derived from the inputs by the lifting. + /// + /// Taking no arguments confines an element's dtype to a property of its Rust type, so an output + /// whose dtype depends on runtime data (a tensor, whose dtype carries its shape) cannot be an + /// element. Such an output uses an [`OutputSink`](crate::scalar_fn::OutputSink), whose + /// [`sink_dtype`](crate::scalar_fn::OutputSink::sink_dtype) does see the input dtypes. + fn element_dtype() -> DType; + + /// Build a column from one value per row. Called once per batch. + fn build(values: Vec) -> ArrayRef; + + /// An arbitrary value of this element, pre-filled into the output slots that the + /// branch-and-skip null strategy skips. + /// + /// The value is never observable: the lifting masks every slot holding it before the + /// result escapes. It only has to be cheap to construct and legal to + /// [`build`](Self::build) with. + fn placeholder() -> Self; +} diff --git a/vortex-array/src/scalar_fn/row/element/primitive.rs b/vortex-array/src/scalar_fn/row/element/primitive.rs new file mode 100644 index 00000000000..d54c922bf2d --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/primitive.rs @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure_eq; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::PrimitiveArray; +use crate::dtype::DType; +use crate::dtype::NativePType; +use crate::dtype::Nullability; +use crate::scalar_fn::InputElement; +use crate::scalar_fn::OutputElement; +use crate::validity::Validity; + +impl InputElement for T { + type Column = Buffer; + type Varying<'a> = &'a [T]; + type Elem<'a> = T; + + // Every lane of the buffer holds a `T`, valid or not. + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + let expected = T::PTYPE; + let DType::Primitive(ptype, _) = dtype else { + vortex_bail!("expected a {expected} column, got {dtype}"); + }; + vortex_ensure_eq!( + *ptype, + expected, + "expected a {expected} column, got {dtype}" + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(array.execute::(ctx)?.into_buffer::()) + } + + fn get(column: &Self::Column, index: usize) -> T { + column[index] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column.as_slice() + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> T + where + Self: 'a, + { + column[index] + } +} + +impl OutputElement for T { + fn element_dtype() -> DType { + DType::Primitive(T::PTYPE, Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + PrimitiveArray::new(values, Validity::NonNullable).into_array() + } + + fn placeholder() -> Self { + T::default() + } +} diff --git a/vortex-array/src/scalar_fn/row/element/tuple.rs b/vortex-array/src/scalar_fn/row/element/tuple.rs new file mode 100644 index 00000000000..a3c31cfeb2e --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/tuple.rs @@ -0,0 +1,370 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Argument lists built from [`InputElement`]s, and the per-argument decode behind them. + +use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::arrays::Extension; +use crate::arrays::Masked; +use crate::arrays::extension::ExtensionArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; +use crate::dtype::DType; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::InputElement; + +mod private { + pub trait Sealed {} +} + +/// One decoded input column of an [`ElementTuple`]. +/// +/// A constant operand holds the same value in every row, so it is decoded once as a single row and +/// read at index 0 forever. That is what stops a constant argument costing one decode per row, which +/// matters whenever the decode is more than a buffer read: parsing a geometry from WKB, or +/// canonicalizing an extension row. +pub struct ArgColumn(ArgColumnKind); + +enum ArgColumnKind { + Varying(T::Column), + Constant(T::Column), +} + +impl ArgColumn { + /// Decode one input column, collapsing a constant operand to its single distinct row. + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + // An empty input has no row 0 to slice, and its row loop runs zero times either way. + if let Some(constant) = batch_constant(&array) + && !array.is_empty() + { + return Ok(Self(ArgColumnKind::Constant(T::decode( + constant.slice(0..1)?, + ctx, + )?))); + } + + Ok(Self(ArgColumnKind::Varying(T::decode(array, ctx)?))) + } + + /// Like [`decode`](Self::decode), but a varying column decodes null-tolerantly through + /// [`InputElement::decode_null_tolerant`]. `Ok(None)` means the element cannot, and the + /// caller falls back to the filter strategy. + /// + /// A constant operand still takes the ordinary decode: the lifting short-circuits null + /// constants before any strategy runs, so a constant reaching here is non-null. + fn decode_null_tolerant(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult> { + if let Some(constant) = batch_constant(&array) + && !array.is_empty() + { + return Ok(Some(Self(ArgColumnKind::Constant(T::decode( + constant.slice(0..1)?, + ctx, + )?)))); + } + + Ok(T::decode_null_tolerant(array, ctx)? + .map(ArgColumnKind::Varying) + .map(Self)) + } + + /// Read the element at `index`, which for a constant operand is always its single row. + fn get(&self, index: usize) -> T::Elem<'_> { + match &self.0 { + ArgColumnKind::Varying(column) => T::get(column, index), + ArgColumnKind::Constant(column) => T::get(column, 0), + } + } + + /// The decoded full column, or `None` when this argument was collapsed to one constant row. + fn varying(&self) -> Option<&T::Column> { + match &self.0 { + ArgColumnKind::Varying(column) => Some(column), + ArgColumnKind::Constant(_) => None, + } + } + + /// Whether this argument addresses exactly `row_count` rows. + /// + /// A constant operand was collapsed to its one distinct row and is read at index 0 forever, so + /// it addresses any row count and is exempt. + fn addresses_rows(&self, row_count: usize) -> bool { + match &self.0 { + ArgColumnKind::Varying(column) => T::varying_len(&T::varying(column)) == row_count, + ArgColumnKind::Constant(_) => true, + } + } + + /// The single decoded element of a constant operand, or `None` for a real column. + /// + /// `Some` exactly when [`decode`](Self::decode) collapsed the operand to its one distinct row, + /// in which case the value returned is the element every row of the batch reads. + fn constant(&self) -> Option> { + match &self.0 { + ArgColumnKind::Varying(_) => None, + ArgColumnKind::Constant(column) => Some(T::get(column, 0)), + } + } +} + +/// The array whose every row holds one distinct value, when `array` is constant for the batch. +/// +/// Beyond the constant encoding itself this sees one level through two wrappers that spell "the +/// same value in every row" without being the constant encoding: +/// +/// - [`Masked`], how the compressor spells an all-same-with-nulls chunk: the child carries the +/// value, the wrapper carries only validity. Reading the child's value for a null row is sound +/// here because the lifting owns validity entirely; the row loop's output behind a null +/// row is masked away (dense) or never computed (filter), so which value the loop read there +/// cannot be observed. An all-null constant never reaches decode at all, since the lifting +/// short-circuits it to an all-null result first. +/// - [`Extension`] over constant storage, the shape an extension-typed builder produces before +/// `ExtensionConstantRule` normalizes it to a top-level constant. Every row wraps the same +/// storage value, so the whole array (sliced to one row, keeping its extension dtype) is the +/// constant. +pub(in crate::scalar_fn::row) fn batch_constant(array: &ArrayRef) -> Option { + if array.as_constant().is_some() { + return Some(array.clone()); + } + + if let Some(masked) = array.as_opt::() { + return Some(masked.child().clone()).filter(|child| child.as_constant().is_some()); + } + + array + .as_opt::() + .is_some_and(|ext| ext.storage_array().as_constant().is_some()) + .then(|| array.clone()) +} + +/// Tuples of [`InputElement`]s forming the typed argument list a [`RowFn`](crate::scalar_fn::RowFn) +/// visits with. Implemented for `()` and tuples of one through twelve elements. This trait is +/// framework-only; add a new decode primitive by implementing [`InputElement`], then use it inside +/// one of those tuples. +/// +/// The arities past the widest function in tree are deliberate. This trait is **sealed**, so a +/// downstream crate cannot add the one it needs, and an unused arity costs only its own macro +/// expansion: no monomorphization happens until something instantiates it. +pub trait ElementTuple: 'static + private::Sealed { + /// The decoded column representations. + type Columns; + + /// Direct references to decoded columns when every argument varies within the batch. + type VaryingColumns<'a>; + + /// The borrowed row of element values. + type Elems<'a>; + + /// The batch-constant element values: [`Elems`](Self::Elems) with every argument wrapped in + /// `Option`. + /// + /// `Some` marks an argument whose operand is constant for the batch and carries the element + /// every row reads; `None` marks one that varies by row. This is what + /// [`visit_prepared_into`](crate::scalar_fn::RowVisitor::visit_prepared_into) hands to its prepare + /// closure, so a kernel can hoist work that depends only on a constant argument out of the + /// row loop. + type ConstElems<'a>; + + /// The number of arguments. + const ARITY: usize; + + /// Whether every argument is [`InputElement::DENSE_SAFE`]. + const DENSE_SAFE: bool; + + /// Whether _any_ argument is [`InputElement::DECODE_FALLIBLE`]. + const DECODE_FALLIBLE: bool; + + /// The additive cost of per-row decode work avoided by filtering the arguments first. + const FILTERED_DECODE_COST: usize; + + /// Validate the input dtypes, including that `dtypes` has exactly `ARITY` entries. + /// + /// The expression layer checks the count against [`Arity`](crate::scalar_fn::Arity) before it + /// builds a call, but this is also the entry point of the public + /// [`return_dtype`](crate::scalar_fn::ScalarFnVTable::return_dtype), so the count is enforced + /// here rather than assumed. + fn validate(dtypes: &[DType]) -> VortexResult<()>; + + /// Decode every input column once. Called once per batch. + fn decode(args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Decode every input column once, tolerating null rows, or `Ok(None)` when some argument + /// cannot. Called once per batch by the branch-and-skip null strategy. + fn decode_null_tolerant( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult>; + + /// Read the row of elements at `index`. Must be `O(1)`: it is called in the row loop. + fn get(columns: &Self::Columns, index: usize) -> Self::Elems<'_>; + + /// Borrow every decoded column directly, or `None` when any argument is batch-constant. + /// + /// This is selected once outside the hot loop. Keeping `ArgColumn` out of the resulting tuple + /// gives the optimizer ordinary contiguous column access without a per-row constant check. + fn varying(columns: &Self::Columns) -> Option>; + + /// Whether every varying column contains exactly `row_count` rows. + fn varying_len_matches(columns: &Self::VaryingColumns<'_>, row_count: usize) -> bool; + + /// Whether every argument that varies within the batch contains exactly `row_count` rows. + /// + /// The same guarantee as [`varying_len_matches`](Self::varying_len_matches), for the mixed case + /// [`varying`](Self::varying) declines: a batch-constant argument is exempt because it was + /// collapsed to one row, while every argument beside it still has to address the whole batch. + fn decoded_lens_match(columns: &Self::Columns, row_count: usize) -> bool; + + /// Read one row from columns already known to vary within the batch. + fn get_varying<'a>(columns: &Self::VaryingColumns<'a>, index: usize) -> Self::Elems<'a>; + + /// Read the batch-constant elements out of the decoded columns. Called once per batch. + fn constants(columns: &Self::Columns) -> Self::ConstElems<'_>; +} + +impl private::Sealed for () {} + +impl ElementTuple for () { + type Columns = (); + type VaryingColumns<'a> = (); + type Elems<'a> = (); + type ConstElems<'a> = (); + + const ARITY: usize = 0; + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + const FILTERED_DECODE_COST: usize = 0; + + fn validate(dtypes: &[DType]) -> VortexResult<()> { + vortex_ensure_eq!( + dtypes.len(), + 0, + "expected 0 argument dtypes, got {}", + dtypes.len(), + ); + Ok(()) + } + + fn decode(_args: &dyn ExecutionArgs, _ctx: &mut ExecutionCtx) -> VortexResult { + Ok(()) + } + + fn decode_null_tolerant( + _args: &dyn ExecutionArgs, + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(())) + } + + fn get(_columns: &Self::Columns, _index: usize) -> Self::Elems<'_> {} + + fn varying(_columns: &Self::Columns) -> Option> { + Some(()) + } + + fn varying_len_matches(_columns: &Self::VaryingColumns<'_>, _row_count: usize) -> bool { + true + } + + fn decoded_lens_match(_columns: &Self::Columns, _row_count: usize) -> bool { + true + } + + fn get_varying<'a>(_columns: &Self::VaryingColumns<'a>, _index: usize) -> Self::Elems<'a> {} + + fn constants(_columns: &Self::Columns) -> Self::ConstElems<'_> {} +} + +macro_rules! element_tuple { + ($arity:literal; $($t:ident : $idx:tt),+) => { + impl<$($t: InputElement),+> private::Sealed for ($($t,)+) {} + + impl<$($t: InputElement),+> ElementTuple for ($($t,)+) { + type Columns = ($(ArgColumn<$t>,)+); + type VaryingColumns<'a> = ($($t::Varying<'a>,)+); + type Elems<'a> = ($($t::Elem<'a>,)+); + type ConstElems<'a> = ($(Option<$t::Elem<'a>>,)+); + + const ARITY: usize = $arity; + const DENSE_SAFE: bool = $($t::DENSE_SAFE &&)+ true; + const DECODE_FALLIBLE: bool = $($t::DECODE_FALLIBLE ||)+ false; + const FILTERED_DECODE_COST: usize = $($t::FILTERED_DECODE_COST +)+ 0; + + fn validate(dtypes: &[DType]) -> VortexResult<()> { + vortex_ensure_eq!( + dtypes.len(), + $arity, + "expected {} argument dtypes, got {}", + $arity, + dtypes.len(), + ); + + $($t::validate(&dtypes[$idx])?;)+ + Ok(()) + } + + fn decode( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + Ok(($(ArgColumn::<$t>::decode(args.get($idx)?, ctx)?,)+)) + } + + fn decode_null_tolerant( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(( + $(match ArgColumn::<$t>::decode_null_tolerant(args.get($idx)?, ctx)? { + Some(column) => column, + None => return Ok(None), + },)+ + ))) + } + + fn get(columns: &Self::Columns, index: usize) -> Self::Elems<'_> { + ($(columns.$idx.get(index),)+) + } + + fn varying(columns: &Self::Columns) -> Option> { + Some(($($t::varying(columns.$idx.varying()?),)+)) + } + + fn varying_len_matches( + columns: &Self::VaryingColumns<'_>, + row_count: usize, + ) -> bool { + $($t::varying_len(&columns.$idx) == row_count &&)+ true + } + + fn decoded_lens_match(columns: &Self::Columns, row_count: usize) -> bool { + $(columns.$idx.addresses_rows(row_count) &&)+ true + } + + fn get_varying<'a>( + columns: &Self::VaryingColumns<'a>, + index: usize, + ) -> Self::Elems<'a> { + ($($t::get_varying(&columns.$idx, index),)+) + } + + fn constants(columns: &Self::Columns) -> Self::ConstElems<'_> { + ($(columns.$idx.constant(),)+) + } + } + }; +} + +element_tuple!(1; A:0); +element_tuple!(2; A:0, B:1); +element_tuple!(3; A:0, B:1, C:2); +element_tuple!(4; A:0, B:1, C:2, D:3); +element_tuple!(5; A:0, B:1, C:2, D:3, E:4); +element_tuple!(6; A:0, B:1, C:2, D:3, E:4, F:5); +element_tuple!(7; A:0, B:1, C:2, D:3, E:4, F:5, G:6); +element_tuple!(8; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7); +element_tuple!(9; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8); +element_tuple!(10; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9); +element_tuple!(11; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10); +element_tuple!(12; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10, L:11); diff --git a/vortex-array/src/scalar_fn/row/execute.rs b/vortex-array/src/scalar_fn/row/execute.rs new file mode 100644 index 00000000000..345bed991fe --- /dev/null +++ b/vortex-array/src/scalar_fn/row/execute.rs @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The pieces every row function is built from, whatever its `dispatch` chooses. +//! +//! These back the blanket impls in [`row_fn`](super::row_fn) and are deliberately not public: +//! [`RowFn`](crate::scalar_fn::RowFn) is the abstraction, these are its internals. + +use vortex_error::VortexError; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_mask::AllOr; +use vortex_mask::Mask; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::DeferredError; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::SinkResult; + +/// The value path out of a row executor, keeping a deferred row error distinct from structural +/// execution errors so nullable lifting retries only the former. +pub(super) enum RowExecution { + /// A successfully built output column. + Output(ArrayRef), + + /// A batch-wide row error that nullable lifting may retry over only the valid rows. + DeferredError(VortexError), +} + +impl RowExecution { + /// Return the output or surface its deferred row error. + pub(super) fn into_result(self) -> VortexResult { + match self { + Self::Output(output) => Ok(output), + Self::DeferredError(error) => Err(error), + } + } +} + +/// Validate the input dtypes of a sink-writing row function and return the dtype its sink builds. +/// +/// The output dtype may be a function of the inputs. A sink can also own a batch-wide builder, such +/// as the shared byte and view buffers of a future string transform. +pub(super) fn validate_row_sink( + args: &[DType], +) -> VortexResult { + A::validate(args)?; + let dtype = S::sink_dtype(args)?; + vortex_ensure!( + !dtype.is_nullable(), + "row output sinks must declare a non-nullable dtype, got {dtype}", + ); + Ok(dtype) +} + +/// Decode every input column once, allocate the sink once, then write one row at a time. +/// +/// The sink lives here rather than in the closure, so `apply` stays [`Fn`] and the loop keeps the +/// unconditional shape that lets it vectorize. Monomorphic in `A`, `S` and `R`, so `apply` and +/// [`OutputSink::row`] both inline. +pub(super) fn execute_row_sink_prepared( + args: &dyn ExecutionArgs, + sink_dtype: &DType, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult { + let row_count = args.row_count(); + let mut sink = S::with_capacity(row_count, sink_dtype)?; + let columns = A::decode(args, ctx)?; + let state = prepare(A::constants(&columns)); + let mut accumulated = R::Accumulated::default(); + + { + let mut rows = sink.rows(); + vortex_ensure!( + S::row_count_matches(&rows, row_count), + "the output sink does not address exactly {row_count} rows", + ); + + if let Some(varying) = A::varying(&columns) { + vortex_ensure!( + A::varying_len_matches(&varying, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + for index in 0..row_count { + apply( + &state, + A::get_varying(&varying, index), + S::row(&mut rows, index), + ) + .accumulate(&mut accumulated)?; + } + } else { + vortex_ensure!( + A::decoded_lens_match(&columns, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + for index in 0..row_count { + apply(&state, A::get(&columns, index), S::row(&mut rows, index)) + .accumulate(&mut accumulated)?; + } + } + } + + finish_sink(sink, DeferredError::new(R::occurred(accumulated))) +} + +/// Run a prepared sink over only the rows set in `valid`, or decline when the sink cannot skip. +pub(super) fn execute_row_sink_branch( + args: &dyn ExecutionArgs, + sink_dtype: &DType, + valid: &Mask, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult> { + if !S::SUPPORTS_SKIPPED_ROWS { + return Ok(None); + } + + let Some(columns) = A::decode_null_tolerant(args, ctx)? else { + return Ok(None); + }; + let state = prepare(A::constants(&columns)); + let row_count = args.row_count(); + let mut sink = S::with_capacity(row_count, sink_dtype)?; + let mut accumulated = R::Accumulated::default(); + + let AllOr::Some(valid) = valid.bit_buffer() else { + vortex_bail!("execute_row_sink_branch requires a mixed mask"); + }; + + { + let mut rows = sink.rows(); + vortex_ensure!( + S::row_count_matches(&rows, row_count), + "the output sink does not address exactly {row_count} rows", + ); + + let varying = A::varying(&columns); + let lens_match = match &varying { + Some(varying) => A::varying_len_matches(varying, row_count), + None => A::decoded_lens_match(&columns, row_count), + }; + vortex_ensure!( + lens_match, + "a decoded row input does not address exactly {row_count} rows", + ); + + let mut error = None; + valid.for_each_set_index(|index| { + if error.is_some() { + return; + } + + let result = match &varying { + Some(varying) => apply( + &state, + A::get_varying(varying, index), + S::row(&mut rows, index), + ), + None => apply(&state, A::get(&columns, index), S::row(&mut rows, index)), + }; + if let Err(err) = result.accumulate(&mut accumulated) { + error = Some(err); + } + }); + + if let Some(error) = error { + return Err(error); + } + } + + finish_sink(sink, DeferredError::new(R::occurred(accumulated))).map(Some) +} + +/// Finish a sink while preserving whether its error came from the deferred row accumulator. +fn finish_sink( + sink: S, + deferred_error: DeferredError, +) -> VortexResult { + match sink.finish(deferred_error) { + Ok(output) => Ok(RowExecution::Output(output)), + Err(error) if deferred_error.occurred() => Ok(RowExecution::DeferredError(error)), + Err(error) => Err(error), + } +} diff --git a/vortex-array/src/scalar_fn/row/lift.rs b/vortex-array/src/scalar_fn/row/lift.rs new file mode 100644 index 00000000000..a842ef5deef --- /dev/null +++ b/vortex-array/src/scalar_fn/row/lift.rs @@ -0,0 +1,688 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Lifting a kernel over non-null values into a full [`ScalarFnVTable::execute`]. +//! +//! A [`RowFn`] hands the framework a kernel that only ever computes rows valid in every argument. +//! Everything between that kernel and [`ScalarFnVTable::execute`] lives here: null propagation, +//! constant folding, nullability widening, output dtype reconciliation, and the per-batch choice +//! between dense execution and the two mechanisms that execute only valid rows. +//! +//! This is machinery, not an interface. It takes the kernel as a pair of closures rather than a +//! trait because the one trait that ever occupied the slot (a public `StrictScalarFnVTable`, with +//! [`RowFn`] blanket-implementing it) never found a second implementor, and the indirection cost +//! more than it explained. Extract a trait if and when a non-row user appears. +//! +//! [`RowFn`]: crate::scalar_fn::RowFn +//! [`ScalarFnVTable::execute`]: crate::scalar_fn::ScalarFnVTable::execute + +use smallvec::SmallVec; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_error::vortex_ensure_eq; +use vortex_mask::AllOr; +use vortex_mask::Mask; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::BoolArray; +use crate::arrays::ConstantArray; +use crate::arrays::MaskedArray; +use crate::arrays::PrimitiveArray; +use crate::builtins::ArrayBuiltins; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar::Scalar; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::SinkResult; +use crate::scalar_fn::row::element::batch_constant; +use crate::scalar_fn::row::execute::RowExecution; +use crate::validity::Validity; + +struct BorrowedExecutionArgs<'a> { + inputs: &'a [ArrayRef], + row_count: usize, +} + +impl<'a> BorrowedExecutionArgs<'a> { + fn new(inputs: &'a [ArrayRef], row_count: usize) -> Self { + Self { inputs, row_count } + } +} + +impl ExecutionArgs for BorrowedExecutionArgs<'_> { + fn get(&self, index: usize) -> VortexResult { + self.inputs.get(index).cloned().ok_or_else(|| { + vortex_error::vortex_err!( + "Input index {} out of bounds (num_inputs={})", + index, + self.inputs.len() + ) + }) + } + + fn num_inputs(&self) -> usize { + self.inputs.len() + } + + fn row_count(&self) -> usize { + self.row_count + } +} + +/// The arguments handed to one kernel invocation. +/// +/// `arrays` may be filtered or sliced, while `dtypes` and `sink_dtype` always describe the original +/// planned batch. Keeping them together prevents an execution path from accidentally pairing an +/// input view with unrelated planning metadata. +#[derive(Clone, Copy)] +pub(super) struct KernelArgs<'a> { + /// The executor-facing view, including the row count for this invocation. + pub(super) execution: &'a dyn ExecutionArgs, + + /// The same inputs as concrete arrays for encoding-aware rewrites. + pub(super) arrays: &'a [ArrayRef], + + /// The original input dtypes used to select the row implementation. + pub(super) dtypes: &'a [DType], + + /// The non-nullable dtype allocated by the selected output sink. + pub(super) sink_dtype: &'a DType, +} + +/// The execution policy and output dtype selected by a planning visit. +pub(super) struct BatchPlan { + /// The non-nullable dtype built by the selected sink. + pub(super) sink_dtype: DType, + + /// How this concrete dispatch executes nullable rows. + pub(super) policy: RowPolicy, +} + +/// The nullable execution policy derived from one concrete dispatch. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum RowPolicy { + /// Evaluate all rows and mask the result. + Dense, + + /// Evaluate all rows, retrying only valid rows if a deferred error is raised. + DenseWithRetry, + + /// Execute only valid rows, choosing branch-and-skip or filtering from the mask and decode + /// cost. + ValidOnly { filtered_decode_cost: usize }, +} + +impl RowPolicy { + /// The policy one concrete dispatch executes nullable rows under. + /// + /// Note what is deliberately **not** read here: [`OutputSink::SUPPORTS_SKIPPED_ROWS`]. Hoisting + /// it into the plan so that a non-skipping sink never enters the branch path looks like a free + /// win, and #9130 records it as one, but it is not: the branch path probes + /// [`reduce_encoded`](crate::scalar_fn::RowFn::reduce_encoded) against the _original_ arrays + /// before it ever consults the sink, and that is the only probe that sees them still encoded. + /// Skipping the path early would leave such a function with only the filtered probe, whose + /// canonical arrays match no encoding fast path. For a function whose reduction is defined to + /// answer differently from its row loop, that is a wrong answer rather than a slow one. + /// + /// [`OutputSink::SUPPORTS_SKIPPED_ROWS`]: crate::scalar_fn::OutputSink::SUPPORTS_SKIPPED_ROWS + pub(super) const fn for_dispatch() -> Self { + if A::DENSE_SAFE && !A::DECODE_FALLIBLE && !R::FALLIBLE { + if R::DEFERRED { + Self::DenseWithRetry + } else { + Self::Dense + } + } else { + Self::ValidOnly { + filtered_decode_cost: A::FILTERED_DECODE_COST, + } + } + } +} + +/// How far [`Batch::resolve_validity`] got before a mixed-mask strategy became necessary. +enum ResolvedMask { + /// The batch was answered without one: every row valid, or every row null. + Decided(ArrayRef), + + /// A mask with both set and unset bits, which a strategy must now execute. + Mixed(Mask), +} + +/// One batch of inputs, with everything the lifting reads off them before the kernel runs. +pub(super) struct Batch<'a> { + /// The function being executed, named in the errors this raises. + id: ScalarFnId, + + /// The arguments as the execution layer handed them over. Every path but the filter strategy + /// gives the kernel these untouched, so it sees the original encodings. + args: &'a dyn ExecutionArgs, + + /// The input columns, collected once: constant folding inspects them and the filter strategy + /// filters them. + inputs: SmallVec<[ArrayRef; 4]>, + + /// The input dtypes, collected with the columns and reused by both planning and execution. + arg_dtypes: SmallVec<[DType; 4]>, + + /// The conjoined input validity, so a row of the output is valid iff it is valid in every + /// input. Conjoining is lazy, and nothing materializes it unless the null handling asks. + validity: Validity, + + /// The dtype the function declares for these inputs, which the kernel's output is reconciled + /// against. Already widened to nullable if any input is nullable. + result_dtype: DType, + + /// The non-nullable dtype the dispatched sink builds, computed once while planning. + sink_dtype: DType, + + /// How the concrete dispatch executes nullable rows. + policy: RowPolicy, +} + +impl<'a> Batch<'a> { + /// Collect `args` and read the lifting's facts off them, `return_dtype` being the function's + /// declared return dtype for the input dtypes it is handed. + /// + /// **Not** for a nullary function: with no inputs there is no validity to propagate and no + /// per-row work to fold, and the all-constant check below would vacuously pass. + pub(super) fn new( + id: ScalarFnId, + args: &'a dyn ExecutionArgs, + plan: impl FnOnce(&[DType]) -> VortexResult, + ) -> VortexResult { + let inputs: SmallVec<[ArrayRef; 4]> = (0..args.num_inputs()) + .map(|i| args.get(i)) + .collect::>()?; + + let arg_dtypes: SmallVec<[DType; 4]> = + inputs.iter().map(|input| input.dtype().clone()).collect(); + let plan = plan(&arg_dtypes)?; + let nullability = plan.sink_dtype.nullability() + | Nullability::from(arg_dtypes.iter().any(DType::is_nullable)); + let result_dtype = plan.sink_dtype.with_nullability(nullability); + + let mut validity = Validity::NonNullable; + for input in &inputs { + validity = validity.and(input.validity()?)?; + } + + Ok(Self { + id, + args, + inputs, + arg_dtypes, + validity, + result_dtype, + sink_dtype: plan.sink_dtype, + policy: plan.policy, + }) + } + + /// Run `kernel` over this batch, adding everything the kernel does not do: the null-constant + /// short circuit, the all-constant fold, and the null handling. + /// + /// `kernel` computes the whole column from the arguments it is handed. Those are this batch's + /// arguments untouched, except under the filter strategy, where they are filtered copies, and + /// in the all-constant fold, where they are one row each. What it may assume: + /// + /// - No input is a null constant, and the inputs are not all constant. + /// - Under valid-only execution, every row of every input is valid. + /// - Under dense execution, rows behind nulls hold arbitrary values, and their results are + /// discarded. + /// + /// Either way the kernel can ignore input validity, and its output **must** equal + /// `return_dtype` up to nullability. A kernel that returns nulls of its own keeps them, unioned + /// with the ones the lifting applies, which requires its declared dtype to be nullable. + /// + /// `branch` computes only the rows set in the conjoined mask, over the _unfiltered_ arguments, + /// writing an arbitrary placeholder everywhere else; `Ok(None)` means it cannot for these + /// inputs, which sends the batch to the filter strategy. It is only ever called with a mixed + /// mask, and it **must not** run its row computation (nor any per-row fallible decode) on an + /// unset row, since those rows hold arbitrary values and a fallible kernel would spuriously + /// fail on them. + pub(super) fn execute( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Strictness: any null-constant input forces an all-null result without evaluating the + // kernel. + if self + .inputs + .iter() + .any(|input| input.as_constant().is_some_and(|scalar| scalar.is_null())) + { + return Ok(self.all_null()); + } + + // All inputs constant, and their conjoined validity proves every row non-null. This sees + // through extension and masked wrappers just like argument decoding does. + if self.args.row_count() > 0 + && self.validity.definitely_no_nulls() + && self + .inputs + .iter() + .all(|input| batch_constant(input).is_some()) + { + return self.broadcast_one_row(kernel, ctx); + } + + match self.policy { + RowPolicy::Dense => self.execute_dense(kernel, false, ctx), + RowPolicy::DenseWithRetry => self.execute_dense(kernel, true, ctx), + RowPolicy::ValidOnly { + filtered_decode_cost, + } => self.execute_filtered(kernel, branch, filtered_decode_cost, ctx), + } + } + + /// Evaluate a single row of all-constant inputs and broadcast its value. + /// + /// Reconciling the row's dtype before reading the scalar keeps this path on the same + /// kernel/declaration agreement check as the dense and filter paths, rather than letting `cast` + /// paper over a disagreement. + fn broadcast_one_row( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let one_row: SmallVec<[ArrayRef; 4]> = self + .inputs + .iter() + .map(|input| input.slice(0..1)) + .collect::>()?; + + let args = BorrowedExecutionArgs::new(&one_row, 1); + let result = kernel(self.kernel_args(&args, &one_row), ctx)?.into_result()?; + let scalar = self.with_return_dtype(result, 1)?.execute_scalar(0, ctx)?; + + Ok(ConstantArray::new(scalar, self.args.row_count()).into_array()) + } + + /// Run the kernel over every row, including the rows behind nulls, then mask its result. + /// + /// The arguments reach the kernel untouched, so the inputs keep their original encoding, and + /// the conjoined validity is handed to `mask` as an array rather than materialized into a + /// [`Mask`] first. + fn execute_dense( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + retry_deferred_error: bool, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Every row is null, so the kernel has nothing to contribute. + if matches!(self.validity, Validity::AllInvalid) { + return Ok(self.all_null()); + } + + let values = match kernel(self.kernel_args(self.args, &self.inputs), ctx)? { + RowExecution::Output(values) => values, + RowExecution::DeferredError(error) if retry_deferred_error => { + let valid = self + .validity + .clone() + .execute_mask(self.args.row_count(), ctx)?; + + // The same shortcut pair as `resolve_validity`, with different outcomes: every + // row valid means some valid row genuinely failed, and no row valid means every + // failure was behind a null. An empty mask is both all-true and all-false, but + // cannot reach this arm: a zero-row loop accumulates no evidence, so a zero-row + // batch never reports a deferred error. + if valid.all_true() { + return Err(error); + } + if valid.all_false() { + return Ok(self.all_null()); + } + + // Filtering unconditionally, rather than consulting `branch_beats_filter`. Not + // because branch-and-skip is unavailable in principle: `ERRORS_ARE_DEFERRED` and + // `SUPPORTS_SKIPPED_ROWS` are independent, and a sink may legally set both. It is + // that `execute_dense` is not handed the `branch` closure at all, so filtering is + // the only strategy reachable from here. This is the cold path, taken only after a + // batch has already reported an error, so the choice has not been worth plumbing + // for. + return self.filter_and_scatter(kernel, &valid, ctx); + } + RowExecution::DeferredError(error) => return Err(error), + }; + + match self.validity.clone() { + Validity::NonNullable | Validity::AllValid => { + self.with_return_dtype(values, self.args.row_count()) + } + Validity::Array(valid) => { + self.with_return_dtype(values.mask(valid)?, self.args.row_count()) + } + // Handled by the guard above, before the kernel ran. + Validity::AllInvalid => Ok(self.all_null()), + } + } + + /// Materialize the conjoined validity and resolve everything that does not need a mixed-mask + /// strategy, so that the production selector and the forced-strategy test seam cannot drift + /// apart on the shortcuts they share. The deferred-error retry in + /// [`execute_dense`](Self::execute_dense) repeats the same materialize-then-shortcut shape + /// with different outcomes — all-true is an error there, all-false is all-null — so it stays + /// open-coded, with its own note on why the ordering is safe. + fn resolve_validity( + &self, + kernel: &impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let valid = self + .validity + .clone() + .execute_mask(self.args.row_count(), ctx)?; + + // Check all-true before all-false: an empty mask is both, and must not be treated as + // all-null (a zero-length non-nullable execution keeps its non-nullable dtype). + if valid.all_true() { + return self + .with_return_dtype( + kernel(self.kernel_args(self.args, &self.inputs), ctx)?.into_result()?, + self.args.row_count(), + ) + .map(ResolvedMask::Decided); + } + + if valid.all_false() { + return Ok(ResolvedMask::Decided(self.all_null())); + } + + Ok(ResolvedMask::Mixed(valid)) + } + + /// Materialize the conjoined validity once, take the all-true and all-false shortcuts, and + /// pick a strategy per batch for a mixed mask. + /// + /// Two strategies can execute a mixed mask, and neither is visible to the kernel: + /// + /// - **Branch-and-skip** ([`execute_branched`](Self::execute_branched)): hand the _unfiltered_ + /// arguments plus the mask to `branch`, which computes only the valid rows, then mask the + /// full-length result exactly as the dense path does. This skips the filter and the scatter + /// entirely, at the price of decoding full-length columns. + /// - **Filter** ([`filter_and_scatter`](Self::filter_and_scatter)): filter every input down to + /// the conjoined-valid rows, run the kernel over those, and scatter its results back into a + /// null-padded output. Always available, never encoding-preserving. + /// + /// Branch-and-skip is preferred whenever [`branch_beats_filter`] says so, and the filter + /// strategy is also the fallback for a kernel with no branch execution. + fn execute_filtered( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + filtered_decode_cost: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let valid = match self.resolve_validity(&kernel, ctx)? { + ResolvedMask::Decided(result) => return Ok(result), + ResolvedMask::Mixed(valid) => valid, + }; + + if branch_beats_filter(filtered_decode_cost, &valid) + && let Some(result) = self.execute_branched(branch, &valid, ctx)? + { + return Ok(result); + } + + self.filter_and_scatter(kernel, &valid, ctx) + } + + /// Try the branch-and-skip strategy for a mixed mask: the kernel computes only the rows set in + /// `valid` over the unfiltered inputs, and the full-length result is masked exactly as the + /// dense path masks. `Ok(None)` means the kernel has no branch execution for these inputs, and + /// the caller falls back to [`filter_and_scatter`](Self::filter_and_scatter). + fn execute_branched( + &self, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + valid: &Mask, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let Some(values) = branch(self.kernel_args(self.args, &self.inputs), valid, ctx)? else { + return Ok(None); + }; + let values = values.into_result()?; + + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + self.with_return_dtype(values.mask(mask)?, valid.len()) + .map(Some) + } + + /// The filter strategy for a mixed mask: filter every input down to the rows set in `valid`, + /// run the kernel over those, and scatter its results back into a null-padded output. + fn filter_and_scatter( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + valid: &Mask, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let filtered: SmallVec<[ArrayRef; 4]> = self + .inputs + .iter() + .map(|input| input.filter(valid.clone())) + .collect::>()?; + + let args = BorrowedExecutionArgs::new(&filtered, valid.true_count()); + let values = kernel(self.kernel_args(&args, &filtered), ctx)?.into_result()?; + + self.with_return_dtype(self.scatter_valid(values, valid)?, valid.len()) + } + + /// An all-null result of the function's declared return dtype. + fn all_null(&self) -> ArrayRef { + ConstantArray::new( + Scalar::null(self.result_dtype.clone()), + self.args.row_count(), + ) + .into_array() + } + + /// Pair an input view with this batch's planning metadata. + fn kernel_args<'b>( + &'b self, + execution: &'b dyn ExecutionArgs, + arrays: &'b [ArrayRef], + ) -> KernelArgs<'b> { + KernelArgs { + execution, + arrays, + dtypes: &self.arg_dtypes, + sink_dtype: &self.sink_dtype, + } + } + + /// Reconcile the kernel's output dtype with the function's declared return dtype. + /// + /// The kernel may ignore nullability, so a nullability difference is cast away. Any other + /// difference means the declared dtype and the kernel disagree, which is a bug worth naming + /// rather than silently casting away. + fn with_return_dtype(&self, values: ArrayRef, expected_len: usize) -> VortexResult { + reconcile_return(self.id, &self.result_dtype, expected_len, values) + } + + /// Scatter `values` (one per set bit of `valid`, in order) back to the positions of the set + /// bits, producing an array of length `valid.len()` that is null at every unset position. + fn scatter_valid(&self, values: ArrayRef, valid: &Mask) -> VortexResult { + vortex_ensure_eq!( + values.len(), + valid.true_count(), + "the {} kernel produced {} rows for {} filtered rows", + self.id, + values.len(), + valid.true_count(), + ); + + let AllOr::Some(slices) = valid.slices() else { + // The caller handles the all-true and all-false masks. + vortex_bail!("scatter_valid requires a mixed mask"); + }; + + // Gather indices: row i of the output reads values[rank(i)]. Rows behind nulls read index + // 0, and any in-bounds index would do since they are masked out below (values is non-empty + // here). + let mut indices = vec![0u64; valid.len()]; + let mut rank = 0u64; + for &(start, end) in slices { + for index in &mut indices[start..end] { + *index = rank; + rank += 1; + } + } + let indices = PrimitiveArray::new(indices, Validity::NonNullable).into_array(); + + let scattered = values.take(indices)?; + + // A kernel that produced nulls of its own (only `reduce_encoded` may) cannot be wrapped, + // since a `Masked` child must be all valid. Those nulls have to be unioned with the + // lifting's, which is what the general masking pass does. + if scattered.dtype().is_nullable() { + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + return scattered.mask(mask); + } + + // Attaching the mask as validity rather than masking again: the gathered values are + // already all valid, so recording which rows survive is the whole job and a `Masked` + // wrapper says exactly that. Worth 1.13-1.53x here, growing with null density + // (`null_strategy_bytes`, 65536 rows, divan fastest and median of 100 samples, best of two + // runs, Apple M4 Max). The same substitution on the dense path measured no difference, so + // it is deliberately confined to the scatter. + Ok(MaskedArray::try_new( + scattered, + Validity::from_mask(valid.clone(), Nullability::Nullable), + )? + .into_array()) + } +} + +/// Validate the row count and reconcile nullability against a row function's declared dtype. +pub(super) fn reconcile_return( + id: ScalarFnId, + result_dtype: &DType, + expected_len: usize, + values: ArrayRef, +) -> VortexResult { + vortex_ensure_eq!( + values.len(), + expected_len, + "the {id} kernel produced {} rows for {expected_len} input rows", + values.len(), + ); + vortex_ensure!( + values.dtype().eq_ignore_nullability(result_dtype), + "the {id} kernel produced {} but the function declares {result_dtype}", + values.dtype(), + ); + + if values.dtype() == result_dtype { + Ok(values) + } else { + values.cast(result_dtype.clone()) + } +} + +/// The minimum surviving-row fraction (`true_count / len` of the conjoined mask) at which +/// branch-and-skip is still chosen for one filtered decode unit. +/// +/// From the branch-and-skip measurements (65536 rows, divan fastest of 100 samples, two runs on a +/// shared 4-vCPU VM). A kernel with a _bulk_ decode never lost under branch: `byte_length` over +/// a byte-string element ran 1.8-5.9x faster than filter at every null density from 1% to 90%, so +/// such kernels skip this check entirely. A kernel with a _per-row_ decode (geo `contains`, which +/// arrow-exports and parses one geometry per row) pays that decode over the full column under +/// branch but only over the survivors under filter, so filter wins once validity is sparse: +/// +/// - polygons CONTAINS constant point: branch won 1.07-1.18x at 1-50% nulls; filter won 1.38x at +/// 90% nulls (10% of rows surviving). +/// - polygons CONTAINS points, independent nulls on both: branch won up to ~10% null density +/// (~81% surviving); filter won 1.2x at ~56% surviving, 1.9x at ~25%, 11.3x at ~1%. +/// +/// A single nullable operand still favored branch at 50% surviving, while two independent nullable +/// operands favored filtering at 81% surviving. Keep those cases distinct instead of collapsing +/// every per-row decode into one boolean. There is not yet enough evidence to distinguish two from +/// three or more decode units, so they share the conservative multi-decode threshold. +pub(super) const ONE_DECODE_BRANCH_MIN_SURVIVING_FRACTION: f64 = 0.50; +pub(super) const MULTI_DECODE_BRANCH_MIN_SURVIVING_FRACTION: f64 = 0.85; + +/// Whether the branch-and-skip strategy should be preferred over filtering for the mixed mask +/// `valid`. A zero cost always branches; otherwise the survivor threshold grows when filtering +/// avoids more than one unit of per-row decode work. +pub(super) fn branch_beats_filter(filtered_decode_cost: usize, valid: &Mask) -> bool { + if filtered_decode_cost == 0 { + return true; + } + + let minimum = if filtered_decode_cost == 1 { + ONE_DECODE_BRANCH_MIN_SURVIVING_FRACTION + } else { + MULTI_DECODE_BRANCH_MIN_SURVIVING_FRACTION + }; + valid.true_count() as f64 >= valid.len() as f64 * minimum +} + +/// Which null strategy a forced execution takes for a mixed validity mask. +/// +/// A test and benchmark seam: pinning a strategy is how the two are compared and how their +/// agreement is asserted. Production execution selects per batch inside the lifting and never +/// names one. See [`execute_row_fn_with_strategy`](super::execute_row_fn_with_strategy). +#[cfg(any(test, feature = "_test-harness"))] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum NullStrategy { + /// Filter the inputs down to the conjoined-valid rows, run the kernel, and scatter back. + Filter, + + /// Decode the unfiltered inputs null-tolerantly, compute only the conjoined-valid rows, and + /// mask the full-length result. + BranchAndSkip, +} + +#[cfg(any(test, feature = "_test-harness"))] +impl Batch<'_> { + /// Execute this batch with a forced null strategy, bypassing the per-batch selection. + /// + /// A test and benchmark seam only. It mirrors [`execute_filtered`](Self::execute_filtered) + /// (conjoined validity, the all-true and all-false shortcuts, output dtype reconciliation) but + /// takes the strategy from the caller instead of the selection rule, and it skips the + /// null-constant and all-constant folds, so do not pass such inputs. `Ok(None)` means + /// [`NullStrategy::BranchAndSkip`] was forced on a kernel with no branch execution, which the + /// caller reports rather than silently falling back. + pub(super) fn execute_with_strategy( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let valid = match self.resolve_validity(&kernel, ctx)? { + ResolvedMask::Decided(result) => return Ok(Some(result)), + ResolvedMask::Mixed(valid) => valid, + }; + + match strategy { + NullStrategy::Filter => self.filter_and_scatter(kernel, &valid, ctx).map(Some), + NullStrategy::BranchAndSkip => self.execute_branched(branch, &valid, ctx), + } + } +} diff --git a/vortex-array/src/scalar_fn/row/mod.rs b/vortex-array/src/scalar_fn/row/mod.rs new file mode 100644 index 00000000000..f62d8bb8d49 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/mod.rs @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Defining scalar functions one row at a time. +//! +//! This is the derived way to write a scalar function, and the right default for a kernel that has +//! to read every row anyway. See [choosing a trait](crate::scalar_fn#choosing-a-trait) for when to +//! drop to [`ScalarFnVTable`](crate::scalar_fn::ScalarFnVTable) instead. +//! +//! [`RowFn`] names its arguments and provides a [`dispatch`](RowFn::dispatch) that picks the +//! concrete element and sink types for a batch. Everything else (dtype checks, output dtype, null +//! handling, constants, and validity) is derived from that dispatch. +//! +//! When the element types are fixed, `dispatch` is a single visit at those types. When one function +//! ID has to cover several (`l2_norm` accepts `f16`, `f32` and `f64` columns), `dispatch` matches on +//! the input dtypes and visits at the chosen width. Kernel fallibility is declared separately +//! because callers need it before dispatch. +//! +//! [`RowFn`] does not say how a row is _stored_, which is the element's job: `vortex-tensor` adds a +//! `TensorRow` [`InputElement`] and writes ordinary kernels over it. +//! +//! Output always goes through [`RowVisitor::visit_prepared_into`]. [`ElementSink`] covers one owned +//! [`OutputElement`] per row; custom [`OutputSink`] implementations cover runtime-shaped rows. The +//! prepare closure sees every batch-constant input and returns shared state for the row loop. Pass +//! `|_| ()` when there is nothing to prepare. +//! +//! A kernel that can safely write a provisional value uses [`DeferredError`] instead of returning +//! a per-row result. The executor vector-reduces those bits and hands one batch-wide error to the +//! sink. With nullable fixed-width inputs it runs densely and retries only valid rows on the cold +//! error path. +//! +//! Null handling is derived and executed by the [lifting](lift), never by the row closure, which +//! only ever computes rows valid in every argument. A batch with a mixed validity mask executes by +//! one of two strategies, selected per batch: _branch-and-skip_ (decode the unfiltered columns +//! null-tolerantly via [`InputElement::decode_null_tolerant`], compute only the valid rows a word +//! of the mask at a time, mask the result) whenever it can, and _filter_ (shrink every input to +//! the surviving rows, compute, scatter back) when an argument has no null-tolerant decode for its +//! array or when a per-row decode makes filtering cheaper at sparse validity. Authors do nothing; +//! an element whose decode does expensive per-row work reports that work through +//! [`InputElement::FILTERED_DECODE_COST`]. The costs of all arguments are added together so the +//! batch selector can distinguish one expensive decode from several. A sink opts into +//! branch-and-skip with [`OutputSink::SUPPORTS_SKIPPED_ROWS`]. + +mod element; +pub use element::ElementTuple; +pub use element::InputElement; +pub use element::OutputElement; +#[cfg(any(test, feature = "_test-harness"))] +pub use element::assert_element_conforms; + +mod result; +pub use result::DeferredError; +pub use result::SinkResult; + +mod sink; +pub use sink::ElementSink; +pub use sink::OutputSink; + +mod execute; + +mod lift; +#[cfg(any(test, feature = "_test-harness"))] +pub use lift::NullStrategy; + +mod row_fn; +pub use row_fn::RowFn; +pub use row_fn::RowVisitor; + +mod vtable; +#[cfg(any(test, feature = "_test-harness"))] +pub use vtable::execute_row_fn_with_strategy; + +#[cfg(test)] +mod tests; diff --git a/vortex-array/src/scalar_fn/row/result.rs b/vortex-array/src/scalar_fn/row/result.rs new file mode 100644 index 00000000000..86b90db41f5 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/result.rs @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What a sink-writing row closure may return. + +use std::ops::BitOrAssign; + +use vortex_error::VortexResult; + +mod private { + pub trait Sealed {} +} + +/// A value-dependent failure bit reduced across the whole row loop and handed to the output sink. +/// +/// Unlike [`VortexResult`], this never exits the loop. It is for kernels such as checked addition +/// that can safely write a provisional value for every row and report any failure once at the end. +/// +/// **The reduction is one byte wide on purpose.** It is OR-reduced once per row alongside the +/// kernel's own arithmetic, so a wider accumulator caps how many rows a vector of the reduction +/// covers, whatever the element width. Carrying the bit in an `i64` instead cost the primitive +/// `Mul` kernel 3.1x at `i8`, 1.9x at `i16` and 1.2x at `i32`, and nothing at `i64` where the two +/// widths already agree (`binary_ops`, 65536 rows, divan fastest of 100 samples, best of two runs, +/// Apple M4 Max). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct DeferredError(bool); + +impl DeferredError { + /// Record whether this row encountered an error. + pub const fn new(failed: bool) -> Self { + Self(failed) + } + + /// Whether any row accumulated into this value failed. + pub const fn occurred(self) -> bool { + self.0 + } +} + +impl BitOrAssign for DeferredError { + fn bitor_assign(&mut self, rhs: Self) { + self.0 |= rhs.0; + } +} + +/// What a row computation that _writes_ into an [`OutputSink`](crate::scalar_fn::OutputSink) may +/// produce: nothing, an early [`VortexResult`] error, or non-branching failure evidence. +/// +/// The value is already in the sink by the time the closure returns, so the only thing left to +/// report is failure. +/// +/// [`Accumulated`](Self::Accumulated) is the word the executor OR-reduces in a **local**, which is +/// what keeps the reduction in a register and the row loop vectorizable. It exists so that evidence +/// can be wider than one bit when narrowing it per row would cost more than carrying it: unsigned +/// multiplication hands back the discarded high half of its product, because comparing that half +/// against zero per row is what LLVM folds into `llvm.umul.with.overflow`, which has no vector form. +/// **The word must be no wider than the element**, or the reduction, rather than the arithmetic, +/// bounds how many rows a vector covers. +/// +/// The sink never sees this. It is handed a plain [`DeferredError`] once, after the loop. +/// +/// This trait is framework-only. Row functions choose one of the supplied return forms; custom +/// output representation belongs in [`OutputSink`](crate::scalar_fn::OutputSink). +pub trait SinkResult: 'static + private::Sealed { + /// The word this result reduces into, kept in a loop-local by the executor. + type Accumulated: 'static + Copy + Default; + + /// Whether this return type can carry an error. + const FALLIBLE: bool; + + /// Whether this result carries non-branching failure evidence for the sink. + const DEFERRED: bool; + + /// Merge this row's outcome into the batch-wide reduction. + fn accumulate(self, accumulated: &mut Self::Accumulated) -> VortexResult<()>; + + /// Whether the finished reduction means some row failed. + fn occurred(accumulated: Self::Accumulated) -> bool; +} + +impl private::Sealed for () {} + +impl SinkResult for () { + type Accumulated = (); + + const FALLIBLE: bool = false; + const DEFERRED: bool = false; + + fn accumulate(self, _accumulated: &mut ()) -> VortexResult<()> { + Ok(()) + } + + fn occurred(_accumulated: ()) -> bool { + false + } +} + +impl private::Sealed for VortexResult<()> {} + +impl SinkResult for VortexResult<()> { + type Accumulated = (); + + const FALLIBLE: bool = true; + const DEFERRED: bool = false; + + fn accumulate(self, _accumulated: &mut ()) -> VortexResult<()> { + self + } + + fn occurred(_accumulated: ()) -> bool { + false + } +} + +/// The evidence widths a row closure may reduce. `bool` is the ordinary answer; the unsigned +/// integers exist for a kernel whose per-row comparison would cost it its vectorization. +macro_rules! impl_sink_result_word { + ($($word:ty),+ $(,)?) => { + $( + impl private::Sealed for $word {} + + impl SinkResult for $word { + type Accumulated = $word; + + const FALLIBLE: bool = false; + const DEFERRED: bool = true; + + fn accumulate(self, accumulated: &mut $word) -> VortexResult<()> { + *accumulated |= self; + Ok(()) + } + + fn occurred(accumulated: $word) -> bool { + accumulated != <$word>::default() + } + } + )+ + }; +} + +impl_sink_result_word!(bool, u8, u16, u32, u64); + +#[cfg(test)] +mod tests { + use super::DeferredError; + + #[test] + fn one_failing_row_is_enough() { + let mut error = DeferredError::default(); + assert!(!error.occurred()); + + error |= DeferredError::new(false); + assert!(!error.occurred()); + + error |= DeferredError::new(true); + assert!(error.occurred()); + + error |= DeferredError::new(false); + assert!(error.occurred()); + } +} diff --git a/vortex-array/src/scalar_fn/row/row_fn.rs b/vortex-array/src/scalar_fn/row/row_fn.rs new file mode 100644 index 00000000000..f2a63636b1d --- /dev/null +++ b/vortex-array/src/scalar_fn/row/row_fn.rs @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Scalar functions computed one row at a time. + +use std::fmt::Debug; +use std::fmt::Display; +use std::hash::Hash; + +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_session::VortexSession; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::SinkResult; + +/// A scalar function computed one row at a time. +/// +/// An implementor declares its argument names, then [`dispatch`](Self::dispatch) picks the concrete +/// element and sink types for a batch. The planning visit reads dense safety, decode fallibility, +/// and decode cost from that concrete choice; no representative element types are needed. +/// +/// A function whose kernel is columnar rather than row-at-a-time (negating a whole bit buffer, a +/// zero-copy unwrap) is not a `RowFn`, and implements +/// [`ScalarFnVTable`](crate::scalar_fn::ScalarFnVTable) directly. +pub trait RowFn: 'static + Sized + Clone + Send + Sync { + /// Options for this function, if any. Use [`EmptyOptions`](crate::scalar_fn::EmptyOptions) + /// for none. + type Options: 'static + Send + Sync + Clone + Debug + Display + PartialEq + Eq + Hash; + + /// The arguments in display order. Its length is the function's exact arity. + const ARG_NAMES: &'static [&'static str]; + + /// Whether any legal dispatch can fail while decoding or computing a row. + /// + /// The framework verifies that every fallible dispatched element or result implies this value. + /// A conservative `true` is allowed when only some dtype choices are fallible. + const FALLIBLE: bool = false; + + /// Returns the ID of the scalar function. + fn id(&self) -> ScalarFnId; + + /// Serialize this function's options, or return `None` when the function is not serializable. + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + _ = options; + Ok(None) + } + + /// Restore options written by [`serialize`](Self::serialize). + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + vortex_bail!("Expression {} is not deserializable", self.id()) + } + + /// Choose element types for these input dtypes and visit the framework with them. + /// + /// This is where a per-batch width match lives (`match_each_float_ptype!` and friends panic + /// outside their width class, so check the class first), and where cross-argument dtype + /// constraints belong, since per-argument validation runs inside the visit. Plan time and run + /// time both come through here, so the choice **must** be a pure function of `options` and + /// `args`. + fn dispatch( + &self, + options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult; + + /// An encoding-aware rewrite, tried on the input arrays before the row loop. + /// + /// `Some` skips the row loop entirely, which makes this the escape hatch for a function that is + /// row-shaped in general but has a bulk answer for some encodings: reading stored values back out + /// of a wrapper encoding, or handing back a child array whole. The result may be lazy and + /// nullable, but its nulls **must** be a subset of the rows the lifting will mask, and it + /// **must** have one row per row of `args`, which on the filter strategy is the _filtered_ count + /// rather than the original one. Size the result from `args`, which are filtered to match, and + /// never from a length captured elsewhere. + /// + /// Whether the arrays still carry their original encoding depends on the execution path. + /// Dense execution always passes them through untouched. Valid-only execution does too when + /// no row is null; for a mixed mask, branch-and-skip also passes them through untouched (full + /// length, with the result masked afterwards), while filtering hands over filtered copies, + /// which are canonical and so match no encoding fast path. + /// + /// A non-nullable operand therefore reaches an encoding fast path under either. Note also that + /// filtering a constant yields a constant, so a fast path keyed on + /// [`as_constant`](ArrayRef::as_constant) still fires even for a filtered batch. + fn reduce_encoded( + &self, + options: &Self::Options, + args: &[ArrayRef], + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + _ = (options, args, ctx); + Ok(None) + } +} + +/// One use of a [`RowFn`] at concrete element types. +/// +/// The framework hands a visitor to [`RowFn::dispatch`], which calls one of the visit methods with +/// the element types it chose: at plan time the visit validates dtypes, at run time it executes the row +/// loop. Only the framework implements this trait, and a function only ever _calls_ a visit. +/// +/// The function names one output sink and one preparation step. Passing `|_| ()` is the no-prepare +/// case. +pub trait RowVisitor: private::Sealed { + /// What this visit produces. + type Out; + + /// Visit at argument tuple `A`, preparing shared state once and writing every output row into + /// sink `S`. + /// + /// `prepare` receives [`A::ConstElems`](ElementTuple::ConstElems): the element value of every + /// argument whose operand is constant for the batch, and `None` for each one that varies by + /// row. Whatever it returns is handed to every `apply` call by shared reference. + /// + /// `A` **must** have the arity declared by [`RowFn::ARG_NAMES`]. A fallible element or result + /// also requires [`RowFn::FALLIBLE`] to be `true`; the reverse is not required. A deferred result + /// must be paired with a sink whose [`OutputSink::ERRORS_ARE_DEFERRED`] is `true`. + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult; +} + +pub(super) mod private { + pub trait Sealed {} +} diff --git a/vortex-array/src/scalar_fn/row/sink.rs b/vortex-array/src/scalar_fn/row/sink.rs new file mode 100644 index 00000000000..50d07d1b7ff --- /dev/null +++ b/vortex-array/src/scalar_fn/row/sink.rs @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The column builders a row function can write its output into. + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::dtype::DType; +use crate::scalar_fn::DeferredError; +use crate::scalar_fn::OutputElement; + +/// A column allocated once per batch that a row closure writes into, one row at a time. +/// +/// Every [`RowFn`](crate::scalar_fn::RowFn) writes through one. [`ElementSink`] covers an ordinary +/// owned value per row. A custom sink covers output whose width is runtime data or whose rows append +/// into one batch-wide builder. +/// +/// Two properties of the contract are worth stating, since both are load-bearing: +/// +/// - **The row loop, not the closure, holds the sink.** [`row`](Self::row) is called by the framework +/// and its result passed in, so a writing closure stays [`Fn`] and captures nothing mutable. +/// Relaxing the row closure to `FnMut` instead was measured at 8 to 11%, because a captured `&mut` +/// inhibits vectorization of the loop. +/// - **[`sink_dtype`](Self::sink_dtype) sees the input dtypes**, unlike +/// [`OutputElement::element_dtype`](crate::scalar_fn::OutputElement::element_dtype), which takes +/// none. That is the whole reason a runtime-shaped output fits here: the width comes out of the +/// arguments. +/// +/// Rows arrive in increasing index order. Ordinary execution visits `0..row_count` exactly once; +/// branch-and-skip may omit null rows when [`SUPPORTS_SKIPPED_ROWS`](Self::SUPPORTS_SKIPPED_ROWS) +/// is `true`. +pub trait OutputSink: 'static + Sized { + /// Whether this sink accepts [`DeferredError`] from its row closure instead of requiring a + /// per-row [`VortexResult`]. + /// + /// The executor OR-reduces the row error words and passes the result to + /// [`finish`](Self::finish). When the arguments are safe to read behind nulls, this lets the + /// lifting optimistically run a dense loop. If `finish` reports the deferred error for a + /// nullable batch, the lifting retries over only the valid rows: success means the error came + /// exclusively from null rows, while another deferred error is real. + /// + /// A supporting sink must return an error from `finish` when its `error` argument occurred. + const ERRORS_ARE_DEFERRED: bool = false; + + /// Whether this sink can finish a full-length output when some rows were never visited. + /// + /// A supporting sink must leave a legal arbitrary value at every skipped row. The lifting masks + /// those rows before the result escapes, so that value is never observable. + const SUPPORTS_SKIPPED_ROWS: bool = false; + + /// A loop-local view of all output rows. + /// + /// Borrowed once before execution so the sink's buffer descriptor and shape become loop + /// invariants rather than being re-read through `&mut Self` for every row. + type Rows<'a> + where + Self: 'a; + + /// The place a row closure writes one row through, borrowed from the sink. + type Row<'a> + where + Self: 'a; + + /// The dtype of the column this sink builds, given the function's input dtypes. + /// + /// Must be non-nullable: nullability is derived from the inputs by the lifting, which + /// widens the result and masks the null rows itself. + fn sink_dtype(args: &[DType]) -> VortexResult; + + /// Allocate a sink for `rows` rows of `dtype`, which is this sink's own + /// [`sink_dtype`](Self::sink_dtype). Called once per batch. + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult; + + /// Borrow all output rows for the hot loop. + fn rows(&mut self) -> Self::Rows<'_>; + + /// Whether every index in `0..row_count` is addressable through [`row`](Self::row). + /// + /// Called once before the hot loop. Besides validating the sink contract, this gives the + /// optimizer the output bounds it needs to remove the bounds check hidden in each row accessor. + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool; + + /// Hand out the place to write row `index`. Must be `O(1)`: it is called in the row loop. + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a>; + + /// Finish into the built column, whose dtype **must** be this sink's + /// [`sink_dtype`](Self::sink_dtype). Called once per batch with the OR of every row's deferred + /// error bit. + fn finish(self, error: DeferredError) -> VortexResult; +} + +/// The standard output sink for one owned [`OutputElement`] per row. +pub struct ElementSink { + values: Vec, +} + +impl OutputSink for ElementSink { + const SUPPORTS_SKIPPED_ROWS: bool = true; + + type Rows<'a> = &'a mut [T]; + type Row<'a> = &'a mut T; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(T::element_dtype()) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + // `vec![placeholder; rows]` rather than `resize_with(rows, placeholder)`: the former hands + // a zeroable placeholder (every primitive, `false`) straight to `alloc_zeroed`, while the + // latter always writes one element at a time. Only branch-and-skip ever reads a + // placeholder back, so on the dense and filter paths that write is pure waste. + Ok(Self { + values: vec![T::placeholder(); rows], + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + &mut self.values + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + &mut rows[index] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(T::build(self.values)) + } +} diff --git a/vortex-array/src/scalar_fn/row/tests/conformance.rs b/vortex-array/src/scalar_fn/row/tests/conformance.rs new file mode 100644 index 00000000000..8df438a734c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/conformance.rs @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Conformance tests for every [`InputElement`](crate::scalar_fn::InputElement) in this crate. + +use std::sync::Arc; + +use vortex_buffer::BitBuffer; +use vortex_buffer::ByteBuffer; +use vortex_buffer::buffer; +use vortex_error::VortexResult; + +use crate::IntoArray; +use crate::VortexSessionExecute; +use crate::array_session; +use crate::arrays::BoolArray; +use crate::arrays::PrimitiveArray; +use crate::arrays::VarBinViewArray; +use crate::arrays::varbinview::BinaryView; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar_fn::assert_element_conforms; +use crate::scalar_fn::row::tests::TestBytes; +use crate::validity::Validity; + +/// A `Utf8` column whose single null row carries a view naming a buffer that does not exist, at +/// an offset far past the end of the data. Reading its _bytes_ densely panics; reading its +/// _length_ does not, which is exactly the distinction `DENSE_SAFE` encodes. +fn hostile_views() -> VortexResult { + let views = buffer![ + BinaryView::make_view(b"a longer string here", 0, 0), + BinaryView::new_ref(64, *b"junk", 9, 4096), + ]; + Ok(VarBinViewArray::try_new( + views, + Arc::from([ByteBuffer::copy_from(b"a longer string here")]), + DType::Utf8(Nullability::Nullable), + Validity::from_iter([true, false]), + )? + .into_array()) +} + +#[test] +fn primitive_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + // The extremes sit at the rows that are then marked null. + let array = PrimitiveArray::new( + buffer![i32::MAX, 1, i32::MIN, 2], + Validity::from_iter([false, true, false, true]), + ) + .into_array(); + + assert_element_conforms::(array, &DType::Utf8(Nullability::NonNullable), &mut ctx) +} + +#[test] +fn bool_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let array = BoolArray::new( + BitBuffer::from(vec![true, true, false, true]), + Validity::from_iter([false, true, true, false]), + ) + .into_array(); + + assert_element_conforms::(array, &DType::Utf8(Nullability::NonNullable), &mut ctx) +} + +#[test] +fn test_bytes_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + assert_element_conforms::( + hostile_views()?, + &DType::Bool(Nullability::NonNullable), + &mut ctx, + ) +} diff --git a/vortex-array/src/scalar_fn/row/tests/constant_operands.rs b/vortex-array/src/scalar_fn/row/tests/constant_operands.rs new file mode 100644 index 00000000000..5e86b925587 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/constant_operands.rs @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests that constant operands are decoded once and broadcast across the batch. + +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; + +use vortex_buffer::Buffer; + +use super::*; + +/// Total rows handed to [`CountedI64::decode`] across one execution. Sound as a global because +/// each test binary runs one test per process. +static DECODED_ROWS: AtomicUsize = AtomicUsize::new(0); + +/// Stands in for an element whose decode is expensive per row, recording how wide a column each +/// decode was actually given. +struct CountedI64; + +impl InputElement for CountedI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + DECODED_ROWS.fetch_add(array.len(), Ordering::Relaxed); + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +#[derive(Clone)] +struct AddCounted; + +impl RowFn for AddCounted { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.add_counted"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(CountedI64, CountedI64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs + rhs, + ) + } +} + +/// An element whose decode drops the last row, standing in for a buggy element implementation. +struct ShortDecodeI64; + +impl InputElement for ShortDecodeI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let column = ::decode(array, ctx)?; + Ok(column.slice(0..column.len().saturating_sub(1))) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +/// Pairs a short-decoding argument with an ordinary one, so a batch-constant second operand takes +/// the mixed constant-and-varying read path rather than the all-varying one. +#[derive(Clone)] +struct AddShort; + +impl RowFn for AddShort { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.add_short"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(ShortDecodeI64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs + rhs, + ) + } +} + +/// A constant operand makes the whole tuple decline the all-varying read path, so the row loop +/// indexes each [`ArgColumn`](crate::scalar_fn::ArgColumn) directly. The decoded length still has to +/// be checked there, or a short column reaches an out-of-bounds row read. +#[test] +fn a_short_decode_beside_a_constant_operand_is_rejected() { + let mut ctx = array_session().create_execution_ctx(); + let column = PrimitiveArray::from_iter(0..64i64).into_array(); + let constant = ConstantArray::new(Scalar::from(10i64), 64).into_array(); + + let error = apply(AddShort, [column, constant], &mut ctx).unwrap_err(); + + assert!( + error + .to_string() + .contains("does not address exactly 64 rows"), + "{error}" + ); +} + +#[test] +fn a_constant_operand_is_decoded_once() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let column = PrimitiveArray::from_iter(0..64i64).into_array(); + let constant = ConstantArray::new(Scalar::from(10i64), 64).into_array(); + + let result = apply(AddCounted, [column, constant], &mut ctx)?; + + // 64 rows for the real column, plus exactly one for the constant. + assert_eq!(DECODED_ROWS.load(Ordering::Relaxed), 65); + assert_arrays_eq!( + result, + PrimitiveArray::from_iter((0..64i64).map(|value| value + 10)), + &mut ctx + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs b/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs new file mode 100644 index 00000000000..e3ade44f680 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for fallible input decoding and its effect on execution strategy. + +use super::*; + +/// Stands in for an element that _parses_ its bytes, like a WKB geometry: malformed bytes in a +/// valid row are a domain error, so decoding can fail on otherwise legal input. +struct ParsedBytes; + +impl InputElement for ParsedBytes { + type Column = VarBinViewArray; + type Varying<'a> = &'a VarBinViewArray; + type Elem<'a> = usize; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = true; + + fn validate(_dtype: &DType) -> VortexResult<()> { + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + array.execute::(ctx) + } + + fn get(column: &Self::Column, index: usize) -> usize { + column.views()[index].len() as usize + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> usize + where + Self: 'a, + { + Self::get(column, index) + } +} + +/// Its row computation is total; only the decode can fail. +#[derive(Clone)] +struct TotalKernelOverParsedInput; + +impl RowFn for TotalKernelOverParsedInput { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.total_over_parsed"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(ParsedBytes,), ElementSink, _, _>( + |_| (), + |&(), (len,), output| *output = len as u64, + ) + } +} + +/// The row closure is infallible, so reading only the kernel declaration would report +/// `false` and let dict pushdown speculatively evaluate the parse over unreferenced values. +#[test] +fn a_fallible_decode_makes_the_function_fallible() { + assert!(ScalarFnVTable::is_fallible( + &TotalKernelOverParsedInput, + &EmptyOptions + )); +} + +/// And it must not run densely: rows behind nulls would be parsed too. +#[test] +fn a_fallible_decode_forces_filtering() { + assert_eq!( + policy( + &TotalKernelOverParsedInput, + &[DType::Binary(Nullability::Nullable)] + ), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/dispatched.rs b/vortex-array/src/scalar_fn/row/tests/dispatched.rs new file mode 100644 index 00000000000..ca377bea4b1 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/dispatched.rs @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for row functions that choose their element types per batch. + +use vortex_error::vortex_ensure; + +use super::*; +use crate::match_each_integer_ptype; + +#[derive(Clone)] +struct Max; + +impl RowFn for Max { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.int_max"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + let DType::Primitive(ptype, _) = args[0] else { + vortex_bail!("int_max requires primitive inputs, got {}", args[0]); + }; + vortex_ensure!( + ptype.is_int(), + "int_max requires integer inputs, got {ptype}" + ); + + match_each_integer_ptype!(ptype, |T| { + visitor.visit_prepared_into::<(T, T), ElementSink, _, _>( + |_| (), + |&(), (a, b), output| *output = a.max(b), + ) + }) + } +} + +#[rstest] +#[case::i16(buffer![1i16, 9, 3].into_array(), buffer![4i16, 2, 3].into_array(), buffer![4i16, 9, 3].into_array())] +#[case::i64(buffer![1i64, 9, 3].into_array(), buffer![4i64, 2, 3].into_array(), buffer![4i64, 9, 3].into_array())] +#[case::u8(buffer![1u8, 9, 3].into_array(), buffer![4u8, 2, 3].into_array(), buffer![4u8, 9, 3].into_array())] +fn dispatches_at_each_integer_width( + #[case] lhs: ArrayRef, + #[case] rhs: ArrayRef, + #[case] expected: ArrayRef, +) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let result = apply(Max, [lhs, rhs], &mut ctx)?; + + assert_arrays_eq!(result, expected, &mut ctx); + Ok(()) +} + +#[test] +fn rejects_a_float_width() { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![1.0f64].into_array(); + let rhs = buffer![2.0f64].into_array(); + + let error = apply(Max, [lhs, rhs], &mut ctx) + .expect_err("a float width must be rejected at construction"); + + assert!( + error.to_string().contains("integer inputs"), + "unexpected error: {error}" + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/lifting.rs b/vortex-array/src/scalar_fn/row/tests/lifting.rs new file mode 100644 index 00000000000..f7d2fba4d04 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/lifting.rs @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for null propagation, constant folding, nullability widening, and options serde. + +use super::*; +use crate::dtype::Nullability; +use crate::dtype::PType; + +/// An `i32` element that is [dense-safe] iff `DENSE`, and otherwise the plain `i32` element in +/// every respect. Dense-safety is what decides the null-handling path, so a pair of these is +/// how one kernel gets run under both. +/// +/// [dense-safe]: InputElement::DENSE_SAFE +struct MaybeDenseI32; + +impl InputElement for MaybeDenseI32 { + type Column = ::Column; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i32; + + const DENSE_SAFE: bool = DENSE; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i32 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i32 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +/// Wrapping addition over two [`MaybeDenseI32`] columns. +#[derive(Clone)] +struct Add; + +impl RowFn for Add { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + if DENSE { + static ID: CachedId = CachedId::new("vortex.test.add.dense"); + *ID + } else { + static ID: CachedId = CachedId::new("vortex.test.add.filter"); + *ID + } + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::< + (MaybeDenseI32, MaybeDenseI32), + ElementSink, + _, + _, + >( + |_| (), + |&(), (lhs, rhs), output| *output = lhs.wrapping_add(rhs), + ) + } +} + +/// Adds `lhs` to `rhs` under both null-handling paths and asserts each result equals +/// `expected`, which is what every case below does. +/// +/// Forcing a _strategy_ within the filter contract is a separate axis, covered in +/// [`null_strategies`](super::null_strategies). +fn assert_add(lhs: ArrayRef, rhs: ArrayRef, expected: ArrayRef) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let dense = apply(Add::, [lhs.clone(), rhs.clone()], &mut ctx)?; + let filtered = apply(Add::, [lhs, rhs], &mut ctx)?; + + let args = [ + DType::Primitive(PType::I32, Nullability::NonNullable), + DType::Primitive(PType::I32, Nullability::NonNullable), + ]; + assert_eq!(policy(&Add::, &args), RowPolicy::Dense); + assert_eq!( + policy(&Add::, &args), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + assert_arrays_eq!(dense, expected, &mut ctx); + assert_arrays_eq!(filtered, expected, &mut ctx); + Ok(()) +} + +#[test] +fn no_nulls() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), + PrimitiveArray::from_iter([10i32, 20, 30]).into_array(), + PrimitiveArray::from_iter([11i32, 22, 33]).into_array(), + ) +} + +#[test] +fn nulls_propagate() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_option_iter([Some(1i32), None, Some(3), None]).into_array(), + PrimitiveArray::from_option_iter([Some(10i32), Some(20), None, None]).into_array(), + PrimitiveArray::from_option_iter([Some(11i32), None, None, None]).into_array(), + ) +} + +/// Strictness: a null constant makes the whole output null without the kernel running at all. +#[test] +fn null_constant_short_circuits() -> VortexResult<()> { + let null = Scalar::null(DType::Primitive(PType::I32, Nullability::Nullable)); + + assert_add( + PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), + ConstantArray::new(null, 3).into_array(), + PrimitiveArray::from_option_iter([Option::::None, None, None]).into_array(), + ) +} + +/// All-constant inputs evaluate one row and broadcast it. +#[test] +fn all_constants_broadcast() -> VortexResult<()> { + assert_add( + ConstantArray::new(Scalar::from(2i32), 4).into_array(), + ConstantArray::new(Scalar::from(40i32), 4).into_array(), + PrimitiveArray::from_iter([42i32, 42, 42, 42]).into_array(), + ) +} + +#[test] +fn mixed_constant_and_column() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_option_iter([Some(1i32), None, Some(3)]).into_array(), + ConstantArray::new(Scalar::from(10i32), 3).into_array(), + PrimitiveArray::from_option_iter([Some(11i32), None, Some(13)]).into_array(), + ) +} + +/// An empty batch is neither all-valid nor all-null, and a zero-length non-nullable execution +/// keeps its non-nullable dtype. +#[test] +fn empty_input_keeps_dtype() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let empty = || PrimitiveArray::from_iter(Vec::::new()).into_array(); + + let result = apply(Add::, [empty(), empty()], &mut ctx)?; + + assert_eq!(result.len(), 0); + assert!(!result.dtype().is_nullable()); + Ok(()) +} + +/// The output element dtype is non-nullable, and the lifting widens it iff an input is +/// nullable, which is what makes strictness's dtype contract hold by construction. +#[test] +fn return_dtype_unions_nullability() -> VortexResult<()> { + let non_nullable = DType::Primitive(PType::I32, Nullability::NonNullable); + let nullable = non_nullable.as_nullable(); + + assert_eq!( + ScalarFnVTable::return_dtype( + &Add::, + &EmptyOptions, + &[non_nullable.clone(), non_nullable.clone()] + )?, + non_nullable + ); + assert_eq!( + ScalarFnVTable::return_dtype( + &Add::, + &EmptyOptions, + &[non_nullable, nullable.clone()] + )?, + nullable + ); + Ok(()) +} + +#[test] +fn a_row_fn_is_strict() { + assert!(ScalarFnVTable::is_strict(&Add::, &EmptyOptions)); +} + +/// Output sinks build an all-valid column, so the output validity is exactly the child +/// conjunction and the planner never has to execute the function to learn which rows are null. +#[test] +fn validity_is_the_child_conjunction() -> VortexResult<()> { + let expr = Add::.new_expr(EmptyOptions, [root(), root()]); + + assert!(ScalarFnVTable::validity(&Add::, &EmptyOptions, &expr)?.is_some()); + Ok(()) +} + +/// A row function is not serializable until the function opts into a wire representation. +#[test] +fn options_are_not_serializable_by_default() -> VortexResult<()> { + assert_eq!( + ScalarFnVTable::serialize(&Add::, &EmptyOptions)?, + None + ); + assert!(ScalarFnVTable::deserialize(&Add::, &[], &array_session()).is_err()); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/mod.rs b/vortex-array/src/scalar_fn/row/tests/mod.rs new file mode 100644 index 00000000000..c8f2ab77aee --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/mod.rs @@ -0,0 +1,509 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! End-to-end tests for row function execution. + +use rstest::rstest; +use vortex_buffer::ByteBuffer; +use vortex_buffer::buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_session::registry::CachedId; + +use super::lift::RowPolicy; +use super::vtable::row_policy; +use crate::ArrayRef; +use crate::Canonical; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::VortexSessionExecute; +use crate::array_session; +use crate::arrays::ConstantArray; +use crate::arrays::MaskedArray; +use crate::arrays::PrimitiveArray; +use crate::arrays::VarBinViewArray; +use crate::arrays::scalar_fn::ScalarFnFactoryExt; +use crate::arrays::varbinview::BinaryView; +use crate::assert_arrays_eq; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::expr::root; +use crate::scalar::Scalar; +use crate::scalar_fn::*; + +mod conformance; +mod constant_operands; +mod decode_fallibility; +mod dispatched; +mod lifting; +mod null_strategies; +mod nullable_outputs; +mod prepared; +mod sink; + +/// Builds `scalar_fn` over `args` and executes it end to end, which is what every test below does. +fn apply>( + scalar_fn: F, + args: impl IntoIterator, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = args.into_iter().collect::>(); + let rows = args.first().map_or(0, |arg| arg.len()); + + Ok(scalar_fn + .try_new_array(rows, EmptyOptions, args)? + .execute::(ctx)? + .into_array()) +} + +/// A binary row function over fixed primitive types: `hypot(x, y)`. +#[derive(Clone)] +struct Hypot; + +impl RowFn for Hypot { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["x", "y"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.hypot"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(f64, f64), ElementSink, _, _>( + |_| (), + |&(), (x, y), output| *output = x.hypot(y), + ) + } +} + +/// A byte-string element that resolves each row's view into a data buffer, which is only +/// meaningful for a valid row. +/// +/// This is the crate's only non-dense-safe element, so it exercises valid-only execution, +/// branch-and-skip, and the agreement between the two strategies. It lives here rather than beside +/// the framework because no production row function reads bytes yet. +struct TestBytes; + +/// The canonical views array plus its resolved data buffers. +struct TestBytesColumn { + array: VarBinViewArray, + buffers: Vec, +} + +/// Resolve one view, which is either inlined or an offset into `buffers`. +fn read_view<'a>(view: &'a BinaryView, buffers: &'a [ByteBuffer]) -> &'a [u8] { + if view.is_inlined() { + view.as_inlined().value() + } else { + let view = view.as_view(); + &buffers[view.buffer_index as usize].as_slice()[view.as_range()] + } +} + +impl InputElement for TestBytes { + type Column = TestBytesColumn; + // The views slice, not the array: `VarBinViewArray::views` resolves a host buffer and its + // `vortex_expect` is a side effect the optimizer cannot hoist out of the row loop. + type Varying<'a> = (&'a [BinaryView], &'a [ByteBuffer]); + type Elem<'a> = &'a [u8]; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + matches!(dtype, DType::Utf8(_) | DType::Binary(_)), + "expected a Utf8 or Binary column, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let array = array.execute::(ctx)?; + let buffers = (0..array.data_buffers().len()) + .map(|idx| array.buffer(idx).clone()) + .collect(); + Ok(TestBytesColumn { array, buffers }) + } + + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Self::decode(array, ctx).map(Some) + } + + fn get(column: &Self::Column, index: usize) -> &[u8] { + read_view(&column.array.views()[index], &column.buffers) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + (column.array.views(), &column.buffers) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.0.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a [u8] + where + Self: 'a, + { + read_view(&column.0[index], column.1) + } +} + +impl OutputElement for String { + fn element_dtype() -> DType { + DType::Utf8(Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + VarBinViewArray::from_iter_str(values).into_array() + } + + fn placeholder() -> Self { + String::new() + } +} + +/// A unary row function over strings: uppercased text, exercising [`TestBytes`] input and +/// [`String`] output. +#[derive(Clone)] +struct Shout; + +impl RowFn for Shout { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.shout"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(TestBytes,), ElementSink, _, _>( + |_| (), + |&(), (text,), output| { + *output = String::from_utf8_lossy(text).to_uppercase(); + }, + ) + } +} + +/// A fallible row function: integer division, undefined at a zero divisor. +#[derive(Clone)] +struct CheckedDiv; + +impl RowFn for CheckedDiv { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.checked_div"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| { + if rhs == 0 { + vortex_bail!("division by zero"); + } + *output = lhs / rhs; + Ok(()) + }, + ) + } +} + +#[test] +fn hypot_columns() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0].into_array(); + let y = buffer![4.0f64, 12.0].into_array(); + + let result = apply(Hypot, [x, y], &mut ctx)?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([5.0f64, 13.0]), &mut ctx); + Ok(()) +} + +#[test] +fn hypot_propagates_nulls_and_constants() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = PrimitiveArray::from_option_iter([Some(3.0f64), None, Some(8.0)]).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + + let result = apply(Hypot, [x, y], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} + +#[test] +fn shout_strings() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = + VarBinViewArray::from_iter_nullable_str([Some("hello"), None, Some("Vortex")]).into_array(); + + let result = apply(Shout, [input], &mut ctx)?; + + let expected = + VarBinViewArray::from_iter_nullable_str([Some("HELLO"), None, Some("VORTEX")]).into_array(); + assert_arrays_eq!(result, expected, &mut ctx); + Ok(()) +} + +#[test] +fn display_names_the_function_id() { + let expr = Hypot.new_expr(EmptyOptions, [root(), root()]); + assert_eq!(expr.to_string(), "vortex.test.hypot($, $)"); +} + +#[derive(Clone)] +struct WrongLength; + +impl RowFn for WrongLength { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.wrong_length"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = value, + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + _args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(PrimitiveArray::from_iter([0i64]).into_array())) + } +} + +#[test] +fn kernel_result_length_is_validated() { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, 2, 3].into_array(); + + let error = apply(WrongLength, [input], &mut ctx).unwrap_err(); + + assert!( + error + .to_string() + .contains("produced 1 rows for 3 input rows"), + "{error}" + ); +} + +#[derive(Clone)] +struct FortyTwo; + +impl RowFn for FortyTwo { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &[]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.forty_two"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(), ElementSink, _, _>( + |()| (), + |&(), (), output| *output = 42, + ) + } +} + +#[test] +fn nullary_row_fn_executes_requested_rows() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let result = FortyTwo + .try_new_array(3, EmptyOptions, [])? + .execute::(&mut ctx)?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([42i64; 3]), &mut ctx); + Ok(()) +} + +#[derive(Clone)] +struct SumFour; + +impl RowFn for SumFour { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["a", "b", "c", "d"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.sum_four"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64, i64, i64), ElementSink, _, _>( + |_| (), + |&(), (a, b, c, d), output| *output = a + b + c + d, + ) + } +} + +#[test] +fn four_argument_row_fn_executes() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let result = apply( + SumFour, + [ + buffer![1i64, 2].into_array(), + buffer![10i64, 20].into_array(), + buffer![100i64, 200].into_array(), + buffer![1000i64, 2000].into_array(), + ], + &mut ctx, + )?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([1111i64, 2222]), &mut ctx); + Ok(()) +} + +#[test] +fn tuples_are_supported_through_arity_twelve() { + type TwelveI64s = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64); + + assert_eq!(<() as ElementTuple>::ARITY, 0); + assert_eq!(::ARITY, 12); +} + +#[test] +fn kernel_flag_decides_fallibility() { + assert!(!ScalarFnVTable::is_fallible(&Hypot, &EmptyOptions)); + assert!(ScalarFnVTable::is_fallible(&CheckedDiv, &EmptyOptions)); +} + +#[test] +fn fallible_apply_propagates_its_error() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10].into_array(); + let rhs = buffer![2i64, 0].into_array(); + + let error = apply(CheckedDiv, [lhs, rhs], &mut ctx) + .expect_err("a zero divisor must fail the execution"); + + assert!( + error.to_string().contains("division by zero"), + "unexpected error: {error}" + ); + Ok(()) +} + +/// The divisor's null slot holds a zero, which a dense pass would divide by. Filtering keeps the +/// fallible kernel away from it. +#[test] +fn fallible_apply_never_sees_rows_behind_nulls() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10].into_array(); + let rhs = PrimitiveArray::from_option_iter([Some(2i64), None]).into_array(); + + let result = apply(CheckedDiv, [lhs, rhs], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5i64), None]), + &mut ctx + ); + Ok(()) +} + +/// The internal nullable execution policy selected by a concrete dispatch. +fn policy>(row_fn: &F, args: &[DType]) -> RowPolicy { + row_policy(row_fn, &EmptyOptions, args).expect("test dispatch must produce a policy") +} + +/// The function never declares a policy: the dispatched arguments and result decide it. +#[test] +fn null_handling_follows_from_args_and_fallibility() { + // Primitive arguments, infallible: nothing behind a null row can fault. + assert_eq!( + policy( + &Hypot, + &[ + DType::Primitive(PType::F64, Nullability::NonNullable), + DType::Primitive(PType::F64, Nullability::NonNullable), + ] + ), + RowPolicy::Dense + ); + // `TestBytes` resolves a view into a data buffer, which is only meaningful for valid rows. + assert_eq!( + policy(&Shout, &[DType::Utf8(Nullability::Nullable)]), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + // Fallible: a garbage row could raise an error of its own. + assert_eq!( + policy( + &CheckedDiv, + &[ + DType::Primitive(PType::I64, Nullability::NonNullable), + DType::Primitive(PType::I64, Nullability::NonNullable), + ] + ), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/null_strategies.rs b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs new file mode 100644 index 00000000000..d8a7fe18816 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs @@ -0,0 +1,502 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Equivalence and selection tests for filtering and branch-and-skip null execution. + +use std::sync::Arc; + +use vortex_buffer::ByteBuffer; + +use super::*; +use crate::arrays::varbinview::BinaryView; +use crate::dtype::Nullability; +use crate::validity::Validity; + +/// Executes `scalar_fn` over `args` with `strategy` forced, canonicalized like [`apply`]. +fn apply_forced>( + scalar_fn: &F, + args: &[ArrayRef], + strategy: NullStrategy, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let rows = args.first().map_or(0, |arg| arg.len()); + + Ok( + execute_row_fn_with_strategy(scalar_fn, &EmptyOptions, args.to_vec(), rows, strategy, ctx)? + .execute::(ctx)? + .into_array(), + ) +} + +/// Runs `scalar_fn` under forced filter, forced branch-and-skip, and the automatic per-batch +/// selection, and asserts all three produce identical arrays. +fn assert_strategies_agree>( + scalar_fn: F, + args: Vec, +) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let filtered = apply_forced(&scalar_fn, &args, NullStrategy::Filter, &mut ctx)?; + let branched = apply_forced(&scalar_fn, &args, NullStrategy::BranchAndSkip, &mut ctx)?; + let auto = apply(scalar_fn, args, &mut ctx)?; + + assert_arrays_eq!(branched, filtered, &mut ctx); + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) +} + +/// A `Utf8` column whose null rows carry views naming a buffer that does not exist, at +/// offsets far out of bounds. Resolving such a row's bytes panics, so strategy agreement +/// proves the branch loop never calls `get` behind a null. +fn hostile_nullable_strings() -> VortexResult { + let views = buffer![ + BinaryView::make_view(b"a longer string here", 0, 0), + BinaryView::new_ref(64, *b"junk", 9, 4096), + BinaryView::make_view(b"another non-inlined string", 1, 0), + BinaryView::new_ref(64, *b"junk", 7, 1 << 20), + ]; + + Ok(VarBinViewArray::try_new( + views, + Arc::from([ + ByteBuffer::copy_from(b"a longer string here"), + ByteBuffer::copy_from(b"another non-inlined string"), + ]), + DType::Utf8(Nullability::Nullable), + Validity::from_iter([true, false, true, false]), + )? + .into_array()) +} + +/// `Bytes` is not dense-safe, so `Shout` uses valid-only execution; both strategies must produce +/// the same array without resolving the hostile views behind the nulls. +#[test] +fn branch_matches_filter_for_bytes() -> VortexResult<()> { + assert_strategies_agree(Shout, vec![hostile_nullable_strings()?]) +} + +/// A fallible kernel with a poison value (zero divisor) behind every null: the branch loop +/// must skip those rows rather than spuriously failing on them. +#[test] +fn branch_never_applies_a_fallible_kernel_behind_nulls() -> VortexResult<()> { + let lhs = buffer![10i64, 10, 12, 9].into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 0, 3, 0], + Validity::from_iter([true, false, true, false]), + ) + .into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// Nulls in both operands: the branch loop must honor the _conjoined_ mask, not either +/// input's own validity. +#[test] +fn branch_conjoins_validities() -> VortexResult<()> { + let lhs = + PrimitiveArray::from_option_iter([Some(10i64), None, Some(12), Some(9), None]).into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 0, 3, 0, 0], + Validity::from_iter([true, true, true, false, false]), + ) + .into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// A constant operand under the branch strategy still hoists through the stride-0 decode. +#[test] +fn branch_handles_constant_operands() -> VortexResult<()> { + let lhs = PrimitiveArray::from_option_iter([Some(10i64), None, Some(12)]).into_array(); + let rhs = ConstantArray::new(Scalar::from(2i64), 3).into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// An error from a _valid_ row still propagates under the branch strategy. +#[test] +fn branch_propagates_real_errors() { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10, 12].into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 3, 0], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let error = apply_forced( + &CheckedDiv, + &[lhs, rhs], + NullStrategy::BranchAndSkip, + &mut ctx, + ) + .expect_err("a zero divisor in a valid row must fail"); + + assert!( + error.to_string().contains("division by zero"), + "unexpected error: {error}" + ); +} + +/// The automatic per-batch selection, observed through elements that record which decode ran +/// on how many rows: the branch strategy decodes null-tolerantly at full length, the filter +/// strategy decodes ordinarily over the survivors. +mod selection { + use std::cell::Cell; + use std::cell::RefCell; + + use vortex_buffer::Buffer; + use vortex_error::vortex_err; + use vortex_mask::Mask; + + use super::*; + use crate::scalar_fn::row::lift::branch_beats_filter; + + thread_local! { + /// What the last varying-column decode did: `(null_tolerant, rows)`. Thread-local so + /// concurrent tests in one process cannot race it; execution runs on the calling + /// thread. + static LAST_DECODE: Cell> = const { Cell::new(None) }; + } + + /// An i64 element that records its decodes and reports `COST` units of filtered decode work. + /// It is not dense-safe, so strategy selection actually happens. + struct TrackedI64; + + impl InputElement for TrackedI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + const FILTERED_DECODE_COST: usize = COST; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + LAST_DECODE.set(Some((false, array.len()))); + ::decode(array, ctx) + } + + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + LAST_DECODE.set(Some((true, array.len()))); + ::decode(array, ctx).map(Some) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } + } + + /// Negation over one tracked column. + #[derive(Clone)] + struct TrackedNegate; + + impl RowFn for TrackedNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + if COST == 0 { + static ID: CachedId = CachedId::new("vortex.test.tracked_negate.bulk"); + *ID + } else { + static ID: CachedId = CachedId::new("vortex.test.tracked_negate.per_row"); + *ID + } + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(TrackedI64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + } + + /// A 32-row nullable i64 column whose first `valid_count` rows are valid. + fn column_with_survivors(valid_count: usize) -> ArrayRef { + PrimitiveArray::from_option_iter( + (0..32u16).map(|i| (usize::from(i) < valid_count).then_some(i64::from(i))), + ) + .into_array() + } + + /// Executes the tracked function through the full pipeline and returns what the decode + /// recorded: whether it was null-tolerant, and how many rows it saw. + fn run(valid_count: usize) -> VortexResult<(bool, usize)> { + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + + apply( + TrackedNegate::, + [column_with_survivors(valid_count)], + &mut ctx, + )?; + + LAST_DECODE + .get() + .ok_or_else(|| vortex_err!("no decode ran")) + } + + /// A bulk-decoded element takes branch-and-skip on a mixed mask however sparse the + /// survivors: the decode is null-tolerant and full length. + #[test] + fn bulk_decode_branches_at_any_density() -> VortexResult<()> { + assert_eq!(run::<0>(31)?, (true, 32)); + assert_eq!(run::<0>(4)?, (true, 32)); + Ok(()) + } + + /// One per-row decode still branches when half the rows survive, matching the measured + /// single-nullable-input crossover. + #[test] + fn per_row_decode_filters_when_sparse() -> VortexResult<()> { + // 30/32 surviving: branch, full-length null-tolerant decode. + assert_eq!(run::<1>(30)?, (true, 32)); + // 16/32 = 50% surviving sits exactly on the threshold: still branch. + assert_eq!(run::<1>(16)?, (true, 32)); + // Below 50% surviving: filter, ordinary decode over the survivors. + assert_eq!(run::<1>(15)?, (false, 15)); + Ok(()) + } + + /// An all-true mask short-circuits to the plain kernel and an all-false mask to an + /// all-null constant, before any strategy is selected. + #[test] + fn degenerate_masks_bypass_the_selection() -> VortexResult<()> { + assert_eq!(run::<1>(32)?, (false, 32)); + + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + apply(TrackedNegate::<1>, [column_with_survivors(0)], &mut ctx)?; + assert_eq!(LAST_DECODE.get(), None); + Ok(()) + } + + /// An i64 element that omits `decode_null_tolerant`: the conservative default refuses, so + /// the batch must fall back to the filter strategy even though the selection preferred + /// branch. + struct RefusesNullTolerant; + + impl InputElement for RefusesNullTolerant { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + LAST_DECODE.set(Some((false, array.len()))); + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } + } + + #[derive(Clone)] + struct RefusingNegate; + + impl RowFn for RefusingNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.refusing_negate"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(RefusesNullTolerant,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + } + + /// The fallback is silent and correct: the ordinary decode runs over the survivors and + /// the result matches the expected negation. + #[test] + fn missing_null_tolerant_decode_falls_back_to_filter() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + + let result = apply( + RefusingNegate, + [PrimitiveArray::from_option_iter([Some(3i64), None, Some(5)]).into_array()], + &mut ctx, + )?; + + assert_eq!(LAST_DECODE.get(), Some((false, 2))); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(-3i64), None, Some(-5)]), + &mut ctx + ); + Ok(()) + } + + thread_local! { + /// Every `row_count` `reduce_encoded` was handed, in call order. + static REDUCE_ROW_COUNTS: RefCell> = const { RefCell::new(Vec::new()) }; + } + + /// [`RefusingNegate`] with an encoding-aware rewrite that declines, recording the row count it + /// was offered. + #[derive(Clone)] + struct ProbingNegate; + + impl RowFn for ProbingNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.probing_negate"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(RefusesNullTolerant,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + REDUCE_ROW_COUNTS.with_borrow_mut(|counts| counts.push(args[0].len())); + Ok(None) + } + } + + /// A `reduce_encoded` rewrite must be sized from the arrays it was handed, which under the + /// filter strategy hold the surviving rows rather than the whole batch. This is the only + /// mixed-mask path where the two differ, so nothing else would catch a rewrite sized from a + /// length captured elsewhere. + /// + /// This also pins the double probe as deliberate. The first call sees the original arrays at + /// full length, and is the only one that does; the second sees filtered, canonical copies. An + /// "optimization" that skipped the first because the batch will end up filtering would take an + /// encoding-aware rewrite away from every function whose sink cannot skip rows. + #[test] + fn reduce_encoded_is_probed_before_and_after_filtering() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + REDUCE_ROW_COUNTS.with_borrow_mut(Vec::clear); + + let result = apply( + ProbingNegate, + [PrimitiveArray::from_option_iter([Some(3i64), None, Some(5)]).into_array()], + &mut ctx, + )?; + + assert_eq!( + REDUCE_ROW_COUNTS.with_borrow(|counts| counts.clone()), + vec![3, 2], + "expected an unfiltered probe at the batch length, then a filtered one at the \ + surviving count", + ); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(-3i64), None, Some(-5)]), + &mut ctx + ); + Ok(()) + } + + /// The rule itself, at and around the threshold, without going through an execution. + #[rstest] + #[case::bulk_dense_mask(0, 99, 100, true)] + #[case::bulk_sparse_mask(0, 1, 100, true)] + #[case::one_decode_dense_mask(1, 99, 100, true)] + #[case::one_decode_at_threshold(1, 50, 100, true)] + #[case::one_decode_below_threshold(1, 49, 100, false)] + #[case::two_decodes_at_old_boolean_choice(2, 81, 100, false)] + #[case::two_decodes_dense_mask(2, 90, 100, true)] + fn selects_branch_per_the_measured_rule( + #[case] filtered_decode_cost: usize, + #[case] true_count: usize, + #[case] len: usize, + #[case] expect_branch: bool, + ) { + let valid = Mask::from_indices(len, 0..true_count); + assert_eq!( + branch_beats_filter(filtered_decode_cost, &valid), + expect_branch, + ); + } + + #[test] + fn planning_adds_decode_cost_across_arguments() { + assert_eq!( + RowPolicy::for_dispatch::<(TrackedI64<1>, TrackedI64<1>), ()>(), + RowPolicy::ValidOnly { + filtered_decode_cost: 2 + } + ); + } +} diff --git a/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs b/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs new file mode 100644 index 00000000000..50ea77ef015 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests that row output dtypes cannot introduce their own nulls. + +use super::*; +use crate::dtype::Nullability; +use crate::dtype::PType; + +#[derive(Clone)] +struct NullableI64(i64); + +impl OutputElement for NullableI64 { + fn element_dtype() -> DType { + DType::Primitive(PType::I64, Nullability::Nullable) + } + + fn build(values: Vec) -> ArrayRef { + PrimitiveArray::from_option_iter(values.into_iter().map(|value| Some(value.0))).into_array() + } + + fn placeholder() -> Self { + Self(0) + } +} + +struct NullableSink(usize); + +impl OutputSink for NullableSink { + type Rows<'a> = usize; + type Row<'a> = (); + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::Primitive(PType::I64, Nullability::Nullable)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self(rows)) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.0 + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + *rows == row_count + } + + fn row<'a>(_rows: &'a mut Self::Rows<'_>, _index: usize) -> Self::Row<'a> {} + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(PrimitiveArray::from_option_iter(Vec::>::new()).into_array()) + } +} + +#[derive(Clone)] +struct NullableElementFn; + +impl RowFn for NullableElementFn { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.nullable_element"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = NullableI64(value), + ) + } +} + +#[derive(Clone)] +struct NullableSinkFn; + +impl RowFn for NullableSinkFn { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.nullable_sink"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), NullableSink, _, _>(|_| (), |&(), _, ()| {}) + } +} + +#[test] +fn nullable_element_dtype_is_rejected() { + let input = DType::Primitive(PType::I64, Nullability::NonNullable); + let error = + ScalarFnVTable::return_dtype(&NullableElementFn, &EmptyOptions, &[input]).unwrap_err(); + + assert!(error.to_string().contains("non-nullable dtype"), "{error}"); +} + +#[test] +fn nullable_sink_dtype_is_rejected() { + let input = DType::Primitive(PType::I64, Nullability::NonNullable); + let error = ScalarFnVTable::return_dtype(&NullableSinkFn, &EmptyOptions, &[input]).unwrap_err(); + + assert!(error.to_string().contains("non-nullable dtype"), "{error}"); +} diff --git a/vortex-array/src/scalar_fn/row/tests/prepared.rs b/vortex-array/src/scalar_fn/row/tests/prepared.rs new file mode 100644 index 00000000000..64ed82c8d2c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/prepared.rs @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for preparing batch-constant state once before the row loop. + +use std::cell::Cell; + +use super::*; +use crate::validity::Validity; + +thread_local! { + /// Which operands the last `prepare` saw as constant, as a bitmask (bit 0 for `x`, bit 1 + /// for `y`). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; +} + +/// `sqrt(x^2 + y^2)` through [`RowVisitor::visit_prepared_into`]: the square of any constant +/// operand is hoisted out of the row loop, and recorded in [`SEEN_CONSTANTS`]. +#[derive(Clone)] +struct PreparedHypot; + +impl RowFn for PreparedHypot { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["x", "y"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.prepared_hypot"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(f64, f64), ElementSink, _, _>( + |(x, y)| { + SEEN_CONSTANTS.set(u8::from(x.is_some()) | (u8::from(y.is_some()) << 1)); + (x.map(|x| x * x), y.map(|y| y * y)) + }, + |&(x_sq, y_sq), (x, y), output| { + *output = (x_sq.unwrap_or(x * x) + y_sq.unwrap_or(y * y)).sqrt(); + }, + ) + } +} + +/// A constant operand reaches `prepare` as `Some`, and the result is identical to the same +/// value expanded into a full column, which reaches `prepare` as `None`. +#[test] +fn a_constant_operand_matches_its_expanded_column() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0, 8.0].into_array(); + + let constant = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + let from_constant = apply(PreparedHypot, [x.clone(), constant], &mut ctx)?; + assert_eq!(SEEN_CONSTANTS.get(), 0b10); + + let expanded = buffer![4.0f64, 4.0, 4.0].into_array(); + let from_expanded = apply(PreparedHypot, [x, expanded], &mut ctx)?; + assert_eq!(SEEN_CONSTANTS.get(), 0b00); + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// A masked constant (the same value in every row, some rows null, how the compressor spells +/// an all-same-with-nulls chunk) is a batch constant too: the wrapper carries only validity, +/// which the lifting owns, so `prepare` sees the child's value and the null rows stay +/// null in the result. +#[test] +fn a_masked_constant_operand_is_seen_as_constant() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0, 8.0].into_array(); + + let masked_constant = MaskedArray::try_new( + ConstantArray::new(Scalar::from(4.0f64), 3).into_array(), + Validity::from_iter([true, false, true]), + )? + .into_array(); + let result = apply(PreparedHypot, [x, masked_constant], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b10); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} + +/// With no constant operand every `ConstElems` slot is `None` and the loop computes exactly +/// what unit preparation would. +#[test] +fn all_varying_operands_prepare_nothing() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0].into_array(); + let y = buffer![4.0f64, 12.0].into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b00); + assert_arrays_eq!(result, PrimitiveArray::from_iter([5.0f64, 13.0]), &mut ctx); + Ok(()) +} + +/// Two constant operands are folded to a single-row execution by the lifting, and that +/// row still goes through `prepare`, seeing both constants. +#[test] +fn all_constant_operands_fold_and_still_prepare() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = ConstantArray::new(Scalar::from(3.0f64), 4).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 4).into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b11); + assert_arrays_eq!( + result, + PrimitiveArray::from_iter([5.0f64, 5.0, 5.0, 5.0]), + &mut ctx + ); + Ok(()) +} + +/// Null rows pass through the prepared path exactly as through unit preparation. +#[test] +fn nulls_propagate_through_the_prepared_path() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = PrimitiveArray::from_option_iter([Some(3.0f64), None, Some(8.0)]).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/sink.rs b/vortex-array/src/scalar_fn/row/tests/sink.rs new file mode 100644 index 00000000000..95b5e2df05f --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/sink.rs @@ -0,0 +1,375 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for row functions that write into a batch-wide output sink. + +use std::sync::Arc; + +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; + +use super::*; +use crate::arrays::FixedSizeListArray; +use crate::dtype::NativePType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::validity::Validity; + +/// Builds a `FixedSizeList` column, presenting each row as the `&mut [T]` slice to fill. +/// +/// Its element dtype comes from the input rather than from `T` alone, so it exercises +/// [`OutputSink::sink_dtype`] actually reading `args`. +struct SpreadSink { + dtype: DType, + rows: usize, + elements: BufferMut, +} + +impl OutputSink for SpreadSink { + type Rows<'a> = (&'a mut [T], usize); + type Row<'a> = &'a mut [T]; + + fn sink_dtype(args: &[DType]) -> VortexResult { + let element = args + .first() + .ok_or_else(|| vortex_err!("a spread sink takes its element dtype from its input"))?; + ::validate(element)?; + Ok(DType::FixedSizeList( + Arc::new(element.as_nonnullable()), + u32::try_from(W).vortex_expect("test width fits in u32"), + Nullability::NonNullable, + )) + } + + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult { + Ok(Self { + dtype: dtype.clone(), + rows, + elements: BufferMut::zeroed(rows * W), + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + (self.elements.as_mut_slice(), self.rows) + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.1 == row_count && row_count.checked_mul(W) == Some(rows.0.len()) + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> &'a mut [T] { + &mut rows.0[index * W..][..W] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + vortex_ensure_eq!( + self.dtype, + Self::sink_dtype(&[DType::Primitive(T::PTYPE, self.dtype.nullability())])?, + "the sink must build the dtype it named", + ); + Ok(FixedSizeListArray::try_new( + PrimitiveArray::new(self.elements.freeze(), Validity::NonNullable).into_array(), + u32::try_from(W).vortex_expect("test width fits in u32"), + Validity::NonNullable, + self.rows, + )? + .into_array()) + } +} + +/// A sink that reports a data-dependent error only after every row has been written. +struct NonNegativeSink { + values: BufferMut, +} + +struct NonNegativeRow<'a> { + value: &'a mut i64, +} + +impl NonNegativeRow<'_> { + fn write(self, value: i64) -> bool { + *self.value = value; + value < 0 + } +} + +impl OutputSink for NonNegativeSink { + const ERRORS_ARE_DEFERRED: bool = true; + + type Rows<'a> = &'a mut [i64]; + type Row<'a> = NonNegativeRow<'a>; + + fn sink_dtype(args: &[DType]) -> VortexResult { + let dtype = args + .first() + .ok_or_else(|| vortex_err!("a non-negative sink requires one input"))?; + ::validate(dtype)?; + Ok(DType::Primitive(PType::I64, Nullability::NonNullable)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self { + values: BufferMut::zeroed(rows), + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.values.as_mut_slice() + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + NonNegativeRow { + value: &mut rows[index], + } + } + + fn finish(self, error: DeferredError) -> VortexResult { + if error.occurred() { + vortex_bail!("negative output"); + } + Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) + } +} + +/// Broadcasts each input value across a fixed-size list row: `spread(x) == [x, x, x]`. +#[derive(Clone)] +struct Spread; + +impl RowFn for Spread { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.spread"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), SpreadSink, _, _>( + |_| (), + |&(), (x,), out| out.fill(x), + ) + } +} + +/// The same, but refusing negative inputs, so its row closure returns `VortexResult<()>`. +#[derive(Clone)] +struct SpreadNonNegative; + +impl RowFn for SpreadNonNegative { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.spread_non_negative"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), SpreadSink, _, _>( + |_| (), + |&(), (x,), out| { + if x < 0 { + vortex_bail!("negative input {x}"); + } + out.fill(x); + Ok(()) + }, + ) + } +} + +/// Writes infallibly and lets its output sink report the error after the loop. +#[derive(Clone)] +struct DeferredNonNegative; + +impl RowFn for DeferredNonNegative { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.deferred_non_negative"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), NonNegativeSink, _, _>( + |_| (), + |&(), (value,), output| output.write(value), + ) + } +} + +/// `SpreadSink`'s three-element rows, built from `values`. +fn spread_rows(values: impl IntoIterator>) -> VortexResult { + let values = values.into_iter().collect::>(); + let rows = values.len(); + let flat = values + .iter() + .flat_map(|value| [value.unwrap_or(0); 3]) + .collect::>(); + let validity = if values.iter().all(Option::is_some) { + Validity::NonNullable + } else { + Validity::from_iter(values.iter().map(Option::is_some)) + }; + + Ok(FixedSizeListArray::try_new( + PrimitiveArray::new(flat, Validity::NonNullable).into_array(), + 3, + validity, + rows, + )? + .into_array()) +} + +/// The output dtype is the sink's, with its width, and every row holds the written slice. +#[test] +fn writes_one_row_at_a_time() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![7i64, -2, 0].into_array(); + + let result = apply(Spread, [input], &mut ctx)?; + + assert_arrays_eq!(result, spread_rows([Some(7), Some(-2), Some(0)])?, &mut ctx); + Ok(()) +} + +/// A null input row is written densely and masked away afterwards, exactly as on the value path. +#[test] +fn nulls_are_masked_after_the_sink() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::from_option_iter([Some(7i64), None, Some(4)]).into_array(); + + let result = apply(Spread, [input], &mut ctx)?; + + assert!(result.dtype().is_nullable()); + assert_arrays_eq!(result, spread_rows([Some(7), None, Some(4)])?, &mut ctx); + Ok(()) +} + +/// A sink whose closure cannot fail is dense, while one whose closure can fail is valid-only. +#[test] +fn null_handling_follows_from_declared_fallibility() { + let args = [DType::Primitive(PType::I64, Nullability::NonNullable)]; + assert_eq!(policy(&Spread, &args), RowPolicy::Dense); + assert!(!ScalarFnVTable::is_fallible(&Spread, &EmptyOptions)); + + assert_eq!( + policy(&SpreadNonNegative, &args), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + assert!(ScalarFnVTable::is_fallible( + &SpreadNonNegative, + &EmptyOptions + )); +} + +/// An error from a writing closure aborts the batch rather than being written into the sink. +#[test] +fn a_failing_row_propagates() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, -5, 3].into_array(); + + let error = apply(SpreadNonNegative, [input], &mut ctx).unwrap_err(); + + assert!(error.to_string().contains("negative input -5"), "{error}"); + Ok(()) +} + +/// A sink may accumulate a failure while its row closure remains infallible. +#[test] +fn a_sink_can_defer_its_error_until_finish() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, -5, 3].into_array(); + + let error = apply(DeferredNonNegative, [input], &mut ctx).unwrap_err(); + + assert!(error.to_string().contains("negative output"), "{error}"); + Ok(()) +} + +/// A deferred failure behind a null triggers a valid-row retry and is then discarded. +#[test] +fn a_deferred_error_behind_a_null_is_ignored() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::new( + buffer![1i64, -5, 3], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let result = apply(DeferredNonNegative, [input], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(1i64), None, Some(3)]), + &mut ctx + ); + Ok(()) +} + +/// Being fallible, `SpreadNonNegative` is filtered, so its closure never sees the value behind a +/// null row. A negative payload there must therefore not raise. +#[test] +fn a_failing_row_is_never_reached_behind_a_null() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::new( + buffer![1i64, -5, 3], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let result = apply(SpreadNonNegative, [input], &mut ctx)?; + + assert_arrays_eq!(result, spread_rows([Some(1), None, Some(3)])?, &mut ctx); + Ok(()) +} + +/// The sink names its output dtype from the input, so a wrong input dtype is rejected at plan +/// time rather than producing a mis-typed column. +#[test] +fn the_sink_dtype_validates_its_input() { + let dtype = DType::Primitive(PType::F64, Nullability::NonNullable); + assert!(ScalarFnVTable::return_dtype(&Spread, &EmptyOptions, &[dtype]).is_err()); +} + +/// The width the sink declares is the width it builds, over the element dtype it read off the +/// input. +#[test] +fn the_return_dtype_is_the_sinks() -> VortexResult<()> { + let dtype = DType::Primitive(PType::I64, Nullability::NonNullable); + assert_eq!( + ScalarFnVTable::return_dtype(&Spread, &EmptyOptions, std::slice::from_ref(&dtype))?, + DType::FixedSizeList(Arc::new(dtype), 3, Nullability::NonNullable), + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/vtable.rs b/vortex-array/src/scalar_fn/row/vtable.rs new file mode 100644 index 00000000000..cff4831aa8f --- /dev/null +++ b/vortex-array/src/scalar_fn/row/vtable.rs @@ -0,0 +1,398 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Blanket scalar-function implementation and execution visitors for row functions. + +use std::marker::PhantomData; + +use vortex_error::VortexResult; +#[cfg(any(test, feature = "_test-harness"))] +use vortex_error::vortex_err; +use vortex_mask::Mask; +use vortex_session::VortexSession; + +use super::row_fn::RowFn; +use super::row_fn::RowVisitor; +use super::row_fn::private; +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::expr::Expression; +use crate::expr::union_child_validities; +use crate::scalar_fn::Arity; +use crate::scalar_fn::ChildName; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +#[cfg(any(test, feature = "_test-harness"))] +use crate::scalar_fn::NullStrategy; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::SinkResult; +#[cfg(any(test, feature = "_test-harness"))] +use crate::scalar_fn::VecExecutionArgs; +use crate::scalar_fn::row::execute::RowExecution; +use crate::scalar_fn::row::execute::execute_row_sink_branch; +use crate::scalar_fn::row::execute::execute_row_sink_prepared; +use crate::scalar_fn::row::execute::validate_row_sink; +use crate::scalar_fn::row::lift::Batch; +use crate::scalar_fn::row::lift::BatchPlan; +use crate::scalar_fn::row::lift::KernelArgs; +use crate::scalar_fn::row::lift::RowPolicy; +use crate::scalar_fn::row::lift::reconcile_return; + +/// Compile-time check that a dispatched `(A, S, R)` agrees with `F`'s public metadata. Evaluated by +/// monomorphizing +/// [`visit_prepared_into`](RowVisitor::visit_prepared_into), so even a dispatch arm that never runs +/// is checked. +const fn assert_dispatch_agrees() { + assert!( + A::ARITY == F::ARG_NAMES.len(), + "dispatch visited a tuple whose arity differs from RowFn::ARG_NAMES", + ); + // Dictionary pushdown treats an infallible function as safe to evaluate over values no code + // references, so every dispatch must fit the function-wide declaration. + assert!( + !A::DECODE_FALLIBLE || F::FALLIBLE, + "dispatch decoded fallibly without declaring RowFn::FALLIBLE", + ); + assert!( + !R::FALLIBLE || F::FALLIBLE, + "dispatch returned an error without declaring RowFn::FALLIBLE", + ); + assert!( + !R::DEFERRED || F::FALLIBLE, + "dispatch deferred an error without declaring RowFn::FALLIBLE", + ); + assert!( + S::ERRORS_ARE_DEFERRED == R::DEFERRED, + "a deferred-error sink and row closure must be used together", + ); +} + +/// The plan-time visit: validate the dtypes and derive execution from the concrete sink and row +/// closure selected by dispatch. +struct PlanRows<'a, F> { + args: &'a [DType], + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for PlanRows<'_, F> {} + +impl RowVisitor for PlanRows<'_, F> { + type Out = BatchPlan; + + fn visit_prepared_into( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult { + const { assert_dispatch_agrees::() }; + + Ok(BatchPlan { + sink_dtype: validate_row_sink::(self.args)?, + policy: RowPolicy::for_dispatch::(), + }) + } +} + +/// The run-time visit: decode every column once and run the row loop. +struct ExecuteRows<'a, 'b, F> { + args: &'a dyn ExecutionArgs, + + /// The sink dtype computed by the planning visit. + sink_dtype: &'a DType, + + ctx: &'b mut ExecutionCtx, + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for ExecuteRows<'_, '_, F> {} + +impl RowVisitor for ExecuteRows<'_, '_, F> { + type Out = RowExecution; + + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult { + const { assert_dispatch_agrees::() }; + execute_row_sink_prepared::( + self.args, + self.sink_dtype, + self.ctx, + prepare, + apply, + ) + } +} + +/// The run-time visit for the branch-and-skip null strategy: compute only the conjoined-valid +/// rows over unfiltered columns. +/// +/// `Ok(None)` means the visit cannot take that strategy because the sink cannot skip rows or an +/// argument has no null-tolerant decode, and the lifting falls back to the filter strategy. +struct ExecuteRowsBranch<'a, 'b, F> { + args: &'a dyn ExecutionArgs, + + /// The sink dtype computed by the planning visit. + sink_dtype: &'a DType, + + /// The conjoined validity, materialized by the lifting and guaranteed mixed. + valid: &'a Mask, + + ctx: &'b mut ExecutionCtx, + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for ExecuteRowsBranch<'_, '_, F> {} + +impl RowVisitor for ExecuteRowsBranch<'_, '_, F> { + type Out = Option; + + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult> { + const { assert_dispatch_agrees::() }; + execute_row_sink_branch::( + self.args, + self.sink_dtype, + self.valid, + self.ctx, + prepare, + apply, + ) + } +} + +/// The kernel the lifting runs: the encoding-aware rewrite if it answers, otherwise the row loop +/// over whichever arguments the lifting hands over. +fn execute_rows( + row_fn: &F, + options: &F::Options, + args: KernelArgs<'_>, + ctx: &mut ExecutionCtx, +) -> VortexResult { + if let Some(reduced) = row_fn.reduce_encoded(options, args.arrays, ctx)? { + return Ok(RowExecution::Output(reduced)); + } + + row_fn.dispatch( + options, + args.dtypes, + ExecuteRows:: { + args: args.execution, + sink_dtype: args.sink_dtype, + ctx, + row_fn: PhantomData, + }, + ) +} + +/// The branch-and-skip kernel: compute only the rows set in `valid`, over the unfiltered `args`. +/// +/// `Ok(None)` sends the batch to the filter strategy instead. +fn execute_rows_branch( + row_fn: &F, + options: &F::Options, + args: KernelArgs<'_>, + valid: &Mask, + ctx: &mut ExecutionCtx, +) -> VortexResult> { + // The encoding-aware rewrite runs before the row loop exactly as in [`execute_rows`]. Here it + // sees the original (unfiltered) encodings, and its full-length result is masked by the caller + // like any other branch result. + if let Some(reduced) = row_fn.reduce_encoded(options, args.arrays, ctx)? { + return Ok(Some(RowExecution::Output(reduced))); + } + + row_fn.dispatch( + options, + args.dtypes, + ExecuteRowsBranch:: { + args: args.execution, + sink_dtype: args.sink_dtype, + valid, + ctx, + row_fn: PhantomData, + }, + ) +} + +/// The batch facts for `row_fn` over `args`, derived from its dispatched elements and sink. +fn lift_batch<'a, F: RowFn>( + row_fn: &F, + options: &F::Options, + args: &'a dyn ExecutionArgs, +) -> VortexResult> { + Batch::new(RowFn::id(row_fn), args, |arg_dtypes| { + let plan = row_fn.dispatch( + options, + arg_dtypes, + PlanRows:: { + args: arg_dtypes, + row_fn: PhantomData, + }, + )?; + Ok(plan) + }) +} + +/// The nullable execution policy selected by one concrete dispatch. +#[cfg(test)] +pub(super) fn row_policy( + row_fn: &F, + options: &F::Options, + args: &[DType], +) -> VortexResult { + row_fn + .dispatch( + options, + args, + PlanRows:: { + args, + row_fn: PhantomData, + }, + ) + .map(|plan| plan.policy) +} + +/// Every [`RowFn`] is a [`ScalarFnVTable`], the row loop lifted by `Batch`. +/// +/// This impl is why a [`RowFn`] cannot also implement [`ScalarFnVTable`] itself: coherence forbids +/// the second impl. Nothing in tree needs to, since everything a row function can vary lives on +/// [`RowFn`]; mirror another [`ScalarFnVTable`] method onto it when something actually does. +impl ScalarFnVTable for F { + type Options = F::Options; + + fn id(&self) -> ScalarFnId { + RowFn::id(self) + } + + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + RowFn::serialize(self, options) + } + + fn deserialize(&self, metadata: &[u8], session: &VortexSession) -> VortexResult { + RowFn::deserialize(self, metadata, session) + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(F::ARG_NAMES.len()) + } + + fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { + ChildName::from(F::ARG_NAMES[child_idx]) + } + + /// The visited output element's dtype, widened to nullable iff any input is nullable, which is + /// what makes the strictness dtype contract hold by construction. + fn return_dtype(&self, options: &Self::Options, args: &[DType]) -> VortexResult { + let plan = self.dispatch( + options, + args, + PlanRows:: { + args, + row_fn: PhantomData, + }, + )?; + + let nullability = + plan.sink_dtype.nullability() | Nullability::from(args.iter().any(DType::is_nullable)); + Ok(plan.sink_dtype.with_nullability(nullability)) + } + + fn execute( + &self, + options: &Self::Options, + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Nullary functions have no input values that could be null, so there is nothing to lift. + if args.num_inputs() == 0 { + let result_dtype = ScalarFnVTable::return_dtype(self, options, &[])?; + let values = execute_rows( + self, + options, + KernelArgs { + execution: args, + arrays: &[], + dtypes: &[], + sink_dtype: &result_dtype, + }, + ctx, + )? + .into_result()?; + return reconcile_return(RowFn::id(self), &result_dtype, args.row_count(), values); + } + + lift_batch(self, options, args)?.execute( + |args, ctx| execute_rows(self, options, args, ctx), + |args, valid, ctx| execute_rows_branch(self, options, args, valid, ctx), + ctx, + ) + } + + /// Output sinks build an all-valid column, so a row kernel cannot turn a wholly non-null row into + /// a null and the output validity is exactly the conjunction of the inputs'. Letting a sink + /// produce nulls would invalidate this. + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + union_child_validities(expression) + } + + /// A row kernel maps a null input row to a null output row, and computes non-null outputs from + /// non-null inputs alone, which is exactly strictness. The lifting is what makes it true. + fn is_strict(&self, _options: &Self::Options) -> bool { + true + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + F::FALLIBLE + } +} + +/// Execute `row_fn` over `inputs` with a forced null strategy, bypassing the per-batch selection. +/// +/// A test and benchmark seam only, and the only way to name a strategy from outside: it is how the +/// two are compared and how their agreement is asserted. It skips the null-constant and +/// all-constant folds, so do not pass such inputs. Forcing [`NullStrategy::BranchAndSkip`] on a +/// dispatch with no branch execution is an error rather than a silent fallback to filtering. +#[cfg(any(test, feature = "_test-harness"))] +pub fn execute_row_fn_with_strategy( + row_fn: &F, + options: &F::Options, + inputs: Vec, + row_count: usize, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = VecExecutionArgs::new(inputs, row_count); + + lift_batch(row_fn, options, &args)? + .execute_with_strategy( + |args, ctx| execute_rows(row_fn, options, args, ctx), + |args, valid, ctx| execute_rows_branch(row_fn, options, args, valid, ctx), + strategy, + ctx, + )? + .ok_or_else(|| { + vortex_err!( + "{} has no branch-and-skip execution for these inputs", + RowFn::id(row_fn), + ) + }) +} diff --git a/vortex-array/src/scalar_fn/vtable.rs b/vortex-array/src/scalar_fn/vtable.rs index 5d3561ff039..e5b074e9cf3 100644 --- a/vortex-array/src/scalar_fn/vtable.rs +++ b/vortex-array/src/scalar_fn/vtable.rs @@ -361,7 +361,7 @@ impl ExecutionArgs for VecExecutionArgs { } } -#[derive(Clone, Debug, PartialEq, Eq, Hash)] +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] pub struct EmptyOptions; impl Display for EmptyOptions { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { From ae099e8909d387b881ed932689467955818349d6 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 02/12] Benchmark the row executor and the null strategies Progress towards #9128. `row_fn_executor` measures the derived execution loop against a hand-written columnar kernel of the same arithmetic, so the cost of going through `RowFn` is separated from the cost of the operation. `strict_validity` measures the three null strategies against each other across validity densities, which is the evidence behind the per-batch selection rule. `like` gains a pair. `like_per_row_repeated_patterns` carries one infix pattern on every row so the compile cache always hits, and `like_per_row_distinct_patterns` varies that pattern so it never hits. Both compile the same shape and match the same way, so the difference between them is compilation alone, which is what a kernel that cannot cache across rows pays. The existing `like_per_row_patterns` keeps its input, so its measurements stay comparable with develop. Signed-off-by: Connor Tsui Co-authored-by: Claude --- vortex-array/Cargo.toml | 8 + vortex-array/benches/like.rs | 44 ++- vortex-array/benches/row_fn_executor.rs | 419 ++++++++++++++++++++++++ vortex-array/benches/strict_validity.rs | 217 ++++++++++++ 4 files changed, 683 insertions(+), 5 deletions(-) create mode 100644 vortex-array/benches/row_fn_executor.rs create mode 100644 vortex-array/benches/strict_validity.rs diff --git a/vortex-array/Cargo.toml b/vortex-array/Cargo.toml index d00b811a387..2da7f59309f 100644 --- a/vortex-array/Cargo.toml +++ b/vortex-array/Cargo.toml @@ -129,6 +129,10 @@ harness = false name = "binary_ops" harness = false +[[bench]] +name = "row_fn_executor" +harness = false + [[bench]] name = "kleene_bool" harness = false @@ -282,3 +286,7 @@ harness = false [[bench]] name = "slice_dict_primitive" harness = false + +[[bench]] +name = "strict_validity" +harness = false diff --git a/vortex-array/benches/like.rs b/vortex-array/benches/like.rs index 68219724717..e83fae69b28 100644 --- a/vortex-array/benches/like.rs +++ b/vortex-array/benches/like.rs @@ -87,13 +87,9 @@ fn like_regex(bencher: Bencher) { bench_like(bencher, "h_llo%w%d", LikeOptions::default()); } -#[divan::bench] -fn like_per_row_patterns(bencher: Bencher) { +fn bench_per_row_patterns(bencher: Bencher, patterns: ArrayRef) { let session = vortex_array::array_session(); let array = strings(); - // A non-constant pattern child takes the per-row path; repeated patterns hit the - // compile cache. - let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "hello%")).into_array(); bencher .with_inputs(|| { ( @@ -109,6 +105,44 @@ fn like_per_row_patterns(bencher: Bencher) { .bench_values(|(array, mut ctx)| array.execute::(&mut ctx).unwrap()); } +#[divan::bench] +fn like_per_row_patterns(bencher: Bencher) { + // A non-constant pattern child takes the per-row path; repeated patterns hit the + // compile cache. + let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "hello%")).into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// The per-row path with the compile cache hit on every row, carrying the infix pattern that +/// [`like_per_row_distinct_patterns`] varies. Both compile the same shape and match the same way, +/// so the only difference between them is how often a pattern is compiled. +#[divan::bench] +fn like_per_row_repeated_patterns(bencher: Bencher) { + let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "%aaa%")).into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// The per-row path with the compile cache defeated: every row carries a distinct pattern of the +/// same shape, so each row pays one [`LikePattern`] compilation. +/// +/// Paired with [`like_per_row_repeated_patterns`] this isolates the cost of compiling a pattern from +/// the cost of matching against it, which is what any kernel that cannot cache across rows pays. +#[divan::bench] +fn like_per_row_distinct_patterns(bencher: Bencher) { + let patterns = VarBinViewArray::from_iter_str( + (0..ARRAY_SIZE).map(|i| format!("%{}%", distinct_trigram(i))), + ) + .into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// A distinct three-letter lowercase infix per row, so `ARRAY_SIZE` rows never repeat a pattern +/// while every pattern keeps the same shape and compiles the same way. +fn distinct_trigram(i: usize) -> String { + let letter = |shift: usize| char::from(b'a' + u8::try_from((i >> shift) % 26).unwrap()); + [letter(0), letter(5), letter(10)].iter().collect() +} + #[divan::bench] fn ilike_contains(bencher: Bencher) { bench_like( diff --git a/vortex-array/benches/row_fn_executor.rs b/vortex-array/benches/row_fn_executor.rs new file mode 100644 index 00000000000..c66865ef26d --- /dev/null +++ b/vortex-array/benches/row_fn_executor.rs @@ -0,0 +1,419 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Compares cheap primitive row functions with the specialized columnar implementation. + +#![expect(clippy::unwrap_used)] + +use std::mem::MaybeUninit; +use std::sync::LazyLock; + +use divan::Bencher; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::array_session; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::DeferredError; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::OutputSink; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_buffer::BufferMut; +use vortex_error::VortexResult; +use vortex_error::vortex_err; +use vortex_session::VortexSession; +use vortex_session::registry::CachedId; + +const ROWS: usize = 65_536; + +static SESSION: LazyLock = LazyLock::new(array_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +#[derive(Clone)] +struct RowWrappingAdd; + +impl RowFn for RowWrappingAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_wrapping_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs.wrapping_add(rhs), + ) + } +} + +#[derive(Clone)] +struct RowCheckedAdd; + +struct CheckedAddSink { + values: BufferMut, + row_count: usize, +} + +struct CheckedAddRows<'a> { + values: &'a mut [MaybeUninit], +} + +struct CheckedAddRow<'a> { + value: &'a mut MaybeUninit, +} + +impl CheckedAddRow<'_> { + fn write(self, lhs: i64, rhs: i64) -> bool { + let value = lhs.wrapping_add(rhs); + let error = (lhs ^ value) & (rhs ^ value); + self.value.write(value); + error < 0 + } +} + +impl OutputSink for CheckedAddSink { + const ERRORS_ARE_DEFERRED: bool = true; + + type Rows<'a> = CheckedAddRows<'a>; + type Row<'a> = CheckedAddRow<'a>; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::from(i64::PTYPE)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self { + values: BufferMut::with_capacity(rows), + row_count: rows, + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + CheckedAddRows { + values: &mut self.values.spare_capacity_mut()[..self.row_count], + } + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.values.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + CheckedAddRow { + value: &mut rows.values[index], + } + } + + fn finish(mut self, error: DeferredError) -> VortexResult { + if error.occurred() { + return Err(vortex_err!("integer overflow in row checked add")); + } + + // SAFETY: dense execution writes every slot before `finish` is called. This sink does not + // support branch-and-skip, and filtered execution allocates exactly one slot per valid row. + unsafe { self.values.set_len(self.row_count) }; + Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) + } +} + +impl RowFn for RowCheckedAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_checked_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), CheckedAddSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| output.write(lhs, rhs), + ) + } +} + +struct I64Sink(BufferMut); + +impl OutputSink for I64Sink { + type Rows<'a> = &'a mut [i64]; + type Row<'a> = &'a mut i64; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::from(i64::PTYPE)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self(BufferMut::zeroed(rows))) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.0.as_mut_slice() + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + &mut rows[index] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(PrimitiveArray::new(self.0.freeze(), Validity::NonNullable).into_array()) + } +} + +#[derive(Clone)] +struct RowSinkWrappingAdd; + +impl RowFn for RowSinkWrappingAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_sink_wrapping_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), I64Sink, _, _>( + |_| (), + |&(), (lhs, rhs), out| { + *out = lhs.wrapping_add(rhs); + }, + ) + } +} + +fn inputs() -> (ArrayRef, ArrayRef) { + let lhs = (0..ROWS) + .map(|index| index as i64) + .collect::>() + .into_array(); + let rhs = (0..ROWS) + .map(|index| (index % 17) as i64) + .collect::>() + .into_array(); + (lhs, rhs) +} + +fn constant_inputs() -> (ArrayRef, ArrayRef) { + let (lhs, _) = inputs(); + let rhs = ConstantArray::new(Scalar::from(7i64), ROWS).into_array(); + (lhs, rhs) +} + +fn nullable_inputs() -> (ArrayRef, ArrayRef) { + let lhs = PrimitiveArray::new( + (0..ROWS).map(|index| index as i64).collect::>(), + Validity::from_iter((0..ROWS).map(|index| !index.is_multiple_of(5))), + ) + .into_array(); + let rhs = PrimitiveArray::new( + (0..ROWS) + .map(|index| (index % 17) as i64) + .collect::>(), + Validity::from_iter((0..ROWS).map(|index| !index.is_multiple_of(7))), + ) + .into_array(); + (lhs, rhs) +} + +fn bench_row_fn(bencher: Bencher, row_fn: F) +where + F: RowFn, +{ + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + row_fn + .clone() + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn specialized_checked_add(bencher: Bencher) { + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add(bencher: Bencher) { + bench_row_fn(bencher, RowWrappingAdd); +} + +#[divan::bench] +fn row_sink_wrapping_add(bencher: Bencher) { + bench_row_fn(bencher, RowSinkWrappingAdd); +} + +#[divan::bench] +fn handrolled_sink_wrapping_add(bencher: Bencher) { + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + let lhs = lhs + .execute::(&mut ctx) + .unwrap() + .into_buffer::(); + let rhs = rhs + .execute::(&mut ctx) + .unwrap() + .into_buffer::(); + let mut output = BufferMut::zeroed(ROWS); + for ((out, lhs), rhs) in output + .as_mut_slice() + .iter_mut() + .zip(lhs.as_slice()) + .zip(rhs.as_slice()) + { + *out = lhs.wrapping_add(*rhs); + } + PrimitiveArray::new(output.freeze(), Validity::NonNullable).into_array() + }); +} + +#[divan::bench] +fn row_checked_add(bencher: Bencher) { + bench_row_fn(bencher, RowCheckedAdd); +} + +#[divan::bench] +fn specialized_checked_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowWrappingAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_checked_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowCheckedAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn specialized_checked_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_checked_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowCheckedAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowWrappingAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} diff --git a/vortex-array/benches/strict_validity.rs b/vortex-array/benches/strict_validity.rs new file mode 100644 index 00000000000..c4e7ca77d8e --- /dev/null +++ b/vortex-array/benches/strict_validity.rs @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Benchmarks how the row lifting applies validity to a dense kernel. +//! +//! Both arms run the same kernel over the same nullable input and differ only in how the output's +//! nulls are restored: +//! +//! - `lazy` is what the lifting behind [`RowFn`] does: conjoin the input validities (itself lazy) +//! and hand the resulting boolean array straight to `mask`. +//! - `eager` is the strategy it replaced: materialize the conjunction into a `Mask`, copy it into a +//! `BitBuffer`, wrap that in a `BoolArray`, and mask with it. +//! +//! `chain` composes three of the same function, which is where a per-call materialization compounds. + +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] + +use std::sync::LazyLock; + +use divan::Bencher; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::array_session; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::PType; +use vortex_array::expr::Expression; +use vortex_array::expr::union_child_validities; +use vortex_array::scalar_fn::Arity; +use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_session::VortexSession; +use vortex_session::registry::CachedId; + +static SESSION: LazyLock = LazyLock::new(array_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +const SIZES: &[usize] = &[65536, 1 << 20]; + +/// The shared kernel: double every lane, ignoring validity. +fn doubled(input: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let input = input.clone().execute::(ctx)?; + let values: Buffer = input + .as_slice::() + .iter() + .map(|value| value.wrapping_mul(2)) + .collect(); + Ok(PrimitiveArray::new(values, Validity::NonNullable).into_array()) +} + +/// Dense row function: the lifting decides how validity is applied. +/// +/// Its [`reduce_encoded`](RowFn::reduce_encoded) answers with [`doubled`] before the row loop is +/// reached, so this arm runs exactly the kernel [`EagerDouble`] runs and the two differ only in +/// validity. The row closure below is what makes it a [`RowFn`] at all, and never executes. +#[derive(Clone)] +struct LazyDouble; + +impl RowFn for LazyDouble { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.lazy_double"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i32,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = value.wrapping_mul(2), + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + args: &[ArrayRef], + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + doubled(&args[0], ctx).map(Some) + } +} + +/// The same function, applying validity the way the adapter used to: materialize a mask first. +#[derive(Clone)] +struct EagerDouble; + +impl ScalarFnVTable for EagerDouble { + type Options = EmptyOptions; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.eager_double"); + *ID + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(1) + } + + fn child_name(&self, _options: &Self::Options, _child_idx: usize) -> ChildName { + ChildName::from("input") + } + + fn return_dtype(&self, _options: &Self::Options, args: &[DType]) -> VortexResult { + Ok(DType::Primitive(PType::I32, args[0].nullability())) + } + + fn execute( + &self, + _options: &Self::Options, + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let input = args.get(0)?; + let valid = input.validity()?.execute_mask(args.row_count(), ctx)?; + let values = doubled(&input, ctx)?; + + if valid.all_true() { + return values.cast(DType::Primitive(PType::I32, input.dtype().nullability())); + } + + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + values.mask(mask) + } + + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + union_child_validities(expression) + } + + fn is_strict(&self, _options: &Self::Options) -> bool { + true + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + false + } +} + +/// A nullable i32 column with array-backed validity (~10% nulls). +fn nullable_input(len: usize) -> ArrayRef { + PrimitiveArray::new( + (0..len as i32).collect::>(), + Validity::from_iter((0..len).map(|i| !i.is_multiple_of(10))), + ) + .into_array() +} + +fn bench_depth(bencher: Bencher, scalar_fn: V, len: usize, depth: usize) +where + V: ScalarFnVTable + Clone, +{ + bencher + .with_inputs(|| nullable_input(len)) + .bench_local_values(|input| { + let mut ctx = SESSION.create_execution_ctx(); + let mut array = input; + for _ in 0..depth { + array = scalar_fn + .clone() + .try_new_array(len, EmptyOptions, [array]) + .unwrap() + .into_array(); + } + array.execute::(&mut ctx).unwrap() + }); +} + +#[divan::bench(args = SIZES)] +fn lazy(bencher: Bencher, len: usize) { + bench_depth(bencher, LazyDouble, len, 1); +} + +#[divan::bench(args = SIZES)] +fn eager(bencher: Bencher, len: usize) { + bench_depth(bencher, EagerDouble, len, 1); +} + +#[divan::bench(args = SIZES)] +fn lazy_chain(bencher: Bencher, len: usize) { + bench_depth(bencher, LazyDouble, len, 3); +} + +#[divan::bench(args = SIZES)] +fn eager_chain(bencher: Bencher, len: usize) { + bench_depth(bencher, EagerDouble, len, 3); +} From b324f3e266774eb8ae79b44a464c2240bc87bebf Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 03/12] Port the numeric binary kernels to RowFn Progress towards #9128. `vortex.numeric` becomes a `RowFn` over a primitive pair, taking `NumericOperator` as its options, which deletes the hand-written null propagation, constant folding, and validity logic in `numeric/primitive.rs`. Checked arithmetic reports overflow as evidence the row closure returns, rather than as a comparison the caller re-derives. Decimal keeps its own columnar implementation. `PrimitiveOperand` was defined in `numeric/primitive.rs` and shared out of `numeric/mod.rs`. The port drops its numeric caller, so it moves into `compare/primitive.rs`, its only remaining user. `NumericOperator` gains `Hash`, which a `RowFn`'s options require. `map_checked_into` in `vortex-compute` loses its last caller with the port and is deleted. Also adds a `list_length` test pinning that a non-nullable fixed-size list keeps a constant result rather than materializing one `u64` per row, which is the reason `vortex.list.length` stays on `ScalarFnVTable`. Signed-off-by: Connor Tsui Co-authored-by: Claude --- .../typed_view/primitive/numeric_operator.rs | 2 +- .../scalar_fn/fns/binary/compare/primitive.rs | 69 +++- .../scalar_fn/fns/binary/numeric/checked.rs | 93 +---- .../src/scalar_fn/fns/binary/numeric/mod.rs | 8 +- .../scalar_fn/fns/binary/numeric/primitive.rs | 389 ++++-------------- .../src/scalar_fn/fns/binary/numeric/row.rs | 269 ++++++++++++ .../src/scalar_fn/fns/binary/numeric/tests.rs | 8 +- vortex-array/src/scalar_fn/fns/list_length.rs | 19 + vortex-compute/src/lane_kernels/map_into.rs | 73 ---- 9 files changed, 461 insertions(+), 469 deletions(-) create mode 100644 vortex-array/src/scalar_fn/fns/binary/numeric/row.rs diff --git a/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs b/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs index 6b389a0554b..d7bcc8d0b4c 100644 --- a/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs +++ b/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs @@ -10,7 +10,7 @@ use vortex_error::vortex_err; use crate::scalar_fn::fns::operators::Operator; -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] /// Binary element-wise operations. pub enum NumericOperator { /// Binary element-wise addition of two arrays or of two scalars. diff --git a/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs b/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs index 3bfb11a266e..58fb22ed73e 100644 --- a/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs +++ b/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs @@ -4,6 +4,7 @@ //! Native comparison of primitive arrays via bit-packing lane kernels. use vortex_buffer::BitBuffer; +use vortex_buffer::Buffer; use vortex_error::VortexResult; use vortex_error::vortex_bail; @@ -11,18 +12,20 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::IntoArray; use crate::arrays::BoolArray; +use crate::arrays::Constant; use crate::arrays::ConstantArray; +use crate::arrays::PrimitiveArray; use crate::dtype::DType; use crate::dtype::NativePType; use crate::dtype::Nullability; use crate::dtype::PType; use crate::match_each_native_ptype; use crate::scalar::Scalar; -use crate::scalar_fn::fns::binary::PrimitiveOperand; use crate::scalar_fn::fns::binary::compare::collect_bits; use crate::scalar_fn::fns::binary::compare::collect_zip_bits; use crate::scalar_fn::fns::binary::compare::compare_validity; use crate::scalar_fn::fns::operators::CompareOperator; +use crate::validity::Validity; /// Compare two primitive arrays of the same [`PType`]. /// @@ -128,3 +131,67 @@ fn compare_slice_constant(lhs: &[T], rhs: T, op: CompareOperator CompareOperator::Lte => collect_bits(lhs, |a: T| a.is_le(rhs)), } } + +/// A primitive binary-operator operand: a materialized buffer, a non-null constant, or an +/// all-null constant. +/// +/// Splitting the constant out of the buffer is what lets the lane kernels above hoist it into a +/// register instead of reading it back per lane. +enum PrimitiveOperand { + /// A decoded column, one value per row. + Array { + values: Buffer, + validity: Validity, + }, + + /// The same non-null value in every row. + Constant { + value: T, + len: usize, + validity: Validity, + }, + + /// A null in every row, carrying only the row count. + Null(usize), +} + +impl PrimitiveOperand { + fn try_new(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + if let Some(constant) = array.as_opt::() { + return Ok( + match constant.scalar().as_primitive().try_typed_value::()? { + Some(value) => Self::Constant { + value, + len: array.len(), + validity: if constant.scalar().dtype().is_nullable() { + Validity::AllValid + } else { + Validity::NonNullable + }, + }, + None => Self::Null(array.len()), + }, + ); + } + + let array = array.clone().execute::(ctx)?; + let validity = array.validity()?; + let values = array.into_buffer::(); + Ok(Self::Array { values, validity }) + } + + fn len(&self) -> usize { + match self { + Self::Array { values, .. } => values.len(), + Self::Constant { len, .. } | Self::Null(len) => *len, + } + } + + fn validity(&self) -> Validity { + match self { + Self::Array { validity, .. } => validity.clone(), + Self::Constant { validity, .. } => validity.clone(), + Self::Null(_) => Validity::AllInvalid, + } + } +} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs index afb709abe1c..47c7d1351b9 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Checked-lane execution for numeric kernels, driven by the shared +//! Checked-lane execution for the decimal kernels, driven by the shared //! `vortex-compute` lane kernels. - -use std::ops::BitOrAssign; +//! +//! The primitive widths do not come through here: they are computed one row at a time by +//! [`row`](super::row), which writes a value for every row and reduces failure as one bit rather +//! than scanning lanes. use vortex_buffer::Buffer; use vortex_buffer::BufferMut; @@ -13,34 +15,22 @@ use vortex_compute::lane_kernels::IndexedSourceExt; use vortex_mask::AllOr; use vortex_mask::Mask; -/// Evidence that a lane failed, anything other than [`Default`] meaning failure. -/// -/// `bool` is the ordinary choice; [`map_checked_into`] explains why the wider members exist and -/// asserts the width bound that membership here does **not** imply. -/// -/// [`map_checked_into`]: IndexedSourceExt::map_checked_into -pub(super) trait Failure: Copy + Default + PartialEq + BitOrAssign {} - -impl Failure for bool {} -impl Failure for u8 {} -impl Failure for u16 {} -impl Failure for u32 {} -impl Failure for u64 {} - /// Apply the fallible `apply` over every lane of `source`, returning -/// `Err(first_failing_valid_lane)` only when it returns `None` on a _valid_ lane. +/// `Err(first_failing_valid_lane)` only when it returns `None` on a valid lane. /// /// `apply` also runs on invalid lanes, whose failures are masked out and whose values are /// unspecified, so it must be total: no panics or side effects on any stored lane value. /// -/// This drives the one-pass early-exit kernels, which abort at the end of the enclosing 64-lane -/// chunk. Use it when per-lane failure handling is cheap relative to the operation, as in integer -/// division. Prefer [`checked_apply_lanes`] when the failure check itself vectorizes. +/// This drives the one-pass early-exit kernels: failures abort at the end of the enclosing +/// 64-lane chunk. It suits an operation whose per-lane failure handling is cheap relative to the +/// operation itself, which is what the decimal kernels and their per-lane casts are. /// -/// `#[inline]`: this must inline into the caller that builds the closure, so that a captured -/// constant operand flattens into a register rather than living behind a pointer the loop reloads -/// on every lane, which blocks vectorization. -#[inline] +/// `#[inline(always)]`: this wrapper and its kernel calls must inline into the caller that +/// constructs the closure, so the closure environment (e.g. a captured constant operand) +/// flattens into registers. Left to its own devices under `codegen-units > 1`, the compiler +/// keeps the environment behind a pointer, and reloading a captured constant on every lane +/// blocks vectorization of the whole loop. +#[inline(always)] pub(super) fn checked_lanes( source: S, valid_rows: &Mask, @@ -61,7 +51,6 @@ where }; let mut values = BufferMut::::with_capacity(len); - let out = &mut values.spare_capacity_mut()[..len]; match valid_bits { None => source.try_map_into(out, apply)?, @@ -73,55 +62,3 @@ where Ok(values.freeze()) } - -/// Apply the split value/failure `apply` over every lane of `source`, returning -/// `Err(first_failing_valid_lane)` only when it flags a _valid_ lane. -/// -/// The hot pass writes every value unconditionally and OR-reduces one piece of evidence, leaving -/// the loop free of per-lane selects and per-chunk exit branches. Only if a lane flagged does a -/// cold second pass re-run `apply` through the early-exit kernels, which drop null-lane failures -/// and attribute the first valid one. -/// -/// Like [`checked_lanes`], `apply` runs on invalid lanes and must be total. -/// -/// `#[inline]`: see [`checked_lanes`]. -#[inline] -pub(super) fn checked_apply_lanes( - source: S, - valid_rows: &Mask, - mut apply: Apply, -) -> Result, usize> -where - S: IndexedSource + Copy, - T: Copy + Default, - Fail: Failure, - Apply: FnMut(S::Item) -> (T, Fail), -{ - let len = source.len(); - debug_assert_eq!(len, valid_rows.len()); - - let valid_bits = match valid_rows.bit_buffer() { - AllOr::All => None, - AllOr::None => return Ok(Buffer::zeroed(len)), - AllOr::Some(valid_bits) => Some(valid_bits), - }; - - let mut values = BufferMut::::with_capacity(len); - - let out = &mut values.spare_capacity_mut()[..len]; - if source.map_checked_into(out, &mut apply) != Fail::default() { - let mut checked = |item: S::Item| { - let (value, failure) = apply(item); - (failure == Fail::default()).then_some(value) - }; - match valid_bits { - None => source.try_map_into(out, &mut checked)?, - Some(valid_bits) => source.try_map_masked_into(valid_bits, out, &mut checked)?, - } - } - - // SAFETY: the kernels initialize every lane in `out`. - unsafe { values.set_len(len) }; - - Ok(values.freeze()) -} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs index 6622e08f82b..21db5dc8c8e 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs @@ -4,17 +4,21 @@ //! Native execution of the arithmetic operators (Add/Sub/Mul/Div) of the [`Binary`] scalar //! function. There is no Arrow fallback. //! +//! The primitive widths are computed by a [`RowFn`](crate::scalar_fn::RowFn), which owns null +//! handling, constants and validity for them; see [`row`]. Decimal keeps its own columnar +//! implementation in [`decimal`]. +//! //! [`Binary`]: super::Binary mod checked; mod decimal; mod primitive; +mod row; #[cfg(test)] mod tests; use decimal::execute_numeric_decimal; -pub(crate) use primitive::PrimitiveOperand; -use primitive::execute_numeric_primitive; +use row::execute_numeric_primitive; use vortex_error::VortexResult; use vortex_error::vortex_ensure; diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs index 357547f25b8..6de544eaaa2 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs @@ -1,65 +1,56 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use vortex_buffer::Buffer; -use vortex_compute::lane_kernels::IndexedSource; -use vortex_compute::lane_kernels::LaneZip; -use vortex_error::VortexResult; -use vortex_error::vortex_err; -use vortex_mask::Mask; - -use super::checked::Failure; -use super::checked::checked_apply_lanes; -use super::checked::checked_lanes; -use crate::ArrayRef; -use crate::ExecutionCtx; -use crate::IntoArray; -use crate::arrays::Constant; -use crate::arrays::ConstantArray; -use crate::arrays::PrimitiveArray; -use crate::builtins::ArrayBuiltins; -use crate::dtype::DType; +//! The checked arithmetic one row of a primitive column is computed with. +//! +//! Each operator is a type implementing [`CheckedPrimitiveOp`] at every native width, and each +//! width implements [`CheckedArithmetic`] with the value and failure evidence written separately. +//! Keeping them apart is what lets [`row`](super::row) write a value for every row and reduce the +//! evidence without a branch, so the loop vectorizes. + use crate::dtype::NativePType; -use crate::dtype::PType; use crate::dtype::half::f16; -use crate::match_each_native_ptype; -use crate::scalar::NumericOperator; -use crate::scalar::Scalar; -use crate::validity::Validity; +use crate::scalar_fn::SinkResult; -struct CheckedAdd; +/// Checked addition, failing on integer overflow. +pub(super) struct CheckedAdd; -struct CheckedSub; +/// Checked subtraction, failing on integer overflow. +pub(super) struct CheckedSub; -struct CheckedMul; +/// Checked multiplication, failing on integer overflow. +pub(super) struct CheckedMul; -struct CheckedDiv; +/// Checked division, failing on integer division by zero and on `MIN / -1`. +pub(super) struct CheckedDiv; -trait CheckedPrimitiveOp: Sized { - /// Message for the error raised when this operation fails on a valid lane. +/// Evidence that some row failed, in a form that OR-reduces across the batch. +/// +/// A plain `bool` is the obvious choice and the right one for most operations. Unsigned +/// multiplication is the exception: deriving a `bool` from the widened product costs a comparison, +/// and LLVM rewrites that comparison plus the product into `llvm.umul.with.overflow`, which has no +/// vector form and scalarizes the whole loop. Carrying the discarded high half instead means the row +/// never compares, so the multiply stays a widening vector multiply and the reduction stays a +/// vector OR. **The width must not exceed the element's**, or the reduction becomes the loop's +/// bottleneck instead of the arithmetic. +pub(super) trait Failure: SinkResult + Copy + Default {} + +impl + Copy + Default> Failure for T {} + +/// One arithmetic operator at one width, as a value and its failure evidence. +/// +/// The pair rather than an `Option` is what a row can write unconditionally: the value is stored +/// whatever the evidence says, and a failing row is either masked away as null or turned into a +/// batch error before anything reads it. +pub(super) trait CheckedPrimitiveOp: 'static + Sized { + /// The error reported for a batch in which some valid row failed. const ERROR: &'static str; - /// Whether to check inside the value loop and exit early, rather than reducing evidence over - /// the whole batch and re-running only once some lane has flagged. - const CHECKED_VALUE_LOOP: bool = false; - - /// How this operation reports a failing lane. See [`Failure`]. + /// How this operation reports a failing row. See [`Failure`]. type Failure: Failure; - /// Compute a lane's value and its failure evidence together. - /// - /// Overflowing-style rather than `Option`, so the vectorizable kernels can write every - /// lane's value unconditionally and reduce the evidence separately. [`Self::checked`] is the - /// shape for the scalar and early-exit paths. + /// The result of this operation, paired with evidence of whether the row failed. fn apply(lhs: T, rhs: T) -> (T, Self::Failure); - - /// [`Self::apply`] folded into the `Option` that the early-exit kernels take. - #[inline(always)] - fn checked(lhs: T, rhs: T) -> Option { - let (value, failed) = Self::apply(lhs, rhs); - - (failed == Self::Failure::default()).then_some(value) - } } impl CheckedPrimitiveOp for CheckedAdd { @@ -97,12 +88,6 @@ impl CheckedPrimitiveOp for CheckedMul { impl CheckedPrimitiveOp for CheckedDiv { const ERROR: &'static str = "integer division by zero or overflow in checked div"; - // Integer division still lowers to scalar divides, so the split - // value/error-scan loop used to auto-vectorize add/sub/mul only adds a - // second full scan. Use the one-pass early-exit checked kernel for integer - // division, matching Arrow/Velox. Float division has no checked errors and - // stays on the split/vectorizable default path. - const CHECKED_VALUE_LOOP: bool = T::DIV_CHECKS_IN_VALUE_LOOP; type Failure = bool; @@ -116,207 +101,21 @@ impl CheckedPrimitiveOp for CheckedDiv { }; (value, failed) } - - #[inline(always)] - fn checked(lhs: T, rhs: T) -> Option { - lhs.div_checked(rhs) - } -} - -/// Execute a numeric operation between two primitive-typed arrays. -pub(super) fn execute_numeric_primitive( - lhs: &ArrayRef, - rhs: &ArrayRef, - op: NumericOperator, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let ptype = PType::try_from(lhs.dtype())?; - - match_each_native_ptype!(ptype, |T| { - match op { - NumericOperator::Add => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Sub => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Mul => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Div => execute_checked_typed::(lhs, rhs, ctx), - } - }) -} - -fn execute_checked_typed( - lhs: &ArrayRef, - rhs: &ArrayRef, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: NativePType, - Op: CheckedPrimitiveOp, - Scalar: From, - Scalar: From>, -{ - let result_dtype = lhs - .dtype() - .with_nullability(lhs.dtype().nullability() | rhs.dtype().nullability()); - let lhs = PrimitiveOperand::::try_new(lhs, ctx)?; - let rhs = PrimitiveOperand::::try_new(rhs, ctx)?; - let len = lhs.len(); - debug_assert_eq!(len, rhs.len()); - - let validity = lhs.validity().and(rhs.validity())?; - let valid_rows = validity.execute_mask(len, ctx)?; - - let values = match (&lhs, &rhs) { - ( - PrimitiveOperand::Array { values: lhs, .. }, - PrimitiveOperand::Array { values: rhs, .. }, - ) => checked_op_lanes::<_, T, Op>( - LaneZip::new(lhs.as_slice(), rhs.as_slice()), - &valid_rows, - |(lhs, rhs)| (lhs, rhs), - ), - ( - PrimitiveOperand::Array { values: lhs, .. }, - PrimitiveOperand::Constant { value: rhs, .. }, - ) => { - // Capture the constant by value so it stays hoisted out of the lane loop. - let rhs = *rhs; - checked_op_lanes::<_, T, Op>(lhs.as_slice(), &valid_rows, move |lhs| (lhs, rhs)) - } - ( - PrimitiveOperand::Constant { value: lhs, .. }, - PrimitiveOperand::Array { values: rhs, .. }, - ) => { - let lhs = *lhs; - checked_op_lanes::<_, T, Op>(rhs.as_slice(), &valid_rows, move |rhs| (lhs, rhs)) - } - ( - PrimitiveOperand::Constant { value: lhs, .. }, - PrimitiveOperand::Constant { value: rhs, .. }, - ) => { - let value = Op::checked(*lhs, *rhs) - .ok_or_else(|| vortex_err!(InvalidArgument: "{}", Op::ERROR))?; - return Ok(constant_result_array(value, len, &result_dtype)); - } - (PrimitiveOperand::Null(_), _) | (_, PrimitiveOperand::Null(_)) => Ok(Buffer::zeroed(len)), - } - .map_err(|_lane| vortex_err!(InvalidArgument: "{}", Op::ERROR))?; - - primitive_result_array::(values, validity, &result_dtype) } -/// Run `Op` over the lanes of `source` in the loop shape it declares: the one-pass early-exit -/// kernel when `Op::CHECKED_VALUE_LOOP` is set, and the split value/failure kernel otherwise. +/// The per-width arithmetic behind [`CheckedPrimitiveOp`], with each operation split into the value +/// it produces and whether producing it failed. /// -/// `#[inline]`: see [`checked_lanes`]. -#[inline] -fn checked_op_lanes( - source: S, - valid_rows: &Mask, - mut to_operands: impl FnMut(S::Item) -> (T, T), -) -> Result, usize> -where - S: IndexedSource + Copy, - T: NativePType, - Op: CheckedPrimitiveOp, -{ - if Op::CHECKED_VALUE_LOOP { - checked_lanes(source, valid_rows, |item| { - let (lhs, rhs) = to_operands(item); - Op::checked(lhs, rhs) - }) - } else { - checked_apply_lanes(source, valid_rows, |item| { - let (lhs, rhs) = to_operands(item); - Op::apply(lhs, rhs) - }) - } -} - -fn primitive_result_array( - values: Buffer, - validity: Validity, - dtype: &DType, -) -> VortexResult { - let array = PrimitiveArray::new(values, validity).into_array(); - if array.dtype() == dtype { - return Ok(array); - } - array.cast(dtype.clone()) -} - -fn constant_result_array(value: T, len: usize, dtype: &DType) -> ArrayRef -where - T: NativePType, - Scalar: From + From>, -{ - if dtype.is_nullable() { - ConstantArray::new(Some(value), len).into_array() - } else { - ConstantArray::new(value, len).into_array() - } -} - -/// A primitive binary-operator operand: a materialized buffer, a non-null constant, or an -/// all-null constant. -pub(crate) enum PrimitiveOperand { - Array { - values: Buffer, - validity: Validity, - }, - Constant { - value: T, - len: usize, - validity: Validity, - }, - Null(usize), -} - -impl PrimitiveOperand { - pub(crate) fn try_new(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { - if let Some(constant) = array.as_opt::() { - return Ok( - match constant.scalar().as_primitive().try_typed_value::()? { - Some(value) => Self::Constant { - value, - len: array.len(), - validity: if constant.scalar().dtype().is_nullable() { - Validity::AllValid - } else { - Validity::NonNullable - }, - }, - None => Self::Null(array.len()), - }, - ); - } - - let array = array.clone().execute::(ctx)?; - let validity = array.validity()?; - let values = array.into_buffer::(); - Ok(Self::Array { values, validity }) - } - - pub(crate) fn len(&self) -> usize { - match self { - Self::Array { values, .. } => values.len(), - Self::Constant { len, .. } | Self::Null(len) => *len, - } - } - - pub(crate) fn validity(&self) -> Validity { - match self { - Self::Array { validity, .. } => validity.clone(), - Self::Constant { validity, .. } => validity.clone(), - Self::Null(_) => Validity::AllInvalid, - } - } -} - -trait CheckedArithmetic: NativePType { - const DIV_CHECKS_IN_VALUE_LOOP: bool; - - /// How multiplication reports a failing lane, which is width-dependent in a way the other - /// operations are not. `impl_checked_unsigned!` and `impl_checked_signed!` say why each family - /// reports what it reports. +/// Every `_value` method **must** be total: it is called for rows behind nulls, whose operands are +/// arbitrary, so it may not panic or trap. Integer division is the one that needs care, and +/// [`CheckedDiv`] supplies the default instead of dividing when the divisor is rejected. +pub(super) trait CheckedArithmetic: NativePType { + /// How multiplication reports a failing row. + /// + /// `Self` for the unsigned widths that have a widening multiply, so the row can hand back the + /// discarded high half rather than comparing. `bool` everywhere else: the narrow signed widths + /// already vectorize through a two-sided range check, floats never overflow, and the 64-bit + /// widths use a full-width evidence word. type MulFailure: Failure; fn add_value(self, rhs: Self) -> Self; @@ -327,16 +126,10 @@ trait CheckedArithmetic: NativePType { fn mul_failure(self, rhs: Self) -> Self::MulFailure; fn div_value(self, rhs: Self) -> Self; fn div_error(self, rhs: Self) -> bool; - fn div_checked(self, rhs: Self) -> Option; } /// The integer arithmetic every width shares, given the two things that actually differ between -/// them: how multiplication reports a failing lane, and how add/sub/div detect one. -/// -/// Only `mul_failure` genuinely varies per width, so it is the macro's parameter and everything -/// else is written once. The `$mul_failure_ty` and body are supplied by the caller because the -/// choice between a widened high half and a `bool` is exactly the vectorization decision described -/// on [`Failure`]. +/// them: how multiplication reports a failing row, and how add/sub/div detect one. macro_rules! impl_checked_integer { ( $ty:ty, @@ -347,8 +140,6 @@ macro_rules! impl_checked_integer { = |$mf_lhs:ident, $mf_rhs:ident| $mul_failure:expr, ) => { impl CheckedArithmetic for $ty { - const DIV_CHECKS_IN_VALUE_LOOP: bool = true; - type MulFailure = $mul_failure_ty; #[inline(always)] @@ -395,19 +186,12 @@ macro_rules! impl_checked_integer { let ($div_lhs, $div_rhs) = (self, rhs); $div_error } - - #[inline(always)] - fn div_checked(self, rhs: Self) -> Option { - self.checked_div(rhs) - } } }; } -/// The unsigned widths. `add`, `sub` and `div` are written once here, and `widening_mul` covers -/// every width including the 64-bit one, which widens into `u128`: the discarded high half of the -/// widened product is the failure evidence, and costs none of the comparison LLVM folds into -/// `umul.with.overflow`. +/// The unsigned widths. The discarded high half of the widened product is the failure evidence, +/// and costs none of the comparison LLVM folds into `umul.with.overflow`. macro_rules! impl_checked_unsigned { ($ty:ty, widening_mul: $wide:ty) => { impl_checked_integer!( @@ -420,12 +204,8 @@ macro_rules! impl_checked_unsigned { }; } -/// The signed widths, on the same principle. `widening_mul` is the shorthand for the narrow widths, -/// whose two-sided range check over a wider product is not the shape LLVM folds into an overflow -/// intrinsic, so they vectorize while reporting a plain `bool`. The 64-bit width cannot: deriving a -/// `bool` there costs the comparison that scalarizes the loop, so `high_half_mul` hands back the -/// discarded high half as a word. Both arms derive every shift and bound from `$ty`, so -/// instantiating one at a new width cannot silently keep another width's constants. +/// The signed widths. The narrow widths report a two-sided range check as `bool`; the 64-bit width +/// reports the discarded high half as a word so deriving the evidence does not scalarize the loop. macro_rules! impl_checked_signed { ($ty:ty, widening_mul: $wide:ty) => { impl_checked_signed!($ty, mul_failure: bool = |lhs, rhs| { @@ -433,9 +213,6 @@ macro_rules! impl_checked_signed { product < <$ty>::MIN as $wide || product > <$ty>::MAX as $wide }); }; - // Zero exactly when the product fits: a signed multiply overflows iff the high half of the true - // product differs from the sign extension of the half that was kept, so the two XOR to zero on - // the lanes that fit. `tests::test_multiply_overflow_boundaries` pins the boundaries. ($ty:ty, high_half_mul: $wide:ty => $failure:ty) => { impl_checked_signed!($ty, mul_failure: #[expect( clippy::cast_possible_truncation, @@ -451,7 +228,7 @@ macro_rules! impl_checked_signed { ( $ty:ty, mul_failure: $(#[$mul_failure_attr:meta])* $mul_failure_ty:ty - = |$l:ident, $r:ident| $mul_failure:expr + = |$lhs:ident, $rhs:ident| $mul_failure:expr ) => { impl_checked_integer!( $ty, @@ -464,7 +241,7 @@ macro_rules! impl_checked_signed { ((lhs ^ rhs) & (lhs ^ value)) < 0 }, div_error: |lhs, rhs| rhs == 0 || (lhs == <$ty>::MIN && rhs == -1), - mul_failure: $(#[$mul_failure_attr])* $mul_failure_ty = |$l, $r| $mul_failure, + mul_failure: $(#[$mul_failure_attr])* $mul_failure_ty = |$lhs, $rhs| $mul_failure, ); }; } @@ -473,8 +250,6 @@ macro_rules! impl_checked_float { ($($ty:ty),+ $(,)?) => { $( impl CheckedArithmetic for $ty { - const DIV_CHECKS_IN_VALUE_LOOP: bool = false; - type MulFailure = bool; #[inline(always)] @@ -516,11 +291,6 @@ macro_rules! impl_checked_float { fn div_error(self, _rhs: Self) -> bool { false } - - #[inline(always)] - fn div_checked(self, rhs: Self) -> Option { - Some(self / rhs) - } } )+ }; @@ -539,34 +309,34 @@ impl_checked_float!(f16, f32, f64); #[cfg(test)] mod tests { use super::CheckedArithmetic; + use crate::scalar_fn::SinkResult; /// Values whose pairwise products are worth probing: the saturating boundaries, the sign-change /// pivots, and a spread of magnitudes that straddles the 64-bit split. const PROBES: &[i64] = &[ - 0, // - 1, // - -1, // - 2, // - -2, // - 3, // - i64::MIN, // - i64::MIN + 1, // - i64::MAX, // - i64::MAX - 1, // - 1 << 31, // - 1 << 32, // - 1 << 62, // - -(1 << 62), // - 0x7FFF_FFFF, // - -0x8000_0000, // + 0, + 1, + -1, + 2, + -2, + 3, + i64::MIN, + i64::MIN + 1, + i64::MAX, + i64::MAX - 1, + 1 << 31, + 1 << 32, + 1 << 62, + -(1 << 62), + 0x7FFF_FFFF, + -0x8000_0000, ]; - /// Every `mul_failure` impl is either a bit trick or a two-sided range check, so hold each - /// against the obvious reference: `reference` is the width's own `checked_mul`, whose `None` - /// _is_ the definition of overflow. + /// Every `mul_failure` implementation is either a bit trick or a two-sided range check, so + /// hold each against `checked_mul`, whose `None` is the definition of overflow. #[track_caller] fn assert_agrees_with_checked_mul(lhs: T, rhs: T, reference: Option) { - let failed = lhs.mul_failure(rhs) != ::default(); + let failed = ::occurred(lhs.mul_failure(rhs)); assert_eq!(failed, reference.is_none(), "{lhs:?} * {rhs:?}"); } @@ -578,14 +348,13 @@ mod tests { assert_agrees_with_checked_mul(lhs, rhs, lhs.checked_mul(rhs)); let (lhs, rhs) = (lhs as u64, rhs as u64); - assert_agrees_with_checked_mul(lhs, rhs, lhs.checked_mul(rhs)); } } } - /// The 8-bit widths are cheap enough to check exhaustively, which pins the shift of the - /// unsigned formula and the range check of the signed one against every product that exists. + /// The 8-bit widths are cheap enough to check exhaustively, pinning the unsigned shift and the + /// signed range check against every product that exists. #[test] fn mul_failure_agrees_with_checked_mul_exhaustively_at_8_bits() { for lhs in u8::MIN..=u8::MAX { diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs new file mode 100644 index 00000000000..aab36bf196b --- /dev/null +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs @@ -0,0 +1,269 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The primitive arithmetic operators as a [`RowFn`]. +//! +//! [`Binary`] keeps its ID, its options serialization, and its strictness, fallibility and validity +//! contracts, and delegates only the _execution_ of `Add`, `Sub`, `Mul` and `Div` over primitive +//! columns to [`NumericBinary`]. Delegation rather than conversion is what makes the port possible +//! at all: `Binary` also covers Kleene `And`/`Or`, which are not strict, and the six comparisons, +//! which are infallible, so no single [`RowFn`] can stand in for the whole function. +//! +//! [`NumericBinary`] is not registered and appears in no serialized expression. It is reached only +//! through the [`ScalarFnVTable::execute`] that the blanket [`RowFn`] implementation provides, so +//! it needs no rewrite rule, no ID in the registry, and no wire format of its own. +//! +//! Everything the previous hand-written implementation did around the arithmetic itself now comes +//! from the lifting: input decoding, the constant operand collapse, the all-constant fold, the +//! null-constant short circuit, output allocation, nullability widening, and masking. What is left +//! here is the per-type checked operation and the sink that carries its overflow bit. +//! +//! [`Binary`]: crate::scalar_fn::fns::binary::Binary + +use std::marker::PhantomData; +use std::mem::MaybeUninit; + +use vortex_buffer::BufferMut; +use vortex_error::VortexResult; +use vortex_error::vortex_err; +use vortex_session::registry::CachedId; + +use super::primitive::CheckedAdd; +use super::primitive::CheckedDiv; +use super::primitive::CheckedMul; +use super::primitive::CheckedPrimitiveOp; +use super::primitive::CheckedSub; +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::PrimitiveArray; +use crate::dtype::DType; +use crate::dtype::NativePType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::match_each_native_ptype; +use crate::scalar::NumericOperator; +use crate::scalar_fn::DeferredError; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::RowFn; +use crate::scalar_fn::RowVisitor; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::VecExecutionArgs; +use crate::validity::Validity; + +/// Execute a numeric operation between two primitive-typed arrays. +/// +/// The caller has already established that both operands are primitive, of the same type, and of +/// the same length. +pub(super) fn execute_numeric_primitive( + lhs: &ArrayRef, + rhs: &ArrayRef, + op: NumericOperator, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = VecExecutionArgs::new(vec![lhs.clone(), rhs.clone()], lhs.len()); + + ScalarFnVTable::execute(&NumericBinary, &op, &args, ctx) +} + +/// The four arithmetic operators of [`Binary`] over primitive columns, as one row function per +/// operator and width. +/// +/// [`Binary`]: crate::scalar_fn::fns::binary::Binary +#[derive(Clone)] +struct NumericBinary; + +impl RowFn for NumericBinary { + type Options = NumericOperator; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + /// Only the integer widths can overflow, and only integer division can divide by zero, but + /// fallibility is declared without input dtypes. The float widths are therefore covered by the + /// same `true`, which costs them nothing: a deferred error keeps the batch on the dense path. + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.numeric_binary"); + *ID + } + + fn dispatch( + &self, + op: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + let ptype = operand_ptype(args)?; + + match_each_native_ptype!(ptype, |T| { + match op { + NumericOperator::Add => visit_checked::(visitor), + NumericOperator::Sub => visit_checked::(visitor), + NumericOperator::Mul => visit_checked::(visitor), + NumericOperator::Div => visit_checked::(visitor), + } + }) + } +} + +/// The width both operands are read at. +/// +/// Only the left operand is inspected. `(T, T)` validates each argument against the chosen width, +/// so a right operand of a different type is rejected by the visit rather than here. +fn operand_ptype(args: &[DType]) -> VortexResult { + let lhs = args + .first() + .ok_or_else(|| vortex_err!("a numeric operator takes two operands, got none"))?; + + PType::try_from(lhs) +} + +/// Visit at two `T` columns, applying `Op` per row into the sink that defers its overflow bit. +/// +/// The const block enforces, at monomorphization time, the width rule stated on +/// [`Failure`](super::primitive::Failure): evidence wider than the element would make the +/// OR-reduction rather than the arithmetic decide how many rows fit in a vector. +fn visit_checked(visitor: V) -> VortexResult +where + T: NativePType, + Op: CheckedPrimitiveOp, + V: RowVisitor, +{ + const { + assert!( + size_of::() <= size_of::(), + "failure evidence must be no wider than the value, or it bounds the vector width" + ) + }; + + visitor.visit_prepared_into::<(T, T), CheckedSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| output.write(lhs, rhs), + ) +} + +/// The output column of one checked arithmetic batch, reporting failure once after the row loop. +/// +/// Deferring the failure is what keeps a fallible kernel on the dense path: every row writes a +/// value unconditionally and OR-reduces its failure evidence, so the loop holds no branch and no +/// `Result` discriminant. The lifting retries a nullable batch over only its valid rows if that +/// reduction is non-zero, which is what makes an overflow behind a null row invisible. +/// +/// The reduction lives in the sink rather than in the row closure's return type so that its width +/// is [`Op::Failure`](CheckedPrimitiveOp::Failure), the operator's choice, rather than one bit. That +/// is what lets unsigned multiplication report its discarded high half instead of a comparison, and +/// so stay vectorized. +/// +/// **The storage is deliberately uninitialized, not zeroed.** Substituting `BufferMut::zeroed` to +/// make the sink safe was measured at **1.65 to 1.71x** the cost of allocate-and-fill, stable across +/// two runs and every batch size from 8 KiB to 2 MiB, because `alloc_zeroed` does not avoid the +/// write: below glibc's mmap threshold `calloc` recycles a dirty chunk and memsets it, and above it +/// the first touch of each fresh page faults instead. The row loop overwrites every slot regardless, +/// so that pass is pure duplicate work on the hottest kernel in the system. This is the case the +/// repository's "avoid `unsafe` unless it is necessary" rule leaves room for: the safe spelling +/// exists, and it costs a second pass over the output. +/// +/// Rows are written into uninitialized storage, so this sink cannot finish a batch whose rows were +/// not all visited, and leaves [`OutputSink::SUPPORTS_SKIPPED_ROWS`] at `false`. Nothing is lost: +/// `SUPPORTS_SKIPPED_ROWS` is what makes branch-and-skip unavailable, which is the guard that keeps +/// the uninitialized slots sound. Note this is _not_ implied by the dispatch policy alone: a +/// deferred result still reaches the executor's valid-only policy whenever its arguments are not +/// dense-safe, so the `false` here is load-bearing rather than a restatement. +struct CheckedSink> { + /// The result values, initialized one row at a time up to `row_count`. + values: BufferMut, + + /// The batch length, which is the capacity `values` was allocated with. + row_count: usize, + + /// The operation applied to every row, which names the error reported by + /// [`finish`](OutputSink::finish). + op: PhantomData, +} + +/// The uninitialized output slots of a [`CheckedSink`], borrowed once for the row loop. +struct CheckedRows<'a, T: NativePType, Op: CheckedPrimitiveOp> { + values: &'a mut [MaybeUninit], + op: PhantomData, +} + +/// One output slot of a [`CheckedSink`]. +struct CheckedRow<'a, T: NativePType, Op: CheckedPrimitiveOp> { + value: &'a mut MaybeUninit, + op: PhantomData, +} + +impl> CheckedRow<'_, T, Op> { + /// Apply `Op` to one row, writing its value and handing back its failure evidence. + /// + /// The value is written whether or not the operation failed, since a failing row is either + /// masked away as null or turned into a batch error before it can be read. The evidence is + /// returned rather than reduced here so the executor can keep the reduction in a register, and + /// it is `Op`'s own width so the row never has to compare. + fn write(self, lhs: T, rhs: T) -> Op::Failure { + let (value, failure) = Op::apply(lhs, rhs); + self.value.write(value); + + failure + } +} + +impl> OutputSink for CheckedSink { + const ERRORS_ARE_DEFERRED: bool = true; + + type Rows<'a> + = CheckedRows<'a, T, Op> + where + Self: 'a; + type Row<'a> + = CheckedRow<'a, T, Op> + where + Self: 'a; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::Primitive(T::PTYPE, Nullability::NonNullable)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self { + values: BufferMut::with_capacity(rows), + row_count: rows, + op: PhantomData, + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + let row_count = self.row_count; + CheckedRows { + values: &mut self.values.spare_capacity_mut()[..row_count], + op: PhantomData, + } + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.values.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + CheckedRow { + value: &mut rows.values[index], + op: PhantomData, + } + } + + fn finish(mut self, error: DeferredError) -> VortexResult { + if error.occurred() { + return Err(vortex_err!(InvalidArgument: "{}", Op::ERROR)); + } + + // SAFETY: the sink reports `SUPPORTS_SKIPPED_ROWS = false`, so every path that reaches + // `finish` without an error has written all `row_count` slots: dense execution visits + // `0..row_count`, and the valid-row retry runs densely over a sink allocated for exactly + // the filtered rows. + unsafe { self.values.set_len(self.row_count) }; + + Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) + } +} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs index 7ee98ae717e..08d3e2daca9 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs @@ -297,13 +297,13 @@ fn test_multiply_overflow_on_null_lane_ignored( Ok(()) } -/// The hot pass OR-reduces evidence across whole 64-lane chunks before anything looks at it, so an -/// overflow in a late chunk must still be caught, and must still be suppressed when its lane is -/// null. Every other test here fits in a single chunk and cannot show either. +/// The hot pass OR-reduces evidence across the whole row loop before anything looks at it, so an +/// overflow late in the batch must still be caught, and must still be suppressed when its lane is +/// null. #[rstest] #[case::reported(true)] #[case::suppressed_by_null(false)] -fn test_multiply_overflow_survives_chunk_reduction( +fn test_multiply_overflow_survives_batch_reduction( #[case] lane_is_valid: bool, ) -> VortexResult<()> { const LEN: u32 = 1000; diff --git a/vortex-array/src/scalar_fn/fns/list_length.rs b/vortex-array/src/scalar_fn/fns/list_length.rs index 415a65416db..a971fe71141 100644 --- a/vortex-array/src/scalar_fn/fns/list_length.rs +++ b/vortex-array/src/scalar_fn/fns/list_length.rs @@ -350,6 +350,25 @@ mod tests { Ok(()) } + /// A non-nullable fixed-size list has one length for the whole column, so the result stays a + /// constant rather than materializing one `u64` per row. + #[test] + fn test_fixed_size_list_length_stays_constant() -> VortexResult<()> { + let fsl = create_fixed_size_list(Validity::NonNullable); + let mut ctx = array_session().create_execution_ctx(); + + let result = fsl + .apply(&list_length(root()))? + .execute::(&mut ctx)?; + + assert_eq!( + result.as_constant(), + Some(Scalar::primitive(2u64, Nullability::NonNullable)), + "expected a constant length column" + ); + Ok(()) + } + #[test] fn test_fixed_size_list_length_nullable() -> VortexResult<()> { let fsl = create_fixed_size_list(Validity::Array( diff --git a/vortex-compute/src/lane_kernels/map_into.rs b/vortex-compute/src/lane_kernels/map_into.rs index c1e1107b1b9..258913e9fc7 100644 --- a/vortex-compute/src/lane_kernels/map_into.rs +++ b/vortex-compute/src/lane_kernels/map_into.rs @@ -5,7 +5,6 @@ //! caller-provided `&mut [MaybeUninit]`. use std::mem::MaybeUninit; -use std::ops::BitOrAssign; use vortex_buffer::BitBuffer; @@ -218,58 +217,6 @@ pub trait IndexedSourceExt: IndexedSource + Sized { } } - /// Split value/failure map with **no validity awareness at all**: write every lane's value - /// unconditionally and OR-reduce its failure evidence into the return. - /// - /// The fastest checked shape, running at the speed of the unchecked [`map_into`] in exchange - /// for reporting only _that_ some lane failed and never exiting early. Re-run the now known - /// cold input through [`try_map_into`] or [`try_map_masked_into`] to attribute the failure or - /// to drop the null-lane ones. The evidence reduces inside the kernel because a captured `&mut` - /// becomes a loop-carried memory dependence that blocks vectorization. - /// - /// Anything other than [`Default`] means failure, and `bool` is the ordinary `Fail`. Wider - /// words exist for operations where deriving a `bool` costs the vectorization it guards. - /// **`Fail` must be no wider than `R`**, asserted below, or the reduction rather than the - /// operation decides how many lanes fit in a vector. - /// - /// [`map_into`]: IndexedSourceExt::map_into - /// [`try_map_into`]: IndexedSourceExt::try_map_into - /// [`try_map_masked_into`]: IndexedSourceExt::try_map_masked_into - /// - /// # Panics - /// - /// Panics if `out.len() != self.len()`. - #[inline] - fn map_checked_into(self, out: &mut [MaybeUninit], mut apply: Apply) -> Fail - where - Fail: Copy + Default + BitOrAssign, - Apply: FnMut(Self::Item) -> (R, Fail), - { - const { - assert!( - size_of::() <= size_of::(), - "failure evidence must be no wider than the value, or it bounds the vector width" - ) - }; - - let values = self; - let len = values.len(); - assert_eq!(out.len(), len, "out must have the same length as values"); - - let mut failed = Fail::default(); - for idx in 0..len { - // SAFETY: idx < len by the loop bound, and out.len() == len. - let val = unsafe { values.get_unchecked(idx) }; - - let (result, failure) = apply(val); - failed |= failure; - - // SAFETY: idx < len == out.len(). - unsafe { out.get_unchecked_mut(idx).write(result) }; - } - failed - } - /// Fallible map with **no validity awareness at all** — every `None` returned /// by the closure is treated as a failure, even at null lanes. /// @@ -599,26 +546,6 @@ mod tests { assert!(res.is_ok(), "null lane should bypass the range check"); } - #[test] - fn map_checked_into_writes_all_lanes_and_reduces_flag() { - let mut values: Vec = (0..130).collect(); - let mut out = vec![MaybeUninit::::uninit(); 130]; - let failed = values - .as_slice() - .map_checked_into(&mut out, |v| (v as u32, v > u32::MAX as u64)); - assert!(!failed); - assert_eq!(write_t(out), (0..130u32).collect::>()); - - values[77] = (u32::MAX as u64) + 1; - let mut out = vec![MaybeUninit::::uninit(); 130]; - let failed = values - .as_slice() - .map_checked_into(&mut out, |v| (v as u32, v > u32::MAX as u64)); - assert!(failed); - // Failing lanes still write their (wrapped) value. - assert_eq!(write_t(out)[76], 76); - } - #[test] fn map_bits_into_packs_full_and_remainder_words() { let values: Vec = (0..130).collect(); From aebe3caf772948c4e3290dbcb1680e1d03150830 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 04/12] Port the tensor scalar functions to RowFn Progress towards #9128. `vortex.tensor.l2_norm`, `vortex.tensor.inner_product`, and `vortex.tensor.cosine_similarity` become row functions over a `TensorRow` element, which yields a slice of the extension array's storage straight to the closure. Lifting supplies the null propagation, constant folding, nullability, validity, and options serde that the three kernels each wrote by hand. Cosine similarity hoists the norm of a broadcast query vector into `visit_prepared_into`'s prepare step. The prepared and per-row arms must agree bit for bit, which only holds while both accumulate in the same order, so `l2_norm_row` moves into `utils.rs` and both call it. `BinaryTensorOpMetadata` and `build_tensor_array` move there too, shared by the two binary operators and by the normalized encoding. Constant folding through `try_build_constant_normalized` is now derived from the row closure, so the export is gone. The tests move out of the three kernel modules and into `scalar_fns/tests/`. Signed-off-by: Connor Tsui Co-authored-by: Claude --- vortex-tensor/benches/cosine_similarity.rs | 8 +- vortex-tensor/benches/inner_product.rs | 8 +- vortex-tensor/benches/l2_norm.rs | 6 +- .../src/encodings/normalized/execute.rs | 22 +- vortex-tensor/src/encodings/normalized/mod.rs | 1 - .../src/scalar_fns/cosine_similarity.rs | 918 +++++------------- vortex-tensor/src/scalar_fns/inner_product.rs | 490 ++-------- vortex-tensor/src/scalar_fns/l2_norm.rs | 380 +------- vortex-tensor/src/scalar_fns/mod.rs | 4 + vortex-tensor/src/scalar_fns/row.rs | 131 +++ .../src/scalar_fns/tests/cosine_similarity.rs | 586 +++++++++++ .../src/scalar_fns/tests/inner_product.rs | 253 +++++ vortex-tensor/src/scalar_fns/tests/l2_norm.rs | 295 ++++++ vortex-tensor/src/scalar_fns/tests/mod.rs | 9 + vortex-tensor/src/scalar_fns/tests/row.rs | 113 +++ vortex-tensor/src/utils.rs | 224 +++-- vortex-tensor/src/vector_search.rs | 4 +- 17 files changed, 1882 insertions(+), 1570 deletions(-) create mode 100644 vortex-tensor/src/scalar_fns/row.rs create mode 100644 vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs create mode 100644 vortex-tensor/src/scalar_fns/tests/inner_product.rs create mode 100644 vortex-tensor/src/scalar_fns/tests/l2_norm.rs create mode 100644 vortex-tensor/src/scalar_fns/tests/mod.rs create mode 100644 vortex-tensor/src/scalar_fns/tests/row.rs diff --git a/vortex-tensor/benches/cosine_similarity.rs b/vortex-tensor/benches/cosine_similarity.rs index 6cc5eb867ef..fef94a0aa91 100644 --- a/vortex-tensor/benches/cosine_similarity.rs +++ b/vortex-tensor/benches/cosine_similarity.rs @@ -22,10 +22,12 @@ use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity; @@ -85,9 +87,9 @@ fn bench_cosine(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef) { bencher .with_inputs(|| { ( - CosineSimilarity::try_new_array(lhs.clone(), rhs.clone()) - .unwrap() - .into_array(), + CosineSimilarity + .try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/benches/inner_product.rs b/vortex-tensor/benches/inner_product.rs index 796e9b648d6..c0918f87ba2 100644 --- a/vortex-tensor/benches/inner_product.rs +++ b/vortex-tensor/benches/inner_product.rs @@ -20,6 +20,8 @@ use vortex_array::VortexSessionExecute; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::inner_product::InnerProduct; @@ -62,9 +64,9 @@ fn bench_inner_product(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef) { .counter(ItemsCount::new(lhs.len())) .with_inputs(|| { ( - InnerProduct::try_new_array(lhs.clone(), rhs.clone()) - .unwrap() - .into_array(), + InnerProduct + .try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/benches/l2_norm.rs b/vortex-tensor/benches/l2_norm.rs index 6f597084113..d96e4877af9 100644 --- a/vortex-tensor/benches/l2_norm.rs +++ b/vortex-tensor/benches/l2_norm.rs @@ -20,6 +20,8 @@ use vortex_array::VortexSessionExecute; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::l2_norm::L2Norm; @@ -60,7 +62,9 @@ fn bench_l2_norm(bencher: Bencher, input: ArrayRef) { .counter(ItemsCount::new(input.len())) .with_inputs(|| { ( - L2Norm::try_new_array(input.clone()).unwrap().into_array(), + L2Norm + .try_new_array(input.len(), EmptyOptions, [input.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/src/encodings/normalized/execute.rs b/vortex-tensor/src/encodings/normalized/execute.rs index 637c8c07117..b96a03113b7 100644 --- a/vortex-tensor/src/encodings/normalized/execute.rs +++ b/vortex-tensor/src/encodings/normalized/execute.rs @@ -14,7 +14,6 @@ use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; use vortex_array::arrays::fixed_size_list::FixedSizeListArraySlotsExt; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; -use vortex_array::dtype::NativePType; use vortex_array::match_each_float_ptype; use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::fns::operators::Operator; @@ -24,6 +23,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::matcher::AnyTensor; +use crate::utils::build_tensor_array; use crate::utils::extract_flat_elements; use crate::utils::unit_norm_tolerance; @@ -115,26 +115,6 @@ fn denormalize_constant_norms( Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) } -/// Rebuilds a tensor-like extension array from flat primitive elements. -fn build_tensor_array( - dtype: DType, - tensor_flat_size: usize, - row_count: usize, - validity: Validity, - elements: Buffer, -) -> VortexResult { - let list_size = - u32::try_from(tensor_flat_size).vortex_expect("tensor flat size must fit into `u32`"); - - // SAFETY: Tensor elements are always non-nullable, so the validity carries no length. - let elements = unsafe { PrimitiveArray::new_unchecked(elements, Validity::NonNullable) }; - - let storage = - FixedSizeListArray::try_new(elements.into_array(), list_size, validity, row_count)?; - - Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) -} - /// Returns the flattened element count of each row of a tensor-like extension dtype. fn tensor_flat_size(dtype: &DType) -> usize { dtype diff --git a/vortex-tensor/src/encodings/normalized/mod.rs b/vortex-tensor/src/encodings/normalized/mod.rs index 545236bba7d..2c9a72d2988 100644 --- a/vortex-tensor/src/encodings/normalized/mod.rs +++ b/vortex-tensor/src/encodings/normalized/mod.rs @@ -31,7 +31,6 @@ pub use array::NormalizedSlots; mod compress; pub use compress::NormalizedScheme; pub use compress::normalize; -pub(crate) use compress::try_build_constant_normalized; mod execute; diff --git a/vortex-tensor/src/scalar_fns/cosine_similarity.rs b/vortex-tensor/src/scalar_fns/cosine_similarity.rs index ca8fcf0efd4..e786d49b47c 100644 --- a/vortex-tensor/src/scalar_fns/cosine_similarity.rs +++ b/vortex-tensor/src/scalar_fns/cosine_similarity.rs @@ -1,48 +1,49 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Cosine similarity expression for tensor-like types. +//! Cosine similarity between two tensor columns. +use num_traits::Float; use num_traits::Zero; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::ScalarFnArray; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; +use vortex_array::dtype::NativePType; use vortex_array::match_each_float_ptype; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_array::serde::ArrayChildren; +use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::NormalizedOrientation; -use crate::encodings::normalized::try_build_constant_normalized; use crate::scalar_fns::inner_product::InnerProduct; use crate::scalar_fns::l2_norm::L2Norm; +use crate::scalar_fns::row::TensorRow; +#[cfg(test)] +use crate::scalar_fns::row::probe; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::BinaryTensorOpMetadata; use crate::utils::extract_normalized_children; -use crate::utils::validate_binary_tensor_float_inputs; +use crate::utils::l2_norm_row; /// Cosine similarity between two columns. /// /// Computes `dot(a, b) / (||a|| * ||b||)` over the flat backing buffer of each tensor or vector. /// The shape and permutation do not affect the result because cosine similarity only depends on the -/// element values, not their logical arrangement. +/// element values, not their logical arrangement. A zero norm on either side yields `0.0`. /// /// Both inputs must be tensor-like extension arrays ([`FixedShapeTensor`] or [`Vector`]) with the /// same dtype and a float element type. The output is a float column of the same float type. @@ -55,143 +56,79 @@ use crate::utils::validate_binary_tensor_float_inputs; /// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector /// [`Normalized`]: crate::encodings::normalized::Normalized -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct CosineSimilarity; -impl CosineSimilarity { - /// Creates a new [`TypedScalarFnInstance`] wrapping the cosine similarity operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(CosineSimilarity, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the cosine similarity between `lhs` and - /// `rhs`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(CosineSimilarity::new().erased(), vec![lhs, rhs]) - } -} - -impl ScalarFnVTable for CosineSimilarity { +impl RowFn for CosineSimilarity { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.cosine_similarity"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(2) + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("lhs"), - 1 => ChildName::from("rhs"), - _ => unreachable!("CosineSimilarity must have exactly two children"), - } + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let lhs = &arg_dtypes[0]; - let rhs = &arg_dtypes[1]; - - let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; - let ptype = tensor_match.element_ptype(); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow, TensorRow), ElementSink, _, _>( + |(lhs, rhs)| { + #[cfg(test)] + probe::record(lhs.is_some(), rhs.is_some()); + ConstNorms { + lhs: lhs.map(l2_norm_row), + rhs: rhs.map(l2_norm_row), + } + }, + |norms, (lhs, rhs), output| { + *output = cosine_similarity_row_prepared(norms, lhs, rhs); + }, + ) + }) } - fn execute( + /// [`Normalized`]-encoded operands make the *stored* norms and normalized children + /// authoritative: `cos(D(x, s), D(y, t)) = dot(x, y)` and `cos(D(x, s), y) = dot(x, y) / + /// ||y||`, in both cases forced to `0.0` on rows where any authoritative norm is `0.0` (even + /// for lossy children whose decoded coordinates are nonzero). + /// + /// [`Normalized`]: crate::encodings::normalized::Normalized + fn reduce_encoded( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, + args: &[ArrayRef], ctx: &mut ExecutionCtx, - ) -> VortexResult { - let mut lhs_ref = args.get(0)?; - let mut rhs_ref = args.get(1)?; - let len = args.row_count(); - - // If either side is a constant tensor-like extension array, eagerly normalize the single - // stored row and re-encode it as an `Normalized` whose children are both `ConstantArray`s. - // The `Normalized` fast path below then picks it up. - if let Some(normalized_array) = try_build_constant_normalized(&lhs_ref, len, ctx)? { - lhs_ref = normalized_array.into_array(); - } - if let Some(normalized_array) = try_build_constant_normalized(&rhs_ref, len, ctx)? { - rhs_ref = normalized_array.into_array(); - } + ) -> VortexResult> { + let lhs = args[0].clone(); + let rhs = args[1].clone(); - // Take any Normalized read-through fast path that applies. - match NormalizedOrientation::classify(&lhs_ref, &rhs_ref) { + match NormalizedOrientation::classify(&lhs, &rhs) { NormalizedOrientation::Both { lhs, rhs } => { - return self.execute_both_normalized(lhs, rhs, len, ctx); + cosine_both_normalized(lhs, rhs, ctx).map(Some) } NormalizedOrientation::One { normalized_array, plain, - } => { - return self.execute_one_normalized(normalized_array, plain, len, ctx); - } - NormalizedOrientation::Neither => {} + } => cosine_one_normalized(normalized_array, plain, ctx).map(Some), + NormalizedOrientation::Neither => Ok(None), } - - // Compute combined validity. - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - // Compute inner product and norms as columnar operations, and propagate the options. - let norm_lhs_arr = L2Norm::try_new_array(lhs_ref.clone())?; - let norm_rhs_arr = L2Norm::try_new_array(rhs_ref.clone())?; - let dot_arr = InnerProduct::try_new_array(lhs_ref, rhs_ref)?; - - // Execute to get the inner product and norms of the arrays. We only fully decompress - // because we need to perform special logic (guard against 0) during division. - let dot: PrimitiveArray = dot_arr.into_array().execute(ctx)?; - let norm_l: PrimitiveArray = norm_lhs_arr.into_array().execute(ctx)?; - let norm_r: PrimitiveArray = norm_rhs_arr.into_array().execute(ctx)?; - - // TODO(connor): Ideally we would have a `SafeDiv` binary numeric operation. - // TODO(connor): This can be written in a more SIMD-friendly manner. - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let norms_l = norm_l.as_slice::(); - let norms_r = norm_r.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - let denom = norms_l[i] * norms_r[i]; - - if denom == T::zero() { - T::zero() - } else { - dots[i] / denom - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - fn validity( - &self, - _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if either input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false } } @@ -221,578 +158,177 @@ impl ScalarFnArrayVTable for CosineSimilarity { } } -impl CosineSimilarity { - /// Both sides are [`Normalized`]-encoded: treat the normalized children as authoritative, so - /// `cosine_similarity = dot(n_l, n_r)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - fn execute_both_normalized( - &self, - lhs_ref: &ArrayRef, - rhs_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - let (normalized_l, norms_l) = extract_normalized_children(lhs_ref); - let (normalized_r, norms_r) = extract_normalized_children(rhs_ref); - - // `Normalized` makes the normalized children authoritative, so their dot product is the - // cosine similarity even for lossy storage wrappers, except that a zero stored norm still - // represents a zero vector. - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r)? - .into_array() - .execute(ctx)?; - let norms_l: PrimitiveArray = norms_l.execute(ctx)?; - let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let norms_l = norms_l.as_slice::(); - let norms_r = norms_r.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - if norms_l[i] == T::zero() || norms_r[i] == T::zero() { - T::zero() - } else { - dots[i] - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - /// One side is [`Normalized`]-encoded: treat the normalized child as authoritative, so - /// `cosine_similarity = dot(n, b) / ||b||`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - /// - /// The caller must pass the [`Normalized`] array as `normalized_ref` and the plain array as `plain_ref`. - fn execute_one_normalized( - &self, - normalized_ref: &ArrayRef, - plain_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = normalized_ref.validity()?.and(plain_ref.validity()?)?; - - let (normalized, normalized_norms) = extract_normalized_children(normalized_ref); - - let dot_arr = InnerProduct::try_new_array(normalized, plain_ref.clone())?; - let dot: PrimitiveArray = dot_arr.into_array().execute(ctx)?; - - let normalized_norms: PrimitiveArray = normalized_norms.execute(ctx)?; - - let norm_arr = L2Norm::try_new_array(plain_ref.clone())?; - let plain_norm: PrimitiveArray = norm_arr.into_array().execute(ctx)?; - - // TODO(connor): Ideally we would have a `SafeDiv` binary numeric operation. - // TODO(connor): This can be written in a more SIMD-friendly manner. - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let normalized_norms = normalized_norms.as_slice::(); - let plain_norms = plain_norm.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - if normalized_norms[i] == T::zero() || plain_norms[i] == T::zero() { - T::zero() - } else { - dots[i] / plain_norms[i] - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } +/// Per-batch state for the cosine row kernel: the L2 norm of each operand that is constant for +/// the batch. +/// +/// A broadcast query vector holds the same elements in every row, so its norm is the same in +/// every row too. Computing it in the prepare step hoists an `O(width)` pass and a `sqrt` per row +/// out of the row loop. `None` marks an operand that varies by row, whose norm the row closure +/// computes exactly as it did before the hoist. +struct ConstNorms { + /// The norm of the lhs when it is batch-constant. + lhs: Option, + + /// The norm of the rhs when it is batch-constant. + rhs: Option, } -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::encodings::normalized::Normalized; - use crate::scalar_fns::cosine_similarity::CosineSimilarity; - use crate::tests::SESSION; - use crate::types::vector::Vector; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::constant_tensor_array; - use crate::utils::test_helpers::normalized_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates cosine similarity between two tensor arrays and returns the result as `Vec`. - fn eval_cosine_similarity(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - #[test] - fn unit_vectors_1d() -> VortexResult<()> { - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // Tensor 1 - 0.0, 1.0, 0.0, // Tensor 2 - ], - )?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // Tensor 1 - 1.0, 0.0, 0.0, // Tensor 2 - ], - )?; - - // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - /// Single-row cosine similarity for various vector pairs. - #[rstest] - // Antiparallel -> -1.0. - #[case::opposite(&[3], &[1.0, 0.0, 0.0], &[-1.0, 0.0, 0.0], &[-1.0])] - // dot=24, both magnitudes=5 -> 24/25 = 0.96. - #[case::non_unit(&[2], &[3.0, 4.0], &[4.0, 3.0], &[0.96])] - // Zero vector -> guarded to 0.0. - #[case::zero_norm(&[2], &[0.0, 0.0], &[1.0, 0.0], &[0.0])] - fn single_row( - #[case] shape: &[usize], - #[case] lhs_elems: &[f64], - #[case] rhs_elems: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let lhs = tensor_array(shape, lhs_elems)?; - let rhs = tensor_array(shape, rhs_elems)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, expected); - Ok(()) - } - - /// Self-similarity across various tensor shapes should always produce 1.0. - #[rstest] - // 2x3 matrix, flattened to 6 elements. - #[case::matrix_2d( - &[2, 3], - &[ - 1.0, 0.0, 0.0, // row 0 - 0.0, 0.0, 0.0, // row 1 - ], - )] - // 2x2x2 tensor, 8 elements. - #[case::tensor_3d(&[2, 2, 2], &[1.0; 8])] - fn self_similarity(#[case] shape: &[usize], #[case] elements: &[f64]) -> VortexResult<()> { - let lhs = tensor_array(shape, elements)?; - let rhs = tensor_array(shape, elements)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); - Ok(()) - } - - #[test] - fn scalar_0d() -> VortexResult<()> { - // 0-dimensional tensor: each "tensor" is a single scalar value. - let lhs = tensor_array(&[], &[5.0, 3.0])?; - let rhs = tensor_array(&[], &[5.0, -3.0])?; - - // Same sign -> 1.0, opposite sign -> -1.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, -1.0]); - Ok(()) - } - - #[test] - fn many_rows() -> VortexResult<()> { - // 5 tensors of shape [4] compared against themselves -> all 1.0. - let lhs = tensor_array( - &[4], - &[ - 1.0, 2.0, 3.0, 4.0, // tensor 0 - 0.0, 1.0, 0.0, 0.0, // tensor 1 - 5.0, 0.0, 5.0, 0.0, // tensor 2 - 1.0, 1.0, 1.0, 1.0, // tensor 3 - 0.0, 0.0, 0.0, 7.0, // tensor 4 - ], - )?; - let rhs = lhs.clone(); - - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0, 1.0, 1.0, 1.0, 1.0], - ); - Ok(()) - } - - #[test] - fn constant_query_tensor() -> VortexResult<()> { - // Compare 4 tensors of shape [3] against a single constant query tensor [1,0,0]. - let data = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // tensor 0 - 0.0, 1.0, 0.0, // tensor 1 - 0.0, 0.0, 1.0, // tensor 2 - 1.0, 0.0, 0.0, // tensor 3 - ], - )?; - let query = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 4)?; - - assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); - Ok(()) - } - - #[test] - fn vector_unit_vectors() -> VortexResult<()> { - let lhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 0.0, 1.0, 0.0, // vector 1 - ], - )?; - let rhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 1.0, 0.0, 0.0, // vector 1 - ], - )?; - - // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - #[test] - fn vector_constant_query() -> VortexResult<()> { - let data = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 0.0, 1.0, 0.0, // vector 1 - 0.0, 0.0, 1.0, // vector 2 - 1.0, 0.0, 0.0, // vector 3 - ], - )?; - let query = Vector::constant_array(&[1.0, 0.0, 0.0], 4)?; - - assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 2 rows of dim-2 vectors. Row 1 of rhs is masked as null. - let lhs = tensor_array(&[2], &[3.0, 4.0, 1.0, 0.0])?; - let rhs = tensor_array(&[2], &[3.0, 4.0, 0.0, 1.0])?; - let rhs = MaskedArray::try_new(rhs, Validity::from_iter([true, false]))?.into_array(); - - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: self-similarity = 1.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[1.0]); - Ok(()) - } - - #[test] - fn both_normalized_self_similarity() -> VortexResult<()> { - // [3.0, 4.0] has norm 5.0, normalized [0.6, 0.8]. - // [1.0, 0.0] has norm 1.0, normalized [1.0, 0.0]. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - // Self-similarity should always be 1.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 1.0]); - Ok(()) - } - - #[test] - fn both_normalized_orthogonal() -> VortexResult<()> { - // [3.0, 0.0] normalized [1.0, 0.0], norm 3.0. - // [0.0, 4.0] normalized [0.0, 1.0], norm 4.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[1.0, 0.0], &[3.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.0, 1.0], &[4.0], &mut ctx)?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); - Ok(()) - } - - #[test] - fn both_normalized_zero_norm() -> VortexResult<()> { - // Zero-norm row: normalized is [0.0, 0.0], norm is 0.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - // Row 0: dot([0.6, 0.8], [0.6, 0.8]) = 1.0, row 1: dot([0,0], [1,0]) = 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lhs() -> VortexResult<()> { - // LHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // RHS is plain [3.0, 4.0]. - // cosine_similarity([3.0, 4.0], [3.0, 4.0]) = 1.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = tensor_array(&[2], &[3.0, 4.0])?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_rhs() -> VortexResult<()> { - // LHS is plain [1.0, 0.0], RHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // cosine_similarity([1.0, 0.0], [3.0, 4.0]) = 3.0 / (1.0 * 5.0) = 0.6. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = tensor_array(&[2], &[1.0, 0.0])?; - let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.6]); - Ok(()) - } - - #[test] - fn both_normalized_null_norms() -> VortexResult<()> { - // Row 0: valid, row 1: null (via nullable norms on rhs). - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - let normalized_r = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; - let norms_r = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); - let rhs = Normalized::try_new(normalized_r, norms_r, &mut ctx)?.into_array(); - - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[1.0]); - Ok(()) - } - - #[test] - fn both_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { - // Mimics a lossy encoding where the stored norm is authoritative but - // the decoded normalized child is physically nonzero. With a stored norm of `0.0`, cosine - // similarity for that row must be `0.0` even though the dot product of the normalized - // children is nonzero. - let normalized_l = tensor_array(&[2], &[0.6, 0.8])?; - let norms_l = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Intentionally violates the unit-norm invariant by pairing a nonzero normalized row - // with a stored norm of `0.0`, mimicking lossy storage. - // SAFETY: The children are structurally valid. - let lhs = unsafe { Normalized::new_unchecked(normalized_l, norms_l) }.into_array(); - - let normalized_r = tensor_array(&[2], &[0.6, 0.8])?; - let norms_r = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Same as above for the rhs operand. - // SAFETY: The children are structurally valid. - let rhs = unsafe { Normalized::new_unchecked(normalized_r, norms_r) }.into_array(); - - // `dot(normalized_l, normalized_r) = 1.0`, but the authoritative stored norms are both - // `0.0`, so cosine similarity must be `0.0`. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { - // Mimics a lossy encoding where the stored norm is authoritative but - // the decoded normalized child is physically nonzero. The plain side is a normal nonzero - // tensor with positive norm. cosine similarity must still be `0.0` because the - // authoritative stored norm on the normalized_array side is `0.0`. - let normalized = tensor_array(&[2], &[0.6, 0.8])?; - let norms = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Intentionally pairs a nonzero normalized row with a stored norm of `0.0`, mimicking - // lossy storage where the stored norm is authoritative. - // SAFETY: The children are structurally valid. - let normalized_array = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); - - let plain = tensor_array(&[2], &[1.0, 0.0])?; - - // Normalized encoding on the lhs: `One { normalized_array: lhs, plain: rhs }`. - assert_close( - &eval_cosine_similarity(normalized_array.clone(), plain.clone())?, - &[0.0], - ); - - // Normalized encoding on the rhs: `One { normalized_array: rhs, plain: lhs }`. The same - // zero-norm guard must fire regardless of operand order. - assert_close(&eval_cosine_similarity(plain, normalized_array)?, &[0.0]); - Ok(()) - } - - #[test] - fn constant_lhs_matches_plain_tensor() -> VortexResult<()> { - // The constant query `[1, 2, 2]` has norm 3, so its normalized form is `[1/3, 2/3, 2/3]`. - // Expected cosine similarity against each row is `dot([1, 2, 2], row) / (3 * ||row||)`. - let lhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // dot=1, ||rhs||=1, expected=1/3 - 1.0, 2.0, 2.0, // dot=9, ||rhs||=3, expected=1 - 0.0, 0.0, 1.0, // dot=2, ||rhs||=1, expected=2/3 - 2.0, 1.0, 2.0, // dot=8, ||rhs||=3, expected=8/9 - ], - )?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[test] - fn constant_rhs_matches_plain_tensor() -> VortexResult<()> { - // Mirror of `constant_lhs_matches_plain_tensor` with the constant on the right. - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // - 1.0, 2.0, 2.0, // - 0.0, 0.0, 1.0, // - 2.0, 1.0, 2.0, // - ], - )?; - let rhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[test] - fn both_constant_tensors() -> VortexResult<()> { - // `[1, 0, 0]` vs `[1, 1, 0]`. dot=1, ||lhs||=1, ||rhs||=sqrt(2), expected=1/sqrt(2). - let lhs = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 3)?; - let rhs = constant_tensor_array(&[3], &[1.0, 1.0, 0.0], 3)?; - let expected = 1.0 / 2.0_f64.sqrt(); - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[expected, expected, expected], - ); - Ok(()) - } - - #[test] - fn constant_zero_norm_query() -> VortexResult<()> { - // A zero-norm constant query must produce `0.0` for every row via the zero-norm guard in - // `execute_one_normalized` and `execute_both_normalized`. - let lhs = constant_tensor_array(&[3], &[0.0, 0.0, 0.0], 3)?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 2.0, 3.0, // - 4.0, 5.0, 6.0, // - 7.0, 8.0, 9.0, // - ], - )?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0, 0.0]); - Ok(()) - } - - #[test] - fn constant_self_similarity_nonunit() -> VortexResult<()> { - // A non-unit constant query compared to itself must produce `1.0`. This exercises the - // helper's division: after normalization, both sides must be exactly unit so the - // Normalized fast path's inner product yields 1. - let lhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; - let rhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0; 5]); - Ok(()) - } - - #[test] - fn vector_constant_matches_plain() -> VortexResult<()> { - // Exercise the `Vector` extension variant through the new pre-pass. - let lhs = Vector::constant_array(&[1.0, 2.0, 2.0], 4)?; - let rhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // - 1.0, 2.0, 2.0, // - 0.0, 0.0, 1.0, // - 2.0, 1.0, 2.0, // - ], - )?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[rstest] - #[case::vector(cosine_vector_lhs(), cosine_vector_rhs())] - #[case::fixed_shape_tensor(cosine_tensor_lhs(), cosine_tensor_rhs())] - fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { - let original = CosineSimilarity::try_new_array(lhs.clone(), rhs.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(CosineSimilarity); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("CosineSimilarity serialize must produce metadata"); - - let children = vec![lhs, rhs]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) +/// Computes the cosine similarity of one row, taking any hoisted norm from `norms` and computing +/// the rest exactly as [`cosine_similarity_row`] does. +/// +/// Each arm accumulates the same values in the same order as [`cosine_similarity_row`], and the +/// denominator keeps its lhs-times-rhs order, so the result is bit-identical whether a norm was +/// hoisted or not. The match costs one predictable branch per row: the arm is the same for the +/// whole batch. +fn cosine_similarity_row_prepared( + norms: &ConstNorms, + a: &[T], + b: &[T], +) -> T { + match (norms.lhs, norms.rhs) { + (None, None) => cosine_similarity_row(a, b), + (Some(norm_a), None) => { + let mut dot = T::zero(); + let mut norm_sq_b = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_b = norm_sq_b + y * y; + } + cosine_from_parts(dot, norm_a * norm_sq_b.sqrt()) + } + (None, Some(norm_b)) => { + let mut dot = T::zero(); + let mut norm_sq_a = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_a = norm_sq_a + x * x; + } + cosine_from_parts(dot, norm_sq_a.sqrt() * norm_b) + } + (Some(norm_a), Some(norm_b)) => { + let mut dot = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + } + cosine_from_parts(dot, norm_a * norm_b) + } } +} - fn cosine_vector_lhs() -> ArrayRef { - vector_array(3, &[1.0, 0.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") - } +/// Computes the cosine similarity of two equal-length float slices. +/// +/// Returns `dot(a, b) / (||a|| * ||b||)`, or `0.0` when either norm is zero. +fn cosine_similarity_row(a: &[T], b: &[T]) -> T { + let mut dot = T::zero(); + let mut norm_sq_a = T::zero(); + let mut norm_sq_b = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_a = norm_sq_a + x * x; + norm_sq_b = norm_sq_b + y * y; + } + + cosine_from_parts(dot, norm_sq_a.sqrt() * norm_sq_b.sqrt()) +} - fn cosine_vector_rhs() -> ArrayRef { - vector_array(3, &[0.0, 1.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +/// The shared tail of every cosine arm: `dot / denom`, guarded to `0.0` when the denominator is +/// zero. +fn cosine_from_parts(dot: T, denom: T) -> T { + if denom == T::zero() { + T::zero() + } else { + dot / denom } +} - fn cosine_tensor_lhs() -> ArrayRef { - tensor_array(&[2], &[1.0, 0.0, 3.0, 4.0]).expect("valid tensor array") - } +/// Both sides are [`Normalized`]-encoded: the normalized children are authoritative, so their dot +/// product is the cosine similarity, except that a row with a zero *stored* norm is a zero vector. +/// +/// Unlike [`InnerProduct::reduce_encoded`], which composes lazy `Mul` arrays over the norm columns, +/// this executes and materializes. The zero-norm guard is a conditional per row rather than an +/// arithmetic factor, so there is no lazy array that expresses it; the norm columns are one value +/// per row rather than one per coordinate, so materializing them is cheap next to the decode this +/// avoids. +/// +/// [`InnerProduct::reduce_encoded`]: InnerProduct::reduce_encoded +/// [`Normalized`]: crate::encodings::normalized::Normalized +fn cosine_both_normalized( + lhs: &ArrayRef, + rhs: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let len = lhs.len(); + let (normalized_l, norms_l) = extract_normalized_children(lhs); + let (normalized_r, norms_r) = extract_normalized_children(rhs); + + let dot: PrimitiveArray = InnerProduct + .try_new_array(len, EmptyOptions, [normalized_l, normalized_r])? + .execute(ctx)?; + let norms_l: PrimitiveArray = norms_l.execute(ctx)?; + let norms_r: PrimitiveArray = norms_r.execute(ctx)?; + + match_each_float_ptype!(dot.ptype(), |T| { + let dots = dot.as_slice::(); + let norms_l = norms_l.as_slice::(); + let norms_r = norms_r.as_slice::(); + // Zipped rather than indexed by `0..len`: one bounds check per iterator instead of three + // per row. A length disagreement between the children shortens the result, which the + // lifting reports against the batch row count rather than panicking mid-loop. + let buffer: Buffer = dots + .iter() + .zip(norms_l) + .zip(norms_r) + .map(|((&dot, &norm_l), &norm_r)| { + if norm_l.is_zero() || norm_r.is_zero() { + T::zero() + } else { + dot + } + }) + .collect(); + + Ok(PrimitiveArray::new(buffer, Validity::NonNullable).into_array()) + }) +} - fn cosine_tensor_rhs() -> ArrayRef { - tensor_array(&[2], &[0.0, 1.0, 3.0, 4.0]).expect("valid tensor array") - } +/// One side is [`Normalized`]-encoded: `cos = dot(normalized, plain) / ||plain||`, forced to `0.0` +/// on rows where the stored norm or the plain norm is `0.0`. +/// +/// [`Normalized`]: crate::encodings::normalized::Normalized +fn cosine_one_normalized( + normalized_array: &ArrayRef, + plain: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let len = normalized_array.len(); + let (normalized, normalized_norms) = extract_normalized_children(normalized_array); + + let dot: PrimitiveArray = InnerProduct + .try_new_array(len, EmptyOptions, [normalized, plain.clone()])? + .execute(ctx)?; + let normalized_norms: PrimitiveArray = normalized_norms.execute(ctx)?; + let plain_norm: PrimitiveArray = L2Norm + .try_new_array(len, EmptyOptions, [plain.clone()])? + .execute(ctx)?; + + match_each_float_ptype!(dot.ptype(), |T| { + let dots = dot.as_slice::(); + let normalized_norms = normalized_norms.as_slice::(); + let plain_norms = plain_norm.as_slice::(); + // Zipped for the same reason as [`cosine_both_normalized`]. + let buffer: Buffer = dots + .iter() + .zip(normalized_norms) + .zip(plain_norms) + .map(|((&dot, &stored_norm), &plain_norm)| { + if stored_norm.is_zero() || plain_norm.is_zero() { + T::zero() + } else { + dot / plain_norm + } + }) + .collect(); + + Ok(PrimitiveArray::new(buffer, Validity::NonNullable).into_array()) + }) } diff --git a/vortex-tensor/src/scalar_fns/inner_product.rs b/vortex-tensor/src/scalar_fns/inner_product.rs index 53ae82eb4a2..3d8255f3599 100644 --- a/vortex-tensor/src/scalar_fns/inner_product.rs +++ b/vortex-tensor/src/scalar_fns/inner_product.rs @@ -6,40 +6,30 @@ use num_traits::Float; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; +use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; use vortex_array::match_each_float_ptype; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::scalar_fn::fns::operators::Operator; use vortex_array::serde::ArrayChildren; -use vortex_buffer::Buffer; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::NormalizedOrientation; -use crate::matcher::AnyTensor; +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::BinaryTensorOpMetadata; -use crate::utils::extract_flat_elements; use crate::utils::extract_normalized_children; -use crate::utils::validate_binary_tensor_float_inputs; /// Inner product (dot product) between two columns. /// @@ -52,131 +42,82 @@ use crate::utils::validate_binary_tensor_float_inputs; /// /// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct InnerProduct; -impl InnerProduct { - /// Creates a new [`TypedScalarFnInstance`] wrapping the inner product operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(InnerProduct, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the inner product between `lhs` and - /// `rhs`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(InnerProduct::new().erased(), vec![lhs, rhs]) - } -} - -impl ScalarFnVTable for InnerProduct { +impl RowFn for InnerProduct { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.inner_product"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(2) + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("lhs"), - 1 => ChildName::from("rhs"), - _ => unreachable!("InnerProduct must have exactly two children"), - } + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let lhs = &arg_dtypes[0]; - let rhs = &arg_dtypes[1]; - - // TODO(connor): relax the float-only gate once integer tensors are supported. - let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; - let ptype = tensor_match.element_ptype(); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow, TensorRow), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = inner_product_row(lhs, rhs), + ) + }) } - fn execute( + /// [`Normalized`]-encoded operands factor through their stored norms: with `D(x, s)` denoting + /// `x * s` rowwise, `dot(D(x, s), D(y, t)) = s * t * dot(x, y)` and + /// `dot(D(x, s), y) = s * dot(x, y)`. The rewrite is expressed with lazy [`Operator::Mul`] + /// arrays over the (much smaller) norm columns, so no denormalized coordinates are decoded. + /// + /// [`Normalized`]: crate::encodings::normalized::Normalized + fn reduce_encoded( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let lhs_ref = args.get(0)?; - let rhs_ref = args.get(1)?; - let len = args.row_count(); + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let len = args[0].len(); - // Take any Normalized read-through fast path that applies. - match NormalizedOrientation::classify(&lhs_ref, &rhs_ref) { + Ok(match NormalizedOrientation::classify(&args[0], &args[1]) { NormalizedOrientation::Both { lhs, rhs } => { - return self.execute_both_normalized(lhs, rhs, len, ctx); + let (normalized_l, norms_l) = extract_normalized_children(lhs); + let (normalized_r, norms_r) = extract_normalized_children(rhs); + let dot = + InnerProduct.try_new_array(len, EmptyOptions, [normalized_l, normalized_r])?; + Some( + dot.binary(norms_l, Operator::Mul)? + .binary(norms_r, Operator::Mul)?, + ) } NormalizedOrientation::One { normalized_array, plain, } => { - return self.execute_one_normalized(normalized_array, plain, len, ctx); + let (normalized, norms) = extract_normalized_children(normalized_array); + let dot = + InnerProduct.try_new_array(len, EmptyOptions, [normalized, plain.clone()])?; + Some(dot.binary(norms, Operator::Mul)?) } - NormalizedOrientation::Neither => {} - } - - // Compute combined validity. - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - // Canonicalize so we can perform the math directly. - let lhs: ExtensionArray = lhs_ref.execute(ctx)?; - let rhs: ExtensionArray = rhs_ref.execute(ctx)?; - - // We validated that both inputs have the same type. - let ext = lhs.dtype().as_extension(); - let tensor_match = ext - .metadata_opt::() - .vortex_expect("we already validated this in `return_dtype`"); - let dimensions = tensor_match.list_size() as usize; - - // Extract the storage array from each extension input. We pass the storage (FSL) rather - // than the extension array to avoid canonicalizing the extension wrapper. - let lhs_storage = lhs.storage_array(); - let rhs_storage = rhs.storage_array(); - - let lhs_flat = extract_flat_elements(lhs_storage, dimensions, ctx)?; - let rhs_flat = extract_flat_elements(rhs_storage, dimensions, ctx)?; - - match_each_float_ptype!(lhs_flat.ptype(), |T| { - let buffer: Buffer = (0..len) - .map(|i| inner_product_row(lhs_flat.row::(i), rhs_flat.row::(i))) - .collect(); - - // SAFETY: The buffer length equals `row_count`, which matches the source validity - // length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) + NormalizedOrientation::Neither => None, }) } - - fn validity( - &self, - _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if either input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false - } } impl ScalarFnArrayVTable for InnerProduct { @@ -205,72 +146,6 @@ impl ScalarFnArrayVTable for InnerProduct { } } -impl InnerProduct { - /// Both sides are [`Normalized`]-encoded: `inner_product = s_l * s_r * dot(n_l, n_r)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - fn execute_both_normalized( - &self, - lhs_ref: &ArrayRef, - rhs_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - let (normalized_l, norms_l) = extract_normalized_children(lhs_ref); - let (normalized_r, norms_r) = extract_normalized_children(rhs_ref); - - let norms_l: PrimitiveArray = norms_l.execute(ctx)?; - let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r)? - .into_array() - .execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let nl = norms_l.as_slice::(); - let nr = norms_r.as_slice::(); - let buffer: Buffer = (0..len).map(|i| nl[i] * nr[i] * dots[i]).collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - /// One side is [`Normalized`]-encoded: `inner_product = s * dot(n, other)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - /// - /// The caller must pass the [`Normalized`] array as `normalized_ref` and the plain array as `plain_ref`. - fn execute_one_normalized( - &self, - normalized_ref: &ArrayRef, - plain_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = normalized_ref.validity()?.and(plain_ref.validity()?)?; - - let (normalized, norms) = extract_normalized_children(normalized_ref); - let normalized_norms: PrimitiveArray = norms.execute(ctx)?; - - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized, plain_ref.clone())? - .into_array() - .execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let ns = normalized_norms.as_slice::(); - let buffer: Buffer = (0..len).map(|i| ns[i] * dots[i]).collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } -} - /// Computes the inner product (dot product) of two equal-length float slices. /// /// Returns `sum(a_i * b_i)`. @@ -280,254 +155,3 @@ fn inner_product_row(a: &[T], b: &[T]) -> T { .map(|(&x, &y)| x * y) .fold(T::zero(), |acc, v| acc + v) } - -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::encodings::normalized::Normalized; - use crate::scalar_fns::inner_product::InnerProduct; - use crate::tests::SESSION; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::normalized_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates inner product between two tensor arrays and returns the result as `Vec`. - fn eval_inner_product(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - /// Single-row inner product for various vector pairs. - #[rstest] - // Orthogonal: [1, 0] . [0, 1] = 0. - #[case::orthogonal(&[2], &[1.0, 0.0], &[0.0, 1.0], &[0.0])] - // Parallel: [3, 4] . [3, 4] = 9 + 16 = 25. - #[case::parallel(&[2], &[3.0, 4.0], &[3.0, 4.0], &[25.0])] - // Antiparallel: [1, 2] . [-1, -2] = -1 + -4 = -5. - #[case::antiparallel(&[2], &[1.0, 2.0], &[-1.0, -2.0], &[-5.0])] - // Scaled: [2, 0] . [3, 0] = 6. - #[case::scaled(&[2], &[2.0, 0.0], &[3.0, 0.0], &[6.0])] - fn single_row( - #[case] shape: &[usize], - #[case] lhs_elems: &[f64], - #[case] rhs_elems: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let lhs = tensor_array(shape, lhs_elems)?; - let rhs = tensor_array(shape, rhs_elems)?; - assert_close(&eval_inner_product(lhs, rhs)?, expected); - Ok(()) - } - - #[test] - fn multiple_rows() -> VortexResult<()> { - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // tensor 0 - 3.0, 4.0, 0.0, // tensor 1 - 1.0, 1.0, 1.0, // tensor 2 - ], - )?; - let rhs = tensor_array( - &[3], - &[ - 0.0, 1.0, 0.0, // tensor 0: dot = 0 - 3.0, 4.0, 0.0, // tensor 1: dot = 25 - 2.0, 2.0, 2.0, // tensor 2: dot = 6 - ], - )?; - assert_close(&eval_inner_product(lhs, rhs)?, &[0.0, 25.0, 6.0]); - Ok(()) - } - - #[test] - fn vector_inner_product() -> VortexResult<()> { - let lhs = vector_array( - 2, - &[ - 3.0, 4.0, // vector 0 - 1.0, 0.0, // vector 1 - ], - )?; - let rhs = vector_array( - 2, - &[ - 3.0, 4.0, // vector 0: dot = 25 - 0.0, 1.0, // vector 1: dot = 0 - ], - )?; - assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 3 rows of dim-2 vectors. Row 1 of lhs is masked as null. - let lhs = tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; - let rhs = tensor_array(&[2], &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0])?; - let lhs = MaskedArray::try_new(lhs, Validity::from_iter([true, false, true]))?.into_array(); - - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: 1*7 + 2*8 = 23, row 1: null, row 2: 5*11 + 6*12 = 127. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert!(prim.is_valid(2, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[23.0]); - assert_close(&[prim.as_slice::()[2]], &[127.0]); - Ok(()) - } - - #[test] - fn rejects_non_extension_dtype() { - let lhs = PrimitiveArray::from_iter([1.0_f64, 2.0]).into_array(); - let rhs = PrimitiveArray::from_iter([3.0_f64, 4.0]).into_array(); - let result = InnerProduct::try_new_array(lhs, rhs); - assert!(result.is_err()); - } - - #[test] - fn rejects_mismatched_dtypes() -> VortexResult<()> { - let lhs = tensor_array(&[2], &[1.0_f64, 2.0])?; - let rhs = vector_array(2, &[3.0_f64, 4.0])?; - let result = InnerProduct::try_new_array(lhs, rhs); - assert!(result.is_err()); - Ok(()) - } - - #[test] - fn both_normalized() -> VortexResult<()> { - // LHS: [3.0, 4.0] = Normalized([0.6, 0.8], 5.0). - // RHS: [1.0, 0.0] = Normalized([1.0, 0.0], 1.0). - // dot([3.0, 4.0], [1.0, 0.0]) = 3.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[1.0, 0.0], &[1.0], &mut ctx)?; - - // Expected: 5.0 * 1.0 * dot([0.6, 0.8], [1.0, 0.0]) = 5.0 * 0.6 = 3.0. - assert_close(&eval_inner_product(lhs, rhs)?, &[3.0]); - Ok(()) - } - - #[test] - fn both_normalized_multiple_rows() -> VortexResult<()> { - // Row 0: [3.0, 4.0] dot [3.0, 4.0] = 25.0. - // Row 1: [1.0, 0.0] dot [0.0, 1.0] = 0.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0], &mut ctx)?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lhs() -> VortexResult<()> { - // LHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // RHS: plain [1.0, 2.0]. - // dot([3.0, 4.0], [1.0, 2.0]) = 3.0 + 8.0 = 11.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = tensor_array(&[2], &[1.0, 2.0])?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_rhs() -> VortexResult<()> { - // LHS: plain [1.0, 2.0]. - // RHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // dot([1.0, 2.0], [3.0, 4.0]) = 3.0 + 8.0 = 11.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = tensor_array(&[2], &[1.0, 2.0])?; - let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); - Ok(()) - } - - #[test] - fn both_normalized_null_norms() -> VortexResult<()> { - // Row 0: valid, row 1: null (via nullable norms on lhs). - let normalized_l = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; - let norms_l = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); - let mut ctx = SESSION.create_execution_ctx(); - - let lhs = Normalized::try_new(normalized_l, norms_l, &mut ctx)?.into_array(); - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: 5.0 * 5.0 * dot([0.6, 0.8], [0.6, 0.8]) = 25.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[25.0]); - Ok(()) - } - - #[rstest] - #[case::vector(inner_product_vector_lhs(), inner_product_vector_rhs())] - #[case::fixed_shape_tensor(inner_product_tensor_lhs(), inner_product_tensor_rhs())] - fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { - let original = InnerProduct::try_new_array(lhs.clone(), rhs.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(InnerProduct); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("InnerProduct serialize must produce metadata"); - - let children = vec![lhs, rhs]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) - } - - fn inner_product_vector_lhs() -> ArrayRef { - vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") - } - - fn inner_product_vector_rhs() -> ArrayRef { - vector_array(3, &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0]).expect("valid vector array") - } - - fn inner_product_tensor_lhs() -> ArrayRef { - tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0]).expect("valid tensor array") - } - - fn inner_product_tensor_rhs() -> ArrayRef { - tensor_array(&[2], &[5.0, 6.0, 7.0, 8.0]).expect("valid tensor array") - } -} diff --git a/vortex-tensor/src/scalar_fns/l2_norm.rs b/vortex-tensor/src/scalar_fns/l2_norm.rs index b7e9060ed3f..433a6527636 100644 --- a/vortex-tensor/src/scalar_fns/l2_norm.rs +++ b/vortex-tensor/src/scalar_fns/l2_norm.rs @@ -3,40 +3,23 @@ //! L2 norm expression for tensor-like types. -use num_traits::Float; use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::Constant; -use vortex_array::arrays::ConstantArray; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFn as ScalarFnArrayEncoding; -use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; -use vortex_array::dtype::NativePType; -use vortex_array::dtype::Nullability; use vortex_array::dtype::proto::dtype as pb; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; use vortex_array::match_each_float_ptype; -use vortex_array::scalar::Scalar; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_array::serde::ArrayChildren; -use vortex_buffer::Buffer; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure_eq; use vortex_error::vortex_err; @@ -44,9 +27,10 @@ use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::Normalized; -use crate::matcher::AnyTensor; -use crate::utils::extract_flat_elements; +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::extract_normalized_children; +use crate::utils::l2_norm_row; use crate::utils::validate_tensor_float_input; /// L2 norm (Euclidean norm) of a tensor or vector column. @@ -62,139 +46,62 @@ use crate::utils::validate_tensor_float_input; /// of the storage contract, not a separate lossy-compute mode. /// /// [`Normalized`]: crate::encodings::normalized::Normalized -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct L2Norm; -impl L2Norm { - /// Creates a new [`TypedScalarFnInstance`] wrapping the L2 norm operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(L2Norm, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the L2 norm over `child`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(child: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(L2Norm::new().erased(), vec![child]) - } -} - -impl ScalarFnVTable for L2Norm { +impl RowFn for L2Norm { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["input"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.l2_norm"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(1) - } - - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("input"), - _ => unreachable!("L2Norm must have exactly one child"), - } + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let input_dtype = &arg_dtypes[0]; - let tensor_match = validate_tensor_float_input(input_dtype)?; - let ptype = tensor_match.element_ptype(); - - let nullability = Nullability::from(input_dtype.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn execute( + fn dispatch( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let input_ref = args.get(0)?; - let row_count = args.row_count(); - - let ext = input_ref.dtype().as_extension(); - let tensor_match = ext - .metadata_opt::() - .vortex_expect("we already validated this in `return_dtype`"); - let tensor_flat_size = tensor_match.list_size() as usize; - let element_ptype = tensor_match.element_ptype(); - - let norm_dtype = DType::Primitive(element_ptype, ext.nullability()); - - // L2Norm over a `Normalized`-encoded column is defined to read back the authoritative stored - // norms. Callers of lossy encodings opt into that storage semantics instead of forcing a - // decode-and-recompute path here. - if input_ref.is::() { - let (_, norms) = extract_normalized_children(&input_ref); - vortex_ensure_eq!(norms.dtype(), &norm_dtype); - return Ok(norms); - } - - // Optimize for the constant array case. - if let Some(array) = input_ref.as_opt::() { - let scalar = array.scalar().as_extension().to_storage_scalar(); - - let Some(elements) = scalar.as_list().elements() else { - return Ok(ConstantArray::new(Scalar::null(norm_dtype), row_count).into_array()); - }; - - let norm_scalar = match_each_float_ptype!(element_ptype, |T| { - let values: Vec = elements - .iter() - .map(|s| { - s.as_primitive() - .as_::() - .vortex_expect("element was somehow not the correct float") - }) - .collect(); - let norm = l2_norm_row::(&values); - - Scalar::try_new(norm_dtype, Some(norm.into())) - })?; - - let norms = ConstantArray::new(norm_scalar, row_count).into_array(); - return Ok(norms); - } - - let input: ExtensionArray = input_ref.execute(ctx)?; - let validity = input.as_ref().validity()?; - - let storage = input.storage_array(); - let flat = extract_flat_elements(storage, tensor_flat_size, ctx)?; - - match_each_float_ptype!(flat.ptype(), |T| { - let buffer: Buffer = (0..row_count) - .map(|i| l2_norm_row(flat.row::(i))) - .collect(); - - // SAFETY: The buffer length equals `row_count`, which matches the source validity - // length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow,), ElementSink, _, _>( + |_| (), + |&(), (row,), output| *output = l2_norm_row(row), + ) }) } - fn validity( + /// `L2Norm` over a [`Normalized`]-encoded column is defined to read back the authoritative + /// stored norms. Callers of lossy encodings opt into that storage semantics instead of forcing + /// a decode-and-recompute path here. + fn reduce_encoded( &self, _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if the input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let input = &args[0]; + if !input.is::() { + return Ok(None); + } + let element_ptype = validate_tensor_float_input(input.dtype())?.element_ptype(); + let (_, norms) = extract_normalized_children(input); + vortex_ensure_eq!(norms.dtype().as_ptype(), element_ptype); + Ok(Some(norms)) } } @@ -240,206 +147,3 @@ impl ScalarFnArrayVTable for L2Norm { }) } } - -/// Computes the L2 norm (Euclidean norm) of a float slice. -/// -/// Returns `sqrt(sum(v_i^2))`. A zero-length or all-zero input produces `0.0`. -fn l2_norm_row(v: &[T]) -> T { - let mut sum_sq = T::zero(); - for &x in v { - sum_sq = sum_sq + x * x; - } - sum_sq.sqrt() -} - -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::EmptyMetadata; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::Constant; - use vortex_array::arrays::ConstantArray; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::dtype::DType; - use vortex_array::dtype::Nullability; - use vortex_array::dtype::PType; - use vortex_array::dtype::extension::ExtDType; - use vortex_array::scalar::Scalar; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::scalar_fns::l2_norm::L2Norm; - use crate::tests::SESSION; - use crate::types::vector::Vector; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::literal_vector_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates L2 norm on a tensor/vector array and returns the result as `Vec`. - fn eval_l2_norm(input: ArrayRef) -> VortexResult> { - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - #[rstest] - #[case::three_four_five(&[2], &[3.0, 4.0], &[5.0])] - #[case::zero_vector(&[3], &[0.0, 0.0, 0.0], &[0.0])] - #[case::single_element(&[1], &[7.0], &[7.0])] - #[case::negative_elements(&[2], &[-3.0, -4.0], &[5.0])] - fn known_norms( - #[case] shape: &[usize], - #[case] elements: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let arr = tensor_array(shape, elements)?; - assert_close(&eval_l2_norm(arr)?, expected); - Ok(()) - } - - #[test] - fn multiple_rows() -> VortexResult<()> { - let arr = tensor_array( - &[3], - &[ - 3.0, 4.0, 0.0, // norm = 5.0 - 0.0, 0.0, 0.0, // norm = 0.0 - 1.0, 1.0, 1.0, // norm = sqrt(3) - ], - )?; - assert_close(&eval_l2_norm(arr)?, &[5.0, 0.0, 3.0_f64.sqrt()]); - Ok(()) - } - - #[test] - fn vector_multiple_rows() -> VortexResult<()> { - let arr = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // norm = 1.0 - 3.0, 4.0, 0.0, // norm = 5.0 - ], - )?; - assert_close(&eval_l2_norm(arr)?, &[1.0, 5.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 2 rows of dim-2 vectors. Row 1 is masked as null. - let arr = tensor_array(&[2], &[3.0, 4.0, 0.0, 0.0])?; - let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![arr])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: norm = 5.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[5.0]); - Ok(()) - } - - /// A constant input whose scalar is a non-null tensor should short-circuit to a - /// [`ConstantArray`] output whose scalar is the precomputed norm. Uses [`execute_until`] so - /// execution stops at the [`Constant`] encoding instead of canonicalizing into a - /// [`PrimitiveArray`]. - #[test] - fn constant_non_null_input_yields_constant_output() -> VortexResult<()> { - let input = literal_vector_array(&[3.0f64, 4.0], 4); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); - let mut ctx = SESSION.create_execution_ctx(); - let output = result.execute_until::(&mut ctx)?; - - let constant = output - .as_opt::() - .expect("L2Norm over a constant input must produce a constant output"); - assert_eq!(constant.len(), 4); - let norm = constant - .scalar() - .as_primitive() - .as_::() - .expect("norm scalar must be a non-null primitive"); - assert_close(&[norm], &[5.0]); - Ok(()) - } - - /// A constant input whose scalar is null should short-circuit to a null [`ConstantArray`] of - /// the correct primitive dtype and length. - #[test] - fn constant_null_input_yields_null_constant_output() -> VortexResult<()> { - let storage_dtype = DType::FixedSizeList( - DType::Primitive(PType::F64, Nullability::NonNullable).into(), - 2, - Nullability::Nullable, - ); - let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); - let null_scalar = Scalar::null(DType::Extension(ext_dtype)); - let input = ConstantArray::new(null_scalar, 3).into_array(); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); - let mut ctx = SESSION.create_execution_ctx(); - let output = result.execute_until::(&mut ctx)?; - - let constant = output - .as_opt::() - .expect("null constant input must produce a constant output"); - assert_eq!(constant.len(), 3); - assert!(constant.scalar().is_null()); - assert_eq!( - constant.dtype(), - &DType::Primitive(PType::F64, Nullability::Nullable) - ); - Ok(()) - } - - #[rstest] - #[case::fixed_shape_tensor(l2_norm_tensor_child())] - #[case::vector(l2_norm_vector_child())] - fn serde_round_trip(#[case] child: ArrayRef) -> VortexResult<()> { - let original = L2Norm::try_new_array(child.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(L2Norm); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("L2Norm serialize must produce metadata"); - - let children = vec![child]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) - } - - fn l2_norm_tensor_child() -> ArrayRef { - tensor_array(&[3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid tensor array") - } - - fn l2_norm_vector_child() -> ArrayRef { - vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") - } -} diff --git a/vortex-tensor/src/scalar_fns/mod.rs b/vortex-tensor/src/scalar_fns/mod.rs index 68f10ca6b01..706392d3b25 100644 --- a/vortex-tensor/src/scalar_fns/mod.rs +++ b/vortex-tensor/src/scalar_fns/mod.rs @@ -6,3 +6,7 @@ pub mod cosine_similarity; pub mod inner_product; pub mod l2_norm; +pub mod row; + +#[cfg(test)] +mod tests; diff --git a/vortex-tensor/src/scalar_fns/row.rs b/vortex-tensor/src/scalar_fns/row.rs new file mode 100644 index 00000000000..3c02a1d2615 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/row.rs @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What the tensor scalar functions add to the row-function machinery: an element type that reads a +//! tensor row and the width rule they share. + +use std::marker::PhantomData; + +use num_traits::Float; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::PType; +use vortex_array::scalar_fn::InputElement; +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; + +use crate::utils::extract_flat_elements; +use crate::utils::validate_tensor_float_input; +use crate::utils::validate_tensor_float_inputs; + +/// The width rule the tensor scalar functions share: every argument is the same float tensor dtype, +/// and the width is its element ptype. +pub(crate) fn tensor_element_ptype(args: &[DType]) -> VortexResult { + Ok(validate_tensor_float_inputs(args)?.element_ptype()) +} + +/// Marker for tensor-valued input elements: accepts any tensor-like extension column whose +/// elements are `T`, and presents each row as its flat elements, `&[T]`. +pub struct TensorRow(PhantomData); + +/// The decoded form of a [`TensorRow`] column: one flat typed buffer plus the stride to read it at. +/// +/// Typed at decode time rather than per row. `FlatElements::row` re-derives its typed slice on every +/// call, which costs a ptype check and a buffer downcast per row; a row loop reads every row, so it +/// pays that once here instead. +pub struct TensorRows { + /// Every row's elements, back to back. + elements: Buffer, + + /// Number of logical tensor rows, stored so zero-width tensors retain their length. + rows: usize, + + /// Elements per row, the length of each row slice. + list_size: usize, + + /// `list_size` for a full column and `0` for constant-backed storage, so `index * stride` pins a + /// constant to its single materialized row without a branch in the loop. + stride: usize, +} + +impl InputElement for TensorRow { + type Column = TensorRows; + type Varying<'a> = &'a TensorRows; + type Elem<'a> = &'a [T]; + + // Tensor storage is a fully materialized non-nullable primitive buffer, so the elements behind + // a null row are arbitrary values rather than an unresolvable reference. + const DENSE_SAFE: bool = true; + // Tensor storage is a primitive buffer; reading it cannot fail on account of its values. + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + let tensor_match = validate_tensor_float_input(dtype)?; + let expected = T::PTYPE; + vortex_ensure_eq!( + tensor_match.element_ptype(), + expected, + "expected a tensor of {expected} elements, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let rows = array.len(); + let list_size = validate_tensor_float_input(array.dtype())?.list_size() as usize; + let ext: ExtensionArray = array.execute(ctx)?; + let flat = extract_flat_elements(ext.storage_array(), list_size, ctx)?; + + Ok(TensorRows { + rows, + list_size: flat.list_size(), + stride: flat.row_stride(), + elements: flat.into_buffer::(), + }) + } + + fn get(column: &Self::Column, index: usize) -> &[T] { + let start = index * column.stride; + &column.elements.as_slice()[start..start + column.list_size] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.rows + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a [T] + where + Self: 'a, + { + Self::get(column, index) + } +} + +/// Test-only probe recording which operands the last `prepare` step saw as batch-constant, so a +/// test can assert its inputs took the stride-0 decode path rather than merely producing the right +/// values through the varying path. +#[cfg(test)] +pub(crate) mod probe { + use std::cell::Cell; + + thread_local! { + /// Bitmask of the constant operands the last `prepare` saw (bit 0 for the lhs, bit 1 for + /// the rhs). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + pub(crate) static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; + } + + /// Record which operands `prepare` saw as constant. + pub(crate) fn record(lhs_constant: bool, rhs_constant: bool) { + SEEN_CONSTANTS.set(u8::from(lhs_constant) | (u8::from(rhs_constant) << 1)); + } +} diff --git a/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs b/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs new file mode 100644 index 00000000000..60e75792109 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs @@ -0,0 +1,586 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::assert_arrays_eq; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::cosine_similarity::CosineSimilarity; +use crate::scalar_fns::row::probe; +use crate::tests::SESSION; +use crate::types::vector::Vector; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::constant_tensor_array; +use crate::utils::test_helpers::literal_vector_array; +use crate::utils::test_helpers::normalized_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates cosine similarity between two tensor arrays and returns the result as `Vec`. +fn eval_cosine_similarity(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +/// Like [`eval_cosine_similarity`], but returns the executed array for exact array comparisons. +fn eval_cosine_similarity_array( + lhs: ArrayRef, + rhs: ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + Ok(result + .into_array() + .execute::(ctx)? + .into_array()) +} + +#[test] +fn unit_vectors_1d() -> VortexResult<()> { + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // Tensor 1 + 0.0, 1.0, 0.0, // Tensor 2 + ], + )?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // Tensor 1 + 1.0, 0.0, 0.0, // Tensor 2 + ], + )?; + + // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +/// Single-row cosine similarity for various vector pairs. +#[rstest] +// Antiparallel -> -1.0. +#[case::opposite(&[3], &[1.0, 0.0, 0.0], &[-1.0, 0.0, 0.0], &[-1.0])] +// dot=24, both magnitudes=5 -> 24/25 = 0.96. +#[case::non_unit(&[2], &[3.0, 4.0], &[4.0, 3.0], &[0.96])] +// Zero vector -> guarded to 0.0. +#[case::zero_norm(&[2], &[0.0, 0.0], &[1.0, 0.0], &[0.0])] +fn single_row( + #[case] shape: &[usize], + #[case] lhs_elems: &[f64], + #[case] rhs_elems: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let lhs = tensor_array(shape, lhs_elems)?; + let rhs = tensor_array(shape, rhs_elems)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, expected); + Ok(()) +} + +/// Self-similarity across various tensor shapes should always produce 1.0. +#[rstest] +// 2x3 matrix, flattened to 6 elements. +#[case::matrix_2d( + &[2, 3], + &[ + 1.0, 0.0, 0.0, // row 0 + 0.0, 0.0, 0.0, // row 1 + ], +)] +// 2x2x2 tensor, 8 elements. +#[case::tensor_3d(&[2, 2, 2], &[1.0; 8])] +fn self_similarity(#[case] shape: &[usize], #[case] elements: &[f64]) -> VortexResult<()> { + let lhs = tensor_array(shape, elements)?; + let rhs = tensor_array(shape, elements)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); + Ok(()) +} + +#[test] +fn scalar_0d() -> VortexResult<()> { + // 0-dimensional tensor: each "tensor" is a single scalar value. + let lhs = tensor_array(&[], &[5.0, 3.0])?; + let rhs = tensor_array(&[], &[5.0, -3.0])?; + + // Same sign -> 1.0, opposite sign -> -1.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, -1.0]); + Ok(()) +} + +#[test] +fn many_rows() -> VortexResult<()> { + // 5 tensors of shape [4] compared against themselves -> all 1.0. + let lhs = tensor_array( + &[4], + &[ + 1.0, 2.0, 3.0, 4.0, // tensor 0 + 0.0, 1.0, 0.0, 0.0, // tensor 1 + 5.0, 0.0, 5.0, 0.0, // tensor 2 + 1.0, 1.0, 1.0, 1.0, // tensor 3 + 0.0, 0.0, 0.0, 7.0, // tensor 4 + ], + )?; + let rhs = lhs.clone(); + + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0, 1.0, 1.0, 1.0, 1.0], + ); + Ok(()) +} + +#[test] +fn constant_query_tensor() -> VortexResult<()> { + // Compare 4 tensors of shape [3] against a single constant query tensor [1,0,0]. + let data = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // tensor 0 + 0.0, 1.0, 0.0, // tensor 1 + 0.0, 0.0, 1.0, // tensor 2 + 1.0, 0.0, 0.0, // tensor 3 + ], + )?; + let query = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 4)?; + + assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); + Ok(()) +} + +#[test] +fn vector_unit_vectors() -> VortexResult<()> { + let lhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 0.0, 1.0, 0.0, // vector 1 + ], + )?; + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 1.0, 0.0, 0.0, // vector 1 + ], + )?; + + // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +#[test] +fn vector_constant_query() -> VortexResult<()> { + let data = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 0.0, 1.0, 0.0, // vector 1 + 0.0, 0.0, 1.0, // vector 2 + 1.0, 0.0, 0.0, // vector 3 + ], + )?; + let query = Vector::constant_array(&[1.0, 0.0, 0.0], 4)?; + + assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 2 rows of dim-2 vectors. Row 1 of rhs is masked as null. + let lhs = tensor_array(&[2], &[3.0, 4.0, 1.0, 0.0])?; + let rhs = tensor_array(&[2], &[3.0, 4.0, 0.0, 1.0])?; + let rhs = MaskedArray::try_new(rhs, Validity::from_iter([true, false]))?.into_array(); + + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: self-similarity = 1.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[1.0]); + Ok(()) +} + +#[test] +fn both_normalized_self_similarity() -> VortexResult<()> { + // [3.0, 4.0] has norm 5.0, normalized [0.6, 0.8]. + // [1.0, 0.0] has norm 1.0, normalized [1.0, 0.0]. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + // Self-similarity should always be 1.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 1.0]); + Ok(()) +} + +#[test] +fn both_normalized_orthogonal() -> VortexResult<()> { + // [3.0, 0.0] normalized [1.0, 0.0], norm 3.0. + // [0.0, 4.0] normalized [0.0, 1.0], norm 4.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[1.0, 0.0], &[3.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.0, 1.0], &[4.0], &mut ctx)?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); + Ok(()) +} + +#[test] +fn both_normalized_zero_norm() -> VortexResult<()> { + // Zero-norm row: normalized is [0.0, 0.0], norm is 0.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + // Row 0: dot([0.6, 0.8], [0.6, 0.8]) = 1.0, row 1: dot([0,0], [1,0]) = 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lhs() -> VortexResult<()> { + // LHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // RHS is plain [3.0, 4.0]. + // cosine_similarity([3.0, 4.0], [3.0, 4.0]) = 1.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = tensor_array(&[2], &[3.0, 4.0])?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_rhs() -> VortexResult<()> { + // LHS is plain [1.0, 0.0], RHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // cosine_similarity([1.0, 0.0], [3.0, 4.0]) = 3.0 / (1.0 * 5.0) = 0.6. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = tensor_array(&[2], &[1.0, 0.0])?; + let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.6]); + Ok(()) +} + +#[test] +fn both_normalized_null_norms() -> VortexResult<()> { + // Row 0: valid, row 1: null (via nullable norms on rhs). + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + let normalized_r = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms_r = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let rhs = Normalized::try_new(normalized_r, norms_r, &mut ctx)?.into_array(); + + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[1.0]); + Ok(()) +} + +#[test] +fn both_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. With a stored norm of `0.0`, cosine + // similarity for that row must be `0.0` even though the dot product of the normalized + // children is nonzero. + let normalized_l = tensor_array(&[2], &[0.6, 0.8])?; + let norms_l = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally violates the unit-norm invariant by + // pairing a nonzero normalized row with a stored norm of `0.0`, mimicking lossy storage. + let lhs = unsafe { Normalized::new_unchecked(normalized_l, norms_l) }.into_array(); + + let normalized_r = tensor_array(&[2], &[0.6, 0.8])?; + let norms_r = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: Same as above for the rhs operand. + let rhs = unsafe { Normalized::new_unchecked(normalized_r, norms_r) }.into_array(); + + // `dot(normalized_l, normalized_r) = 1.0`, but the authoritative stored norms are both + // `0.0`, so cosine similarity must be `0.0`. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. The plain side is a normal nonzero + // tensor with positive norm. cosine similarity must still be `0.0` because the + // authoritative stored norm on the denorm side is `0.0`. + let normalized = tensor_array(&[2], &[0.6, 0.8])?; + let norms = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally pairs a nonzero normalized row with a + // stored norm of `0.0`, mimicking lossy storage where the stored norm is authoritative. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + let plain = tensor_array(&[2], &[1.0, 0.0])?; + + // Denorm on the lhs: `One { denorm: lhs, plain: rhs }`. + assert_close( + &eval_cosine_similarity(denorm.clone(), plain.clone())?, + &[0.0], + ); + + // Denorm on the rhs: `One { denorm: rhs, plain: lhs }`. The same zero-norm guard must + // fire regardless of operand order. + assert_close(&eval_cosine_similarity(plain, denorm)?, &[0.0]); + Ok(()) +} + +#[test] +fn constant_lhs_matches_plain_tensor() -> VortexResult<()> { + // The constant query `[1, 2, 2]` has norm 3, so its normalized form is `[1/3, 2/3, 2/3]`. + // Expected cosine similarity against each row is `dot([1, 2, 2], row) / (3 * ||row||)`. + let lhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // dot=1, ||rhs||=1, expected=1/3 + 1.0, 2.0, 2.0, // dot=9, ||rhs||=3, expected=1 + 0.0, 0.0, 1.0, // dot=2, ||rhs||=1, expected=2/3 + 2.0, 1.0, 2.0, // dot=8, ||rhs||=3, expected=8/9 + ], + )?; + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) +} + +#[test] +fn constant_rhs_matches_plain_tensor() -> VortexResult<()> { + // Mirror of `constant_lhs_matches_plain_tensor` with the constant on the right. + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let rhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) +} + +#[test] +fn both_constant_tensors() -> VortexResult<()> { + // `[1, 0, 0]` vs `[1, 1, 0]`. dot=1, ||lhs||=1, ||rhs||=sqrt(2), expected=1/sqrt(2). + let lhs = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 3)?; + let rhs = constant_tensor_array(&[3], &[1.0, 1.0, 0.0], 3)?; + let expected = 1.0 / 2.0_f64.sqrt(); + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[expected, expected, expected], + ); + Ok(()) +} + +#[test] +fn constant_zero_norm_query() -> VortexResult<()> { + // A zero-norm constant query must produce `0.0` for every row via the zero-norm guard in + // `cosine_one_normalized` and `execute_both_normalized`. + let lhs = constant_tensor_array(&[3], &[0.0, 0.0, 0.0], 3)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 2.0, 3.0, // + 4.0, 5.0, 6.0, // + 7.0, 8.0, 9.0, // + ], + )?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0, 0.0]); + Ok(()) +} + +#[test] +fn constant_self_similarity_nonunit() -> VortexResult<()> { + // A non-unit constant query compared to itself must produce `1.0`. This exercises the + // helper's division: after normalization, both sides must be exactly unit so the + // Normalized fast path's inner product yields 1. + let lhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + let rhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0; 5]); + Ok(()) +} + +/// An extension array over constant storage (what [`Vector::constant_array`] builds) is a batch +/// constant like any other: the row layer sees through the wrapper, so `prepare` hoists its norm +/// exactly as it does for the literal shape. This used to be intercepted by a hand-written +/// `reduce_encoded` rewrite into `Normalized`, deleted in favor of the framework path. +#[test] +fn vector_constant_matches_plain() -> VortexResult<()> { + let lhs = Vector::constant_array(&[1.0, 2.0, 2.0], 4)?; + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + assert_eq!( + probe::SEEN_CONSTANTS.get(), + 0b01, + "the extension-over-constant lhs must reach prepare as a batch constant", + ); + Ok(()) +} + +/// The literal-constant shape (a [`ConstantArray`] over a [`Vector`] extension scalar, what a +/// `lit(query)` expression produces) reaches the row loop, unlike an extension-wrapped constant, +/// which `reduce_encoded` rewrites into `Normalized`. There the prepared kernel hoists the query's +/// norm once per batch, and the result must be exactly the result of expanding the same query +/// into a full column, which hoists nothing. +/// +/// [`ConstantArray`]: vortex_array::arrays::ConstantArray +#[test] +fn literal_constant_rhs_matches_expanded_column() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); + let lhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let query = [1.0, 2.0, 2.0]; + + let from_constant = + eval_cosine_similarity_array(lhs.clone(), literal_vector_array(&query, 4), &mut ctx)?; + let from_expanded = + eval_cosine_similarity_array(lhs, vector_array(3, &query.repeat(4))?, &mut ctx)?; + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// The mirror of [`literal_constant_rhs_matches_expanded_column`], exercising the hoisted-lhs arm +/// of the prepared kernel. +#[test] +fn literal_constant_lhs_matches_expanded_column() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let query = [1.0, 2.0, 2.0]; + + let from_constant = + eval_cosine_similarity_array(literal_vector_array(&query, 4), rhs.clone(), &mut ctx)?; + let from_expanded = + eval_cosine_similarity_array(vector_array(3, &query.repeat(4))?, rhs, &mut ctx)?; + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// A zero-norm literal constant query must be guarded to `0.0` on every row by the prepared row +/// kernel, exactly as the unprepared kernel guards it. +#[test] +fn literal_constant_zero_norm_query_yields_zero() -> VortexResult<()> { + let lhs = vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; + let rhs = literal_vector_array(&[0.0f64, 0.0, 0.0], 2); + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0]); + Ok(()) +} + +/// Two literal constants are folded to a single-row execution by the row lifting, and that row +/// still runs the prepared kernel with both norms hoisted. +#[test] +fn both_literal_constants() -> VortexResult<()> { + let lhs = literal_vector_array(&[1.0f64, 0.0, 0.0], 3); + let rhs = literal_vector_array(&[1.0f64, 1.0, 0.0], 3); + let expected = 1.0 / 2.0_f64.sqrt(); + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[expected; 3]); + Ok(()) +} + +#[rstest] +#[case::vector(cosine_vector_lhs(), cosine_vector_rhs())] +#[case::fixed_shape_tensor(cosine_tensor_lhs(), cosine_tensor_rhs())] +fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { + let original = + CosineSimilarity.try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(CosineSimilarity); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("CosineSimilarity serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn cosine_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 0.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +} + +fn cosine_vector_rhs() -> ArrayRef { + vector_array(3, &[0.0, 1.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +} + +fn cosine_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 0.0, 3.0, 4.0]).expect("valid tensor array") +} + +fn cosine_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[0.0, 1.0, 3.0, 4.0]).expect("valid tensor array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/inner_product.rs b/vortex-tensor/src/scalar_fns/tests/inner_product.rs new file mode 100644 index 00000000000..af7fbb7bc1a --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/inner_product.rs @@ -0,0 +1,253 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::inner_product::InnerProduct; +use crate::tests::SESSION; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::normalized_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates inner product between two tensor arrays and returns the result as `Vec`. +fn eval_inner_product(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +/// Single-row inner product for various vector pairs. +#[rstest] +// Orthogonal: [1, 0] . [0, 1] = 0. +#[case::orthogonal(&[2], &[1.0, 0.0], &[0.0, 1.0], &[0.0])] +// Parallel: [3, 4] . [3, 4] = 9 + 16 = 25. +#[case::parallel(&[2], &[3.0, 4.0], &[3.0, 4.0], &[25.0])] +// Antiparallel: [1, 2] . [-1, -2] = -1 + -4 = -5. +#[case::antiparallel(&[2], &[1.0, 2.0], &[-1.0, -2.0], &[-5.0])] +// Scaled: [2, 0] . [3, 0] = 6. +#[case::scaled(&[2], &[2.0, 0.0], &[3.0, 0.0], &[6.0])] +fn single_row( + #[case] shape: &[usize], + #[case] lhs_elems: &[f64], + #[case] rhs_elems: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let lhs = tensor_array(shape, lhs_elems)?; + let rhs = tensor_array(shape, rhs_elems)?; + assert_close(&eval_inner_product(lhs, rhs)?, expected); + Ok(()) +} + +#[test] +fn multiple_rows() -> VortexResult<()> { + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // tensor 0 + 3.0, 4.0, 0.0, // tensor 1 + 1.0, 1.0, 1.0, // tensor 2 + ], + )?; + let rhs = tensor_array( + &[3], + &[ + 0.0, 1.0, 0.0, // tensor 0: dot = 0 + 3.0, 4.0, 0.0, // tensor 1: dot = 25 + 2.0, 2.0, 2.0, // tensor 2: dot = 6 + ], + )?; + assert_close(&eval_inner_product(lhs, rhs)?, &[0.0, 25.0, 6.0]); + Ok(()) +} + +#[test] +fn vector_inner_product() -> VortexResult<()> { + let lhs = vector_array( + 2, + &[ + 3.0, 4.0, // vector 0 + 1.0, 0.0, // vector 1 + ], + )?; + let rhs = vector_array( + 2, + &[ + 3.0, 4.0, // vector 0: dot = 25 + 0.0, 1.0, // vector 1: dot = 0 + ], + )?; + assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 3 rows of dim-2 vectors. Row 1 of lhs is masked as null. + let lhs = tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; + let rhs = tensor_array(&[2], &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0])?; + let lhs = MaskedArray::try_new(lhs, Validity::from_iter([true, false, true]))?.into_array(); + + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: 1*7 + 2*8 = 23, row 1: null, row 2: 5*11 + 6*12 = 127. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert!(prim.is_valid(2, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[23.0]); + assert_close(&[prim.as_slice::()[2]], &[127.0]); + Ok(()) +} + +#[test] +fn rejects_non_extension_dtype() { + let lhs = PrimitiveArray::from_iter([1.0_f64, 2.0]).into_array(); + let rhs = PrimitiveArray::from_iter([3.0_f64, 4.0]).into_array(); + let result = InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs, rhs]); + assert!(result.is_err()); +} + +#[test] +fn rejects_mismatched_dtypes() -> VortexResult<()> { + let lhs = tensor_array(&[2], &[1.0_f64, 2.0])?; + let rhs = vector_array(2, &[3.0_f64, 4.0])?; + let result = InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs, rhs]); + assert!(result.is_err()); + Ok(()) +} + +#[test] +fn both_normalized() -> VortexResult<()> { + // LHS: [3.0, 4.0] = Normalized([0.6, 0.8], 5.0). + // RHS: [1.0, 0.0] = Normalized([1.0, 0.0], 1.0). + // dot([3.0, 4.0], [1.0, 0.0]) = 3.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[1.0, 0.0], &[1.0], &mut ctx)?; + + // Expected: 5.0 * 1.0 * dot([0.6, 0.8], [1.0, 0.0]) = 5.0 * 0.6 = 3.0. + assert_close(&eval_inner_product(lhs, rhs)?, &[3.0]); + Ok(()) +} + +#[test] +fn both_normalized_multiple_rows() -> VortexResult<()> { + // Row 0: [3.0, 4.0] dot [3.0, 4.0] = 25.0. + // Row 1: [1.0, 0.0] dot [0.0, 1.0] = 0.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0], &mut ctx)?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lhs() -> VortexResult<()> { + // LHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // RHS: plain [1.0, 2.0]. + // dot([3.0, 4.0], [1.0, 2.0]) = 3.0 + 8.0 = 11.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = tensor_array(&[2], &[1.0, 2.0])?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_rhs() -> VortexResult<()> { + // LHS: plain [1.0, 2.0]. + // RHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // dot([1.0, 2.0], [3.0, 4.0]) = 3.0 + 8.0 = 11.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = tensor_array(&[2], &[1.0, 2.0])?; + let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); + Ok(()) +} + +#[test] +fn both_normalized_null_norms() -> VortexResult<()> { + // Row 0: valid, row 1: null (via nullable norms on lhs). + let normalized_l = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms_l = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + + let lhs = Normalized::try_new(normalized_l, norms_l, &mut ctx)?.into_array(); + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: 5.0 * 5.0 * dot([0.6, 0.8], [0.6, 0.8]) = 25.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[25.0]); + Ok(()) +} + +#[rstest] +#[case::vector(inner_product_vector_lhs(), inner_product_vector_rhs())] +#[case::fixed_shape_tensor(inner_product_tensor_lhs(), inner_product_tensor_rhs())] +fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { + let original = + InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(InnerProduct); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("InnerProduct serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn inner_product_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") +} + +fn inner_product_vector_rhs() -> ArrayRef { + vector_array(3, &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0]).expect("valid vector array") +} + +fn inner_product_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0]).expect("valid tensor array") +} + +fn inner_product_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[5.0, 6.0, 7.0, 8.0]).expect("valid tensor array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/l2_norm.rs b/vortex-tensor/src/scalar_fns/tests/l2_norm.rs new file mode 100644 index 00000000000..a9fda0326d8 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/l2_norm.rs @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::Constant; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::l2_norm::L2Norm; +use crate::tests::SESSION; +use crate::types::vector::Vector; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::literal_vector_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates L2 norm on a tensor/vector array and returns the result as `Vec`. +fn eval_l2_norm(input: ArrayRef) -> VortexResult> { + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +#[rstest] +#[case::three_four_five(&[2], &[3.0, 4.0], &[5.0])] +#[case::zero_vector(&[3], &[0.0, 0.0, 0.0], &[0.0])] +#[case::single_element(&[1], &[7.0], &[7.0])] +#[case::negative_elements(&[2], &[-3.0, -4.0], &[5.0])] +fn known_norms( + #[case] shape: &[usize], + #[case] elements: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let arr = tensor_array(shape, elements)?; + assert_close(&eval_l2_norm(arr)?, expected); + Ok(()) +} + +#[test] +fn multiple_rows() -> VortexResult<()> { + let arr = tensor_array( + &[3], + &[ + 3.0, 4.0, 0.0, // norm = 5.0 + 0.0, 0.0, 0.0, // norm = 0.0 + 1.0, 1.0, 1.0, // norm = sqrt(3) + ], + )?; + assert_close(&eval_l2_norm(arr)?, &[5.0, 0.0, 3.0_f64.sqrt()]); + Ok(()) +} + +#[test] +fn vector_multiple_rows() -> VortexResult<()> { + let arr = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // norm = 1.0 + 3.0, 4.0, 0.0, // norm = 5.0 + ], + )?; + assert_close(&eval_l2_norm(arr)?, &[1.0, 5.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 2 rows of dim-2 vectors. Row 1 is masked as null. + let arr = tensor_array(&[2], &[3.0, 4.0, 0.0, 0.0])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![arr])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: norm = 5.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +/// A constant input whose scalar is a non-null tensor should short-circuit to a +/// [`ConstantArray`] output whose scalar is the precomputed norm. Uses [`execute_until`] so +/// execution stops at the [`Constant`] encoding instead of canonicalizing into a +/// [`PrimitiveArray`]. +#[test] +fn constant_non_null_input_yields_constant_output() -> VortexResult<()> { + let input = literal_vector_array(&[3.0f64, 4.0], 4); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("L2Norm over a constant input must produce a constant output"); + assert_eq!(constant.len(), 4); + let norm = constant + .scalar() + .as_primitive() + .as_::() + .expect("norm scalar must be a non-null primitive"); + assert_close(&[norm], &[5.0]); + Ok(()) +} + +/// An extension array over constant storage is folded just like a top-level constant instead of +/// recomputing the same norm once per row. +#[test] +fn extension_backed_constant_yields_constant_output() -> VortexResult<()> { + let input = Vector::constant_array(&[3.0f64, 4.0], 4)?; + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("L2Norm over constant-backed extension storage must produce a constant output"); + assert_eq!(constant.len(), 4); + let norm = constant + .scalar() + .as_primitive() + .as_::() + .expect("norm scalar must be a non-null primitive"); + assert_close(&[norm], &[5.0]); + Ok(()) +} + +/// A constant input whose scalar is null should short-circuit to a null [`ConstantArray`] of +/// the correct primitive dtype and length. +#[test] +fn constant_null_input_yields_null_constant_output() -> VortexResult<()> { + let storage_dtype = DType::FixedSizeList( + DType::Primitive(PType::F64, Nullability::NonNullable).into(), + 2, + Nullability::Nullable, + ); + let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); + let null_scalar = Scalar::null(DType::Extension(ext_dtype)); + let input = ConstantArray::new(null_scalar, 3).into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("null constant input must produce a constant output"); + assert_eq!(constant.len(), 3); + assert!(constant.scalar().is_null()); + assert_eq!( + constant.dtype(), + &DType::Primitive(PType::F64, Nullability::Nullable) + ); + Ok(()) +} + +/// An `f32` column must dispatch at `f32` and produce an `f32` result, which is the property that +/// makes width polymorphism load-bearing rather than decorative. +#[rstest] +#[case::f32(&[3.0f32, 4.0], PType::F32)] +#[case::f64(&[3.0f64, 4.0], PType::F64)] +fn dispatches_at_input_width( + #[case] elements: &[T], + #[case] expected: PType, +) -> VortexResult<()> { + let arr = tensor_array(&[2], elements)?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = L2Norm + .try_new_array(arr.len(), EmptyOptions, [arr])? + .execute(&mut ctx)?; + assert_eq!(prim.ptype(), expected); + Ok(()) +} + +/// `L2Norm(Normalized(normalized, norms))` reads back the authoritative stored norms rather than +/// recomputing over decoded coordinates. The normalized child here is deliberately *not* +/// unit-norm, mimicking lossy storage, so readthrough and recompute disagree: row 0 decodes to +/// `[6, 8]` (norm `10`) and row 1 to `[6, 0]` (norm `6`), while the stored norms are `5` and `2`. +#[test] +fn normalized_readthrough_returns_stored_norms() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[1.2, 1.6, 3.0, 0.0])?; + let norms = PrimitiveArray::from_iter([5.0f64, 2.0]).into_array(); + // SAFETY: A focused test of the lossy storage contract: the stored norms are authoritative + // even though this normalized child violates the unit-norm invariant. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + assert_close(&eval_l2_norm(denorm)?, &[5.0, 2.0]); + Ok(()) +} + +/// The readthrough must survive a partially-null column. +/// +/// This pins the dense policy the row contract derives. Filtering could hand `reduce_encoded` a +/// filtered input, which is no longer an `ExactScalarFn`, silently falling back to +/// decode-and-recompute. For a lossy child that changes the answer: row 0 below would come back as +/// `10` (recomputed from `[6, 8]`) instead of the authoritative stored `5`. +#[test] +fn normalized_readthrough_survives_null_rows() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[1.2, 1.6, 3.0, 0.0])?; + let norms = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + // SAFETY: Intentionally lossy, as in `normalized_readthrough_returns_stored_norms`, so that + // a recompute fallback is observable. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![denorm])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +/// The readthrough must still propagate nulls carried by the `norms` child. +#[test] +fn normalized_readthrough_propagates_null_norms() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let denorm = Normalized::try_new(normalized, norms, &mut ctx)?.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![denorm])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +#[rstest] +#[case::fixed_shape_tensor(l2_norm_tensor_child())] +#[case::vector(l2_norm_vector_child())] +fn serde_round_trip(#[case] child: ArrayRef) -> VortexResult<()> { + let original = L2Norm.try_new_array(child.len(), EmptyOptions, [child.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(L2Norm); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("L2Norm serialize must produce metadata"); + + let children = vec![child]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn l2_norm_tensor_child() -> ArrayRef { + tensor_array(&[3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid tensor array") +} + +fn l2_norm_vector_child() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/mod.rs b/vortex-tensor/src/scalar_fns/tests/mod.rs new file mode 100644 index 00000000000..bb3726e9329 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/mod.rs @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for the tensor scalar functions. + +mod cosine_similarity; +mod inner_product; +mod l2_norm; +mod row; diff --git a/vortex-tensor/src/scalar_fns/tests/row.rs b/vortex-tensor/src/scalar_fns/tests/row.rs new file mode 100644 index 00000000000..f08f614cfef --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/row.rs @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use num_traits::Float; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::match_each_float_ptype; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::assert_element_conforms; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; +use vortex_session::registry::CachedId; + +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::tensor_array; + +/// The marginal cost of a new tensor scalar function is this entire definition. Everything else +/// (null propagation, constants, validity, f16/f32/f64 dispatch, dtype checks, and constructors) is +/// derived. +#[derive(Clone, Debug, Default)] +struct L1Norm; + +impl RowFn for L1Norm { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.l1_norm"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow,), ElementSink, _, _>( + |_| (), + |&(), (row,), output| *output = l1_norm_row(row), + ) + }) + } +} + +fn l1_norm_row(row: &[T]) -> T { + row.iter().fold(T::zero(), |acc, &x| acc + x.abs()) +} + +#[test] +fn derived_fn_executes_with_nulls() -> VortexResult<()> { + let arr = tensor_array(&[2], &[3.0, -4.0, 1.0, 1.0])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + let prim: PrimitiveArray = L1Norm + .try_new_array(arr.len(), EmptyOptions, [arr])? + .execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[7.0]); + Ok(()) +} + +/// A kernel written once serves every float width. +#[test] +fn derived_fn_dispatches_at_input_width() -> VortexResult<()> { + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + + let f32_result: PrimitiveArray = L1Norm + .try_new_array(1, EmptyOptions, [tensor_array(&[2], &[3.0f32, -4.0])?])? + .execute(&mut ctx)?; + assert_eq!(f32_result.ptype(), PType::F32); + assert_eq!(f32_result.as_slice::(), &[7.0f32]); + + let f64_result: PrimitiveArray = L1Norm + .try_new_array(1, EmptyOptions, [tensor_array(&[2], &[3.0f64, -4.0])?])? + .execute(&mut ctx)?; + assert_eq!(f64_result.ptype(), PType::F64); + Ok(()) +} + +/// Runs the out-of-crate [`TensorRow`] element through `vortex-array`'s shared element conformance +/// check, with `NaN` and infinities sitting behind the null row so a wrong `DENSE_SAFE` would be +/// read rather than skipped. +#[test] +fn tensor_row_element_conforms() -> VortexResult<()> { + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + let arr = tensor_array(&[2], &[3.0, -4.0, f64::NAN, f64::INFINITY])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + assert_element_conforms::>( + arr, + &DType::Primitive(PType::F64, Nullability::NonNullable), + &mut ctx, + ) +} diff --git a/vortex-tensor/src/utils.rs b/vortex-tensor/src/utils.rs index 488694bd47f..460dde82ea7 100644 --- a/vortex-tensor/src/utils.rs +++ b/vortex-tensor/src/utils.rs @@ -1,13 +1,17 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +//! Shared helpers for the tensor scalar functions. + use half::f16; +use num_traits::Float; use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::Constant; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFn; @@ -20,6 +24,8 @@ use vortex_array::dtype::NativePType; use vortex_array::dtype::PType; use vortex_array::dtype::proto::dtype as pb; use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure; @@ -58,6 +64,20 @@ pub fn unit_norm_tolerance(element_ptype: PType, dimensions: usize) -> f64 { SAFETY_FACTOR as f64 * machine_epsilon * dimensions_root } +/// The L2 norm of one row: `sqrt(sum(v_i^2))`. A zero-length or all-zero row gives `0.0`. +/// +/// Shared by `l2_norm` and by cosine similarity's hoisted constant norm. The accumulation order is +/// part of the contract rather than an implementation detail: cosine's prepared and per-row arms +/// must agree bit for bit, which only holds while both sum in this order. Keeping one copy is what +/// stops the two drifting apart. +pub(crate) fn l2_norm_row(v: &[T]) -> T { + let mut sum_sq = T::zero(); + for &x in v { + sum_sq = sum_sq + x * x; + } + sum_sq.sqrt() +} + /// Extracts the `(normalized, norms)` children of a [`Normalized`]-encoded array. /// /// # Panics @@ -97,17 +117,78 @@ pub fn validate_tensor_float_input(input_dtype: &DType) -> VortexResult( - lhs: &'a DType, - rhs: &DType, -) -> VortexResult> { - vortex_ensure!( - lhs.eq_ignore_nullability(rhs), - "binary tensor expression expects inputs to have the same dtype, got {lhs} and {rhs}" - ); - validate_tensor_float_input(lhs) +pub fn validate_tensor_float_inputs(args: &[DType]) -> VortexResult> { + let (first, rest) = args + .split_first() + .ok_or_else(|| vortex_err!("tensor expression expects at least one input"))?; + for arg in rest { + vortex_ensure!( + first.eq_ignore_nullability(arg), + "tensor expression expects inputs to have the same dtype, got {first} and {arg}" + ); + } + validate_tensor_float_input(first) +} + +/// Metadata for a serialized binary tensor-op array (shared by [`InnerProduct`] and +/// [`CosineSimilarity`]). Both operands share the same extension dtype up to nullability +/// (enforced by their `return_dtype` checks), but their individual nullabilities are lost in the +/// parent's unioned output, so both are persisted. +/// +/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity +/// [`InnerProduct`]: crate::scalar_fns::inner_product::InnerProduct +#[derive(Clone, prost::Message)] +pub(crate) struct BinaryTensorOpMetadata { + #[prost(message, optional, tag = "1")] + pub(crate) lhs_dtype: Option, + #[prost(message, optional, tag = "2")] + pub(crate) rhs_dtype: Option, +} + +impl BinaryTensorOpMetadata { + /// Encodes the two children of `view` into a [`BinaryTensorOpMetadata`] byte blob. + pub(crate) fn encode_from_view( + view: &ScalarFnArrayView, + ) -> VortexResult> { + let scalar_fn_array = view.as_::(); + let lhs_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + let rhs_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); + Ok(Self { + lhs_dtype, + rhs_dtype, + } + .encode_to_vec()) + } + + /// Decodes `metadata` and fetches both children from `children` using the decoded dtypes, + /// validating that `lhs` and `rhs` are compatible tensor operands. + pub(crate) fn decode_children( + metadata: &[u8], + len: usize, + children: &dyn vortex_array::serde::ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = Self::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode BinaryTensorOpMetadata: {e}"))?; + let lhs_pb = metadata + .lhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing lhs_dtype"))?; + let rhs_pb = metadata + .rhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing rhs_dtype"))?; + + let lhs_dtype = DType::from_proto(lhs_pb, session)?; + let rhs_dtype = DType::from_proto(rhs_pb, session)?; + validate_tensor_float_inputs(&[lhs_dtype.clone(), rhs_dtype.clone()])?; + + let lhs = children.get(0, &lhs_dtype, len)?; + let rhs = children.get(1, &rhs_dtype, len)?; + Ok(vec![lhs, rhs]) + } } /// The flat primitive elements of a tensor storage array, with typed row access. @@ -132,12 +213,58 @@ impl FlatElements { /// /// When the source was a constant-backed storage, all indices resolve to the single stored /// row. + /// + /// This re-derives the typed slice on every call, which costs a ptype check and a buffer + /// downcast per row. A caller reading every row in a loop should take [`into_buffer`](Self::into_buffer) + /// instead and pay that once. #[must_use] pub fn row(&self, i: usize) -> &[T] { let row_idx = if self.is_constant { 0 } else { i }; let slice = self.elems.as_slice::(); &slice[row_idx * self.list_size..][..self.list_size] } + + /// Elements per row. + #[must_use] + pub fn list_size(&self) -> usize { + self.list_size + } + + /// The row stride: `list_size` for a full column, and `0` for constant-backed storage, whose + /// single materialized row every index reads. + #[must_use] + pub fn row_stride(&self) -> usize { + if self.is_constant { 0 } else { self.list_size } + } + + /// The elements as a typed buffer, checking the ptype once instead of once per row. + pub fn into_buffer(self) -> Buffer { + self.elems.into_buffer::() + } +} + +/// Rebuilds a tensor-like extension array from flat primitive elements. +/// +/// # Errors +/// +/// Returns an error if `elements` does not hold exactly `tensor_flat_size * row_count` values. +pub(crate) fn build_tensor_array( + dtype: DType, + tensor_flat_size: usize, + row_count: usize, + validity: Validity, + elements: Buffer, +) -> VortexResult { + let list_size = + u32::try_from(tensor_flat_size).vortex_expect("tensor flat size must fit into `u32`"); + + // SAFETY: Tensor elements are always non-nullable, so the validity carries no length. + let elements = unsafe { PrimitiveArray::new_unchecked(elements, Validity::NonNullable) }; + + let storage = + FixedSizeListArray::try_new(elements.into_array(), list_size, validity, row_count)?; + + Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) } /// Extracts the flat primitive elements from a tensor storage array (FixedSizeList). @@ -161,10 +288,10 @@ pub fn extract_flat_elements( let fsl: FixedSizeListArray = source.execute(ctx)?; let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + let dtype = elems.dtype(); vortex_ensure!( !elems.nullability().is_nullable(), - "tensor storage elements must be non-nullable, got {}", - elems.dtype(), + "tensor storage elements must be non-nullable, got {dtype}", ); Ok(FlatElements { elems, @@ -216,73 +343,14 @@ pub fn extract_constant_flat_row( let single = ConstantArray::new(constant.scalar().clone(), 1).into_array(); let fsl: FixedSizeListArray = single.execute(ctx)?; let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + let dtype = elems.dtype(); vortex_ensure!( !elems.nullability().is_nullable(), - "tensor storage elements must be non-nullable, got {}", - elems.dtype(), + "tensor storage elements must be non-nullable, got {dtype}", ); Ok(FlatRow { elems }) } -/// Metadata for a serialized binary tensor-op array (shared by [`InnerProduct`] and -/// [`CosineSimilarity`]). Both operands share the same extension dtype up to nullability -/// (enforced by their `return_dtype` checks), but their individual nullabilities are lost in the -/// parent's unioned output, so both are persisted. -/// -/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity -/// [`InnerProduct`]: crate::scalar_fns::inner_product::InnerProduct -#[derive(Clone, prost::Message)] -pub(crate) struct BinaryTensorOpMetadata { - #[prost(message, optional, tag = "1")] - pub(crate) lhs_dtype: Option, - #[prost(message, optional, tag = "2")] - pub(crate) rhs_dtype: Option, -} - -impl BinaryTensorOpMetadata { - /// Encodes the two children of `view` into a [`BinaryTensorOpMetadata`] byte blob. - pub(crate) fn encode_from_view( - view: &ScalarFnArrayView, - ) -> VortexResult> { - let scalar_fn_array = view.as_::(); - let lhs_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); - let rhs_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); - Ok(Self { - lhs_dtype, - rhs_dtype, - } - .encode_to_vec()) - } - - /// Decodes `metadata` and fetches both children from `children` using the decoded dtypes, - /// validating that `lhs` and `rhs` are compatible tensor operands. - pub(crate) fn decode_children( - metadata: &[u8], - len: usize, - children: &dyn vortex_array::serde::ArrayChildren, - session: &VortexSession, - ) -> VortexResult> { - let metadata = Self::decode(metadata) - .map_err(|e| vortex_err!("Failed to decode BinaryTensorOpMetadata: {e}"))?; - let lhs_pb = metadata - .lhs_dtype - .as_ref() - .ok_or_else(|| vortex_err!("metadata missing lhs_dtype"))?; - let rhs_pb = metadata - .rhs_dtype - .as_ref() - .ok_or_else(|| vortex_err!("metadata missing rhs_dtype"))?; - - let lhs_dtype = DType::from_proto(lhs_pb, session)?; - let rhs_dtype = DType::from_proto(rhs_pb, session)?; - validate_binary_tensor_float_inputs(&lhs_dtype, &rhs_dtype)?; - - let lhs = children.get(0, &lhs_dtype, len)?; - let rhs = children.get(1, &rhs_dtype, len)?; - Ok(vec![lhs, rhs]) - } -} - #[cfg(test)] pub mod test_helpers { use vortex_array::ArrayRef; @@ -358,9 +426,9 @@ pub mod test_helpers { } /// Builds a [`ConstantArray`] whose scalar is itself a [`Vector`] extension scalar, broadcast - /// to `len` rows. This is the shape produced by an `lit(vector_scalar)` literal expression — - /// the constant lives at the extension level rather than inside the FSL storage, in contrast - /// to [`Vector::constant_array`]. + /// to `len` rows. This is the shape produced by an `lit(vector_scalar)` literal expression, where + /// the constant lives at the extension level rather than inside the FSL storage, in contrast to + /// [`Vector::constant_array`]. pub fn literal_vector_array>( elements: &[T], len: usize, @@ -401,10 +469,10 @@ pub mod test_helpers { if a.is_nan() && e.is_nan() { continue; } + let diff = (a - e).abs(); assert!( (a - e).abs() < 1e-10, - "element {i}: got {a}, expected {e} (diff = {})", - (a - e).abs() + "element {i}: got {a}, expected {e} (diff = {diff})" ); } } diff --git a/vortex-tensor/src/vector_search.rs b/vortex-tensor/src/vector_search.rs index ad3b96d1bff..492bc837b89 100644 --- a/vortex-tensor/src/vector_search.rs +++ b/vortex-tensor/src/vector_search.rs @@ -35,11 +35,13 @@ use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; use vortex_array::scalar::PValue; use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexResult; @@ -79,7 +81,7 @@ pub fn build_similarity_search_tree>( let num_rows = data.len(); let query_vec = Vector::constant_array(query, num_rows)?; - let cosine = CosineSimilarity::try_new_array(data, query_vec)?.into_array(); + let cosine = CosineSimilarity.try_new_array(num_rows, EmptyOptions, [data, query_vec])?; let threshold_scalar = Scalar::primitive(threshold, Nullability::NonNullable); let threshold_array = ConstantArray::new(threshold_scalar, num_rows).into_array(); From 6c13e8516a0440f17ed74ccaeb87d68bbb38d13b Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 05/12] Port the geo predicates to RowFn Progress towards #9128. `vortex.geo.distance`, `vortex.geo.contains`, and `vortex.geo.intersects` become row functions, which deletes `scalar_fn/execute.rs` and the shared columnar execution it held. Decoding a geometry does expensive per-row work, so the geo elements set `InputElement::FILTERED_DECODE_COST` and sparse batches keep the filter strategy's shrunken decode. Branch-and-skip needs a decode that tolerates null rows without parsing them, so `geometries_null_tolerant` writes a placeholder geometry into null slots for `Point` and `Polygon`. It returns `Ok(None)` for any other geometry type, and the caller falls back to the filter strategy, which never decodes a null row. `contains` prepares a constant geometry once per batch rather than re-preparing it per row. The row layer sees through extension-over-constant, so the hand-written rewrite that used to uncover the constant is gone. `geo` is pinned to `=0.31.0` in the workspace manifest. `contains_route` transcribes geo's `impl_contains_from_relate!` dispatch table, so any bump that moves a row silently changes containment verdicts while the tests stay green wherever relate and the direct algorithm agree. A caret requirement would let `cargo update` take 0.31.x with no diff to review. `null_strategies` benchmarks the three strategies against `GeoContains` across validity densities, which is where the crossover between filtering and branch-and-skip was measured. Signed-off-by: Connor Tsui Co-authored-by: Claude --- Cargo.toml | 8 +- vortex-spatial/Cargo.toml | 8 +- vortex-spatial/benches/null_strategies.rs | 199 ++++++ vortex-spatial/src/extension/mod.rs | 41 ++ vortex-spatial/src/extension/point.rs | 18 + vortex-spatial/src/extension/polygon.rs | 18 + vortex-spatial/src/scalar_fn/contains.rs | 649 +++++++++++++++--- vortex-spatial/src/scalar_fn/distance.rs | 108 +-- vortex-spatial/src/scalar_fn/execute.rs | 19 +- .../src/scalar_fn/execute/binary.rs | 334 --------- .../src/scalar_fn/execute/geo_types.rs | 144 ---- vortex-spatial/src/scalar_fn/execute/unary.rs | 2 - vortex-spatial/src/scalar_fn/intersects.rs | 240 +++++-- vortex-spatial/src/scalar_fn/mod.rs | 1 + vortex-spatial/src/scalar_fn/row.rs | 170 +++++ vortex-spatial/src/test_harness.rs | 2 +- 16 files changed, 1224 insertions(+), 737 deletions(-) create mode 100644 vortex-spatial/benches/null_strategies.rs delete mode 100644 vortex-spatial/src/scalar_fn/execute/binary.rs delete mode 100644 vortex-spatial/src/scalar_fn/execute/geo_types.rs create mode 100644 vortex-spatial/src/scalar_fn/row.rs diff --git a/Cargo.toml b/Cargo.toml index 36aa5b2ac9e..5328905cc95 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -160,7 +160,13 @@ flatbuffers = "25.2.10" fsst-rs = "0.6.0" futures = { version = "0.3.31", default-features = false } fuzzy-matcher = "0.3" -geo = "0.31.0" +# `vortex-spatial`'s `contains_route` transcribes geo's `impl_contains_from_relate!` dispatch table, so +# any bump that moves a row silently changes containment verdicts — the tests stay green wherever +# relate and the direct algorithm agree. Pinned exactly so that taking any new geo, patch releases +# included, is a deliberate edit of this line that re-verifies the table; a caret requirement would +# let `cargo update` (or automated lockfile maintenance) take 0.31.x with no diff to review. See +# `vortex-spatial/src/scalar_fn/contains.rs`. +geo = "=0.31.0" geo-traits = "0.3.0" geo-types = "0.7.19" geoarrow = "0.8.0" diff --git a/vortex-spatial/Cargo.toml b/vortex-spatial/Cargo.toml index 8b790da3c05..139f28181d6 100644 --- a/vortex-spatial/Cargo.toml +++ b/vortex-spatial/Cargo.toml @@ -47,11 +47,15 @@ name = "envelope" harness = false [[bench]] -name = "predicate_bbox" +name = "binary_predicates" harness = false [[bench]] -name = "binary_predicates" +name = "null_strategies" +harness = false + +[[bench]] +name = "predicate_bbox" harness = false [[bench]] diff --git a/vortex-spatial/benches/null_strategies.rs b/vortex-spatial/benches/null_strategies.rs new file mode 100644 index 00000000000..1ef453d645b --- /dev/null +++ b/vortex-spatial/benches/null_strategies.rs @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Null-strategy comparison for the geo `contains` kernel, whose per-row geometry decode is what +//! the selection threshold exists for. +//! +//! Arms: `filter` and `branch` force one strategy through the test-harness seam +//! ([`execute_row_fn_with_strategy`]); `auto` executes the full pipeline and lets the per-batch +//! selection choose, which should track the faster forced arm on both sides of the crossover +//! (branch at dense validity, filter at sparse). +//! +//! Workloads: a column of small polygons CONTAINS a constant point, and polygon column CONTAINS +//! point column with independent nulls on both, each at null densities 0/1/5/10/25/50/90 percent +//! over 65536 rows, nulls placed by a seeded splitmix hash. +//! +//! Run with `cargo bench -p vortex-spatial --bench null_strategies`. + +#![expect(clippy::unwrap_used)] + +use std::sync::LazyLock; + +use divan::Bencher; +use divan::counter::ItemsCount; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::NullStrategy; +use vortex_array::scalar_fn::execute_row_fn_with_strategy; +use vortex_array::validity::Validity; +use vortex_spatial::scalar_fn::contains::SpatialContains; +use vortex_spatial::test_harness::geo_session; +use vortex_spatial::test_harness::point_column; +use vortex_spatial::test_harness::polygon_column; +use vortex_session::VortexSession; + +static SESSION: LazyLock = LazyLock::new(geo_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +const ROWS: usize = 65536; + +/// Null densities in percent. +const DENSITIES: &[usize] = &[0, 1, 5, 10, 25, 50, 90]; + +/// Deterministic pseudo-random value in `[0, 1)` (same generator as `binary_predicates`). +fn unit(i: usize) -> f64 { + ((i.wrapping_mul(2654435761) >> 8) % 10_000) as f64 / 10_000.0 +} + +/// splitmix64, for seeded random null placement. +fn splitmix64(mut x: u64) -> u64 { + x = x.wrapping_add(0x9E3779B97F4A7C15); + x = (x ^ (x >> 30)).wrapping_mul(0xBF58476D1CE4E5B9); + x = (x ^ (x >> 27)).wrapping_mul(0x94D049BB133111EB); + x ^ (x >> 31) +} + +/// A small square (side 2) centered at `(cx, cy)`. +fn square(cx: f64, cy: f64) -> Vec> { + vec![vec![ + (cx - 1.0, cy - 1.0), + (cx + 1.0, cy - 1.0), + (cx + 1.0, cy + 1.0), + (cx - 1.0, cy + 1.0), + (cx - 1.0, cy - 1.0), + ]] +} + +/// [`ROWS`] small squares spread over roughly `[-150, 150)^2`; a handful contain the origin, and +/// each row's verdict is a direct point-in-polygon test. +fn squares() -> ArrayRef { + let rows = (0..ROWS) + .map(|i| square(300.0 * unit(i) - 150.0, 300.0 * unit(i + 1) - 150.0)) + .collect(); + polygon_column(rows).unwrap() +} + +/// [`ROWS`] points over the same region. +fn points() -> ArrayRef { + let xs = (0..ROWS).map(|i| 300.0 * unit(i + 7) - 150.0).collect(); + let ys = (0..ROWS).map(|i| 300.0 * unit(i + 8) - 150.0).collect(); + point_column(xs, ys).unwrap() +} + +/// The constant point operand, at the origin so some squares contain it. +fn constant_point(ctx: &mut ExecutionCtx) -> ArrayRef { + let scalar = point_column(vec![0.0], vec![0.0]) + .unwrap() + .execute_scalar(0, ctx) + .unwrap(); + ConstantArray::new(scalar, ROWS).into_array() +} + +/// Wrap `array` with seeded random nulls at `density` percent. Zero density stays unwrapped, as a +/// non-nullable column would. +fn with_nulls(array: ArrayRef, seed: u64, density: usize) -> ArrayRef { + if density == 0 { + return array; + } + + let valid = (0..ROWS).map(|i| (splitmix64(seed ^ i as u64) % 100) >= density as u64); + MaskedArray::try_new(array, Validity::from_iter(valid)) + .unwrap() + .into_array() +} + +/// One arm over the operand pair: `Some` forces a strategy through the harness seam, `None` runs +/// the full pipeline with the per-batch selection. +fn bench_contains(bencher: Bencher, a: ArrayRef, b: ArrayRef, strategy: Option) { + let mut ctx = SESSION.create_execution_ctx(); + + bencher + .counter(ItemsCount::new(ROWS)) + .bench_local(|| match strategy { + None => SpatialContains::try_new_array(a.clone(), b.clone()) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap(), + Some(strategy) => execute_row_fn_with_strategy( + &SpatialContains, + &EmptyOptions, + vec![a.clone(), b.clone()], + ROWS, + strategy, + &mut ctx, + ) + .unwrap() + .execute::(&mut ctx) + .unwrap(), + }); +} + +/// Column of polygons CONTAINS constant point, nulls on the polygon column. +mod polygons_x_constant_point { + use super::*; + + fn operands(density: usize) -> (ArrayRef, ArrayRef) { + let mut ctx = SESSION.create_execution_ctx(); + (with_nulls(squares(), 1, density), constant_point(&mut ctx)) + } + + #[divan::bench(args = DENSITIES)] + fn filter(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::Filter)); + } + + #[divan::bench(args = DENSITIES)] + fn branch(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::BranchAndSkip)); + } + + #[divan::bench(args = DENSITIES)] + fn auto(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, None); + } +} + +/// Column of polygons CONTAINS column of points, independent nulls on both, so the conjoined +/// valid fraction is roughly `(1 - d)^2`. +mod polygons_x_points { + use super::*; + + fn operands(density: usize) -> (ArrayRef, ArrayRef) { + ( + with_nulls(squares(), 1, density), + with_nulls(points(), 2, density), + ) + } + + #[divan::bench(args = DENSITIES)] + fn filter(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::Filter)); + } + + #[divan::bench(args = DENSITIES)] + fn branch(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::BranchAndSkip)); + } + + #[divan::bench(args = DENSITIES)] + fn auto(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, None); + } +} diff --git a/vortex-spatial/src/extension/mod.rs b/vortex-spatial/src/extension/mod.rs index d1e2c37ebf4..e89ec15b500 100644 --- a/vortex-spatial/src/extension/mod.rs +++ b/vortex-spatial/src/extension/mod.rs @@ -178,6 +178,47 @@ pub(crate) fn geometries( } } +/// The geometry a null row decodes to under [`geometries_null_tolerant`]. Arbitrary: the caller +/// guarantees null rows are never read. +pub(crate) fn placeholder_geometry() -> Geometry { + Geometry::Point(geo_types::Point::new(0.0, 0.0)) +} + +/// Decode a native geometry column that may contain null rows, writing [`placeholder_geometry`] +/// into their slots. The caller guarantees null rows are never read. +/// +/// `Ok(None)` means this geometry type has no null-tolerant decode yet (`Point` and `Polygon` are +/// covered), and the caller falls back to the filter strategy, which never decodes a null row. A +/// column with definitely no nulls delegates to the ordinary [`geometries`] for any type. +pub(crate) fn geometries_null_tolerant( + array: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>>> { + if array.validity()?.definitely_no_nulls() { + return geometries(array, ctx).map(Some); + } + + let Some(ext) = array.dtype().as_extension_opt() else { + vortex_bail!( + "spatial: operand is not a geometry extension type, was {}", + array.dtype() + ); + }; + let storage = array + .clone() + .execute::(ctx)? + .storage_array() + .clone(); + + if ext.is::() { + point_geometries_null_tolerant(&storage, ctx).map(Some) + } else if ext.is::() { + polygon_geometries_null_tolerant(&storage, ctx).map(Some) + } else { + Ok(None) + } +} + /// Decode a constant operand scalar to one geometry, a constant of any /// supported geometry type is decoded exactly like a column. pub(crate) fn single_geometry( diff --git a/vortex-spatial/src/extension/point.rs b/vortex-spatial/src/extension/point.rs index e8f3ad3c169..8189fcba7bf 100644 --- a/vortex-spatial/src/extension/point.rs +++ b/vortex-spatial/src/extension/point.rs @@ -52,6 +52,7 @@ use super::coordinate::coordinate_from_struct; use super::coordinate::coordinate_storage_dtype; use super::geoarrow_metadata; use super::geoarrow_to_wkb; +use super::placeholder_geometry; use super::spatial_metadata_from_arrow; /// A single location: `geoarrow.point`, stored as `Struct` of non-nullable `f64`. @@ -149,6 +150,23 @@ pub(crate) fn point_geometries( .collect() } +/// Like [`point_geometries`], but a null row decodes to the placeholder geometry instead of +/// failing. The caller guarantees null rows are never read. +pub(crate) fn point_geometries_null_tolerant( + storage: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>> { + point_array(storage, ctx)? + .iter() + .map(|geometry| match geometry { + None => Ok(placeholder_geometry()), + Some(geometry) => Ok(geometry + .map_err(|e| vortex_err!("spatial: geometry access failed: {e}"))? + .to_geometry()), + }) + .collect() +} + impl ArrowExportVTable for Point { fn arrow_ext_id(&self) -> Id { *ARROW_POINT diff --git a/vortex-spatial/src/extension/polygon.rs b/vortex-spatial/src/extension/polygon.rs index dcfa8514ff3..362dfe311e9 100644 --- a/vortex-spatial/src/extension/polygon.rs +++ b/vortex-spatial/src/extension/polygon.rs @@ -52,6 +52,7 @@ use super::coordinate::coordinate_dimension; use super::coordinate::coordinate_storage_dtype; use super::geoarrow_metadata; use super::geoarrow_to_wkb; +use super::placeholder_geometry; use super::spatial_metadata_from_arrow; /// A polygon: `geoarrow.polygon`, stored as `List>>` (rings of vertices). @@ -131,6 +132,23 @@ pub(crate) fn polygon_geometries( .collect() } +/// Like [`polygon_geometries`], but a null row decodes to the placeholder geometry instead of +/// failing. The caller guarantees null rows are never read. +pub(crate) fn polygon_geometries_null_tolerant( + storage: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>> { + polygon_array(storage, ctx)? + .iter() + .map(|geometry| match geometry { + None => Ok(placeholder_geometry()), + Some(geometry) => Ok(geometry + .map_err(|e| vortex_err!("spatial: geometry access failed: {e}"))? + .to_geometry()), + }) + .collect() +} + /// Build a geoarrow `PolygonArray` from a `Polygon`'s `List>` storage. fn polygon_array(storage: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { let polygon_type = polygon_type( diff --git a/vortex-spatial/src/scalar_fn/contains.rs b/vortex-spatial/src/scalar_fn/contains.rs index 599c0eee2be..54316678f5f 100644 --- a/vortex-spatial/src/scalar_fn/contains.rs +++ b/vortex-spatial/src/scalar_fn/contains.rs @@ -3,44 +3,30 @@ //! `ST_Contains`: OGC containment test between two native geometries. +use std::cell::OnceCell; + +use geo::BoundingRect; use geo::Contains; +use geo::PreparedGeometry; +use geo::Relate; +use geo_types::Geometry; +use geo_types::Rect; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Contains`. -fn validate_contains_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: contains requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: contains operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; +#[cfg(test)] +use crate::scalar_fn::row::probe; /// OGC `ST_Contains` between two native geometry operands, each a column or a constant /// literal: true where operand `b` lies completely inside operand `a` (boundary contact alone @@ -59,83 +45,297 @@ impl SpatialContains { } } -impl ScalarFnVTable for SpatialContains { +impl RowFn for SpatialContains { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.contains"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { Ok(EmptyOptions) } - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) + /// Containment is not symmetric, so `a` is always the container and `b` the contained. + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |(a, b)| { + #[cfg(test)] + probe::record(a.is_some(), b.is_some()); + ConstOperands { + a: a.map(PreparedOperand::new), + b: b.map(PreparedOperand::new), + } + }, + |operands, (a, b), output| *output = contains_row_prepared(operands, a, b), + ) } +} + +/// Per-batch state for the contains row kernel: the prepared form of whichever operand is +/// constant for the batch. `None` marks an operand that varies by row. +struct ConstOperands { + /// Operand `a` (the container) when it is batch-constant. + a: Option, + + /// Operand `b` (the contained) when it is batch-constant. + b: Option, +} + +/// One batch-constant operand: the geometry cloned out of its decoded column (the state must not +/// borrow from the columns), plus its [`PreparedGeometry`], built on the first row whose pairing +/// routes through relate. +/// +/// The build is lazy because preparation (self-noding the topology graph plus an R*-tree over the +/// edges) costs `O(edges log edges)` and pays off only on relate-routed pairings; a batch of +/// point rows against a constant polygon never touches it, and preparing a large constant eagerly +/// would charge such a batch for nothing. +struct PreparedOperand { + /// The constant's decoded geometry, owned so [`prepared`](Self::prepared) can be `'static`. + geometry: Geometry, + + /// The constant's bounding rectangle, folded once for conservative row rejection. + bbox: Option>, + + /// The lazily built prepared form of [`geometry`](Self::geometry). + prepared: OnceCell, f64>>, +} - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("contains has exactly two children"), +impl PreparedOperand { + fn new(geometry: &Geometry) -> Self { + Self { + geometry: geometry.clone(), + bbox: geometry.bounding_rect(), + prepared: OnceCell::new(), } } - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_contains_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Bool(nullability)) + /// The prepared geometry, built on first use. + fn get(&self) -> &PreparedGeometry<'static, Geometry, f64> { + self.prepared + .get_or_init(|| PreparedGeometry::from(self.geometry.clone())) } +} - fn execute( - &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Containment is not symmetric: `a` is always the container and `b` the contained. A - // container's rect must cover the contained's rect (`Rect::contains` is the closed - // test), so a contained rect poking outside proves the row false. - execute_binary_geo_types( - &a, - &b, - |a, b| a.contains(b), - Some(|ra, rb| (!ra.contains(rb)).then_some(false)), - ctx, - ) - } +/// How geo's `a.contains(b)` computes its verdict for a pairing. +enum ContainsRoute { + /// `a.relate(b).is_contains()`. + ForwardRelate, - fn validity( - &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) + /// `b.relate(a).is_within()`, how geo phrases relate for `MultiPolygon` containers. + ReversedRelate, + + /// A direct algorithm (coordinate position, point arithmetic); nothing to prepare. + Direct, +} + +/// The route geo 0.31's `Contains` dispatch takes for `a.contains(b)`. +/// +/// The prepared substitution in [`contains_row_prepared`] **must** run relate exactly where geo +/// runs relate, with the same argument order, because geo's direct algorithms are not everywhere +/// bit-identical to a relate matrix query (they resolve degenerate and boundary cases with +/// different arithmetic). The relate rows below transcribe geo's `impl_contains_from_relate!` +/// lists per container type; everything else, notably every `Point`/`MultiPoint` contained side +/// and every `Point` container, is direct. +/// +/// **This table is coupled to the geo version.** It transcribes a dispatch that geo is free to +/// reshuffle in any release, and a wrong row is a silently wrong verdict rather than a build error. +/// The workspace therefore pins `geo = "=0.31.0"`: taking any new geo, patch releases included, is +/// a deliberate edit of that line, and the edit must re-verify this table against +/// `impl_contains_from_relate!`. +/// +/// `constant_operands_agree_with_columns` is the mechanical check, and it is **not** complete: it +/// compares the prepared route against plain `a.contains(b)` only for the container types it has +/// cases for. `routes_agree_with_geo_for_every_container` covers the rest, one representative +/// pairing per container variant, and is the one to extend when geo grows a geometry type. Both +/// stay green wherever relate and the direct algorithm agree, so neither replaces the pin. +fn contains_route(a: &Geometry, b: &Geometry) -> ContainsRoute { + use Geometry as G; + + match (a, b) { + // Line contains [Polygon, MultiLineString, MultiPolygon, GeometryCollection, Rect, + // Triangle]. + ( + G::Line(_), + G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // LineString contains [Polygon, MultiPoint, MultiLineString, MultiPolygon, + // GeometryCollection, Rect, Triangle]. + | ( + G::LineString(_), + G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // MultiLineString contains everything except Point. + | ( + G::MultiLineString(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // MultiPoint contains [Line, LineString, Polygon, MultiLineString, MultiPolygon, + // GeometryCollection, Rect, Triangle]. + | ( + G::MultiPoint(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // Polygon contains everything except Point and MultiPoint. + | ( + G::Polygon(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // Rect contains [Line, LineString, MultiPoint, MultiLineString, MultiPolygon, + // GeometryCollection, Triangle]; Rect contains Rect and Polygon are direct. + | ( + G::Rect(_), + G::Line(_) + | G::LineString(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Triangle(_), + ) + // Triangle and GeometryCollection contain everything except Point. + | ( + G::Triangle(_) | G::GeometryCollection(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) => ContainsRoute::ForwardRelate, + + // MultiPolygon contains everything except Point and MultiPoint, phrased reversed. + ( + G::MultiPolygon(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) => ContainsRoute::ReversedRelate, + + _ => ContainsRoute::Direct, } +} - fn is_strict(&self, _: &Self::Options) -> bool { - true +/// Computes one row of contains, substituting a prepared graph for a constant operand on the +/// pairings geo itself answers through relate. +/// +/// [`PreparedGeometry`] carries the operand's self-noded topology graph and edge R*-tree, so a +/// relate against it skips rebuilding both and reads its bounding rect from cache; geo asserts +/// the cached graph equal to a freshly built one (its `swap_arg_index` test), which is what makes +/// the substitution result-preserving. Before dispatch, a disjoint constant-side bounding rect +/// conservatively rejects the row, matching the columnar implementation's #9076 optimization. +/// All other rows delegate to the same direct or relate route as `a.contains(b)`. +fn contains_row_prepared(operands: &ConstOperands, a: &Geometry, b: &Geometry) -> bool { + let rejected = match (&operands.a, &operands.b) { + (None, None) => false, + (Some(const_a), Some(const_b)) => const_a + .bbox + .zip(const_b.bbox) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + (Some(const_a), None) => const_a + .bbox + .zip(b.bounding_rect()) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + (None, Some(const_b)) => a + .bounding_rect() + .zip(const_b.bbox) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + }; + + if rejected { + return false; } - fn is_fallible(&self, _: &Self::Options) -> bool { - false + match contains_route(a, b) { + ContainsRoute::Direct => a.contains(b), + ContainsRoute::ForwardRelate => match (&operands.a, &operands.b) { + (Some(const_a), Some(const_b)) => const_a.get().relate(const_b.get()).is_contains(), + (Some(const_a), None) => const_a.get().relate(b).is_contains(), + (None, Some(const_b)) => a.relate(const_b.get()).is_contains(), + (None, None) => a.contains(b), + }, + ContainsRoute::ReversedRelate => match (&operands.a, &operands.b) { + (Some(const_a), Some(const_b)) => const_b.get().relate(const_a.get()).is_within(), + (Some(const_a), None) => b.relate(const_a.get()).is_within(), + (None, Some(const_b)) => const_b.get().relate(a).is_within(), + (None, None) => a.contains(b), + }, } } #[cfg(test)] mod tests { + use geo::Contains; + use geo_types::Coord; use geo_types::Geometry; + use geo_types::GeometryCollection; + use geo_types::Line; use geo_types::LineString; + use geo_types::MultiLineString; + use geo_types::MultiPoint; + use geo_types::MultiPolygon; use geo_types::Point; use geo_types::Polygon; + use geo_types::Rect; + use geo_types::Triangle; use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::Canonical; @@ -144,23 +344,31 @@ mod tests { use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; + use vortex_array::arrays::MaskedArray; use vortex_array::assert_arrays_eq; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::EmptyOptions; + use vortex_array::scalar_fn::NullStrategy; use vortex_array::scalar_fn::ScalarFnVTable; + use vortex_array::scalar_fn::execute_row_fn_with_strategy; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; use vortex_error::VortexResult; use vortex_error::vortex_err; use wkb::writer::WriteOptions; + use super::ConstOperands; + use super::PreparedOperand; use super::SpatialContains; + use super::contains_row_prepared; + use crate::scalar_fn::row::probe::assert_prepared_agrees_with_columns; use crate::test_harness::linestring_column; use crate::test_harness::nullable_point_column; use crate::test_harness::point_column; + use crate::test_harness::polygon_column; /// A rectangle polygon with corners `(x0, y0)` and `(x1, y1)`, no holes. fn rect_polygon(x0: f64, y0: f64, x1: f64, y1: f64) -> Polygon { @@ -244,6 +452,20 @@ mod tests { assert_contains(container, points, [true, false, false]) } + /// Constant container vs a linestring column: a row whose bounding rect pokes outside the + /// container's is not contained, while one wholly inside is. Carried over from the columnar + /// bounding-rect rejection in #9076, since it constrains the verdict rather than the mechanism. + #[test] + fn constant_container_vs_row_rect_poking_outside() -> VortexResult<()> { + let container = geometry_constant(&Geometry::Polygon(rect_polygon(0.0, 0.0, 4.0, 4.0)), 3)?; + let lines = linestring_column(vec![ + vec![(1.0, 1.0), (3.0, 3.0)], + vec![(1.0, 1.0), (9.0, 1.0)], + vec![(5.0, 5.0), (9.0, 9.0)], + ])?; + assert_contains(container, lines, [true, false, false]) + } + /// Polygon column vs constant point: only the polygon around the point contains it. #[test] fn polygon_column_vs_constant_point() -> VortexResult<()> { @@ -264,20 +486,6 @@ mod tests { assert_contains(away, point, [false; 2]) } - /// Constant container vs a linestring column: a row whose bounding rect pokes outside the - /// container's rect is proven false by the rect pre-check alone; a fully inside row still - /// needs (and passes) the exact test. - #[test] - fn constant_container_vs_row_rect_poking_outside() -> VortexResult<()> { - let container = geometry_constant(&Geometry::Polygon(rect_polygon(0.0, 0.0, 4.0, 4.0)), 3)?; - let lines = linestring_column(vec![ - vec![(1.0, 1.0), (3.0, 3.0)], - vec![(1.0, 1.0), (9.0, 1.0)], - vec![(5.0, 5.0), (9.0, 9.0)], - ])?; - assert_contains(container, lines, [true, false, false]) - } - /// Column vs column pairs rows: each polygon row is tested against the point row at the /// same position. #[test] @@ -408,6 +616,117 @@ mod tests { Ok(()) } + /// A nullable polygon column: unit squares at `centers`, the rows where `nulls` is true + /// masked out, spelled as `Masked` over non-nullable storage. + fn nullable_squares(centers: &[(f64, f64)], nulls: &[bool]) -> VortexResult { + let squares = centers + .iter() + .map(|&(x, y)| { + vec![vec![ + (x - 1.0, y - 1.0), + (x + 1.0, y - 1.0), + (x + 1.0, y + 1.0), + (x - 1.0, y + 1.0), + (x - 1.0, y - 1.0), + ]] + }) + .collect(); + let polygons = polygon_column(squares)?; + + Ok( + MaskedArray::try_new(polygons, Validity::from_iter(nulls.iter().map(|n| !n)))? + .into_array(), + ) + } + + /// Executes `SpatialContains(a, b)` with a forced null strategy, canonicalized. + fn contains_forced( + a: &ArrayRef, + b: &ArrayRef, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + Ok(execute_row_fn_with_strategy( + &SpatialContains, + &EmptyOptions, + vec![a.clone(), b.clone()], + a.len(), + strategy, + ctx, + )? + .execute::(ctx)? + .into_array()) + } + + /// The branch-and-skip and filter strategies, plus the automatic per-batch selection, must + /// return identical arrays for nullable geometry operands: `Masked` polygons against nullable + /// points, with independent nulls conjoined. + #[test] + fn branch_matches_filter_for_nullable_geometries() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + let centers = [(0.0, 0.0), (5.0, 5.0), (0.5, -0.2), (9.0, 9.0), (0.0, 1.0)]; + let nulls = [false, true, false, false, true]; + let polygons = nullable_squares(¢ers, &nulls)?; + let points = nullable_point_column(vec![ + Some((0.0, 0.0)), + Some((5.0, 5.0)), + None, + Some((0.0, 0.0)), + Some((0.0, 1.0)), + ])?; + + let filtered = contains_forced(&polygons, &points, NullStrategy::Filter, &mut ctx)?; + let branched = contains_forced(&polygons, &points, NullStrategy::BranchAndSkip, &mut ctx)?; + let auto = SpatialContains::try_new_array(polygons, points)? + .into_array() + .execute::(&mut ctx)? + .into_array(); + + assert_arrays_eq!(branched, filtered, &mut ctx); + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) + } + + /// Geometry types without a null-tolerant decode refuse the branch strategy: forcing it is an + /// error, and the automatic selection (which prefers branch at this density) silently falls + /// back to filtering with the correct result. + #[test] + fn unsupported_geometry_falls_back_to_filter() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + // Four rows with one null: 75% surviving, so the selection prefers branch. + let lines = MaskedArray::try_new( + linestring_column(vec![ + vec![(0.0, 0.0), (4.0, 4.0)], + vec![(0.0, 0.0), (1.0, 1.0)], + vec![(2.0, 2.0), (3.0, 3.0)], + vec![(0.0, 4.0), (4.0, 0.0)], + ])?, + Validity::from_iter([true, false, true, true]), + )? + .into_array(); + let point = geometry_constant(&Geometry::Point(Point::new(2.0, 2.0)), 4)?; + + let error = contains_forced(&lines, &point, NullStrategy::BranchAndSkip, &mut ctx) + .expect_err("a linestring column with nulls has no branch decode"); + assert!( + error.to_string().contains("branch-and-skip"), + "unexpected error: {error}" + ); + + let filtered = contains_forced(&lines, &point, NullStrategy::Filter, &mut ctx)?; + let auto = SpatialContains::try_new_array(lines, point)? + .into_array() + .execute::(&mut ctx)? + .into_array(); + + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) + } + /// A non-geometry operand dtype is rejected up front, before execution. #[test] fn non_geometry_operand_is_rejected() -> VortexResult<()> { @@ -417,4 +736,166 @@ mod tests { assert!(result.is_err()); Ok(()) } + + // The prepared-vs-expanded agreement grid: every constant arrangement of a pairing must + // return exactly what the fully expanded columns return. + + /// A point geometry. + fn point(x: f64, y: f64) -> Geometry { + Geometry::Point(Point::new(x, y)) + } + + /// A linestring geometry through `coords`. + fn line(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::LineString(LineString::from(coords)) + } + + /// A multipoint geometry over `coords`. + fn multipoint(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::MultiPoint(MultiPoint::from(coords)) + } + + /// A two-point line segment geometry, the `Line` container variant. + fn line_geometry(start: (f64, f64), end: (f64, f64)) -> Geometry { + Geometry::Line(Line::new( + Coord { + x: start.0, + y: start.1, + }, + Coord { x: end.0, y: end.1 }, + )) + } + + /// A multilinestring geometry over one linestring per entry of `parts`. + fn multilinestring(parts: Vec>) -> Geometry { + Geometry::MultiLineString(MultiLineString::new( + parts.into_iter().map(LineString::from).collect(), + )) + } + + /// A geometry collection wrapping `parts`. + fn collection(parts: Vec) -> Geometry { + Geometry::GeometryCollection(GeometryCollection::from(parts)) + } + + /// An axis-aligned rectangle geometry, the `Rect` container variant. + fn rect_geometry(x0: f64, y0: f64, x1: f64, y1: f64) -> Geometry { + Geometry::Rect(Rect::new(Coord { x: x0, y: y0 }, Coord { x: x1, y: y1 })) + } + + /// A triangle geometry large enough to contain the small test polygons. + fn triangle_geometry() -> Geometry { + Geometry::Triangle(Triangle::new( + Coord { x: 0.0, y: 0.0 }, + Coord { x: 8.0, y: 0.0 }, + Coord { x: 0.0, y: 8.0 }, + )) + } + + /// A two-part multipolygon: `4x4` squares at the origin and at `(10, 10)`. + fn two_part_multipolygon() -> Geometry { + Geometry::MultiPolygon(MultiPolygon::new(vec![ + rect_polygon(0.0, 0.0, 4.0, 4.0), + rect_polygon(10.0, 10.0, 14.0, 14.0), + ])) + } + + /// Every container variant `contains_route` distinguishes, checked against plain + /// `a.contains(b)` in all four constant arrangements. + /// + /// Every case is a containment geo answers `true`, which the test asserts: a pairing that is + /// false regardless of route (a lower-dimensional container, say) also agrees regardless of + /// route, and pins nothing. A true case fails when the prepared substitution diverges from + /// geo — a table row whose relate phrasing disagrees with geo's dispatch on this input, or a + /// bounding-rect prescreen that wrongly rejects a contained row. It is **not** a version + /// tripwire: a geo release that reshuffles its dispatch stays green wherever relate and the + /// direct algorithm agree, which is why the workspace pins `geo` exactly. + /// + /// This is the table's own regression, and the one to extend when geo grows a geometry type: + /// `constant_operands_agree_with_columns` below goes through real arrays and so is the better + /// end-to-end check, but it only covers the container types it has cases for, and WKB decoding + /// limits which types those can be. The MultiPoint and Line containers route relate only for + /// contained types a MultiPoint or Line can rarely contain, so their true cases lean on + /// `GeometryCollection` membership and collinear `MultiLineString` parts respectively. + #[rstest] + #[case::point(point(1.0, 1.0), point(1.0, 1.0))] + #[case::line(line_geometry((0.0, 0.0), (4.0, 4.0)), point(2.0, 2.0))] + #[case::line_x_multilinestring(line_geometry((0.0, 0.0), (4.0, 4.0)), multilinestring(vec![vec![(1.0, 1.0), (2.0, 2.0)]]))] + #[case::linestring(line(vec![(0.0, 0.0), (4.0, 4.0)]), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::multipoint(multipoint(vec![(0.0, 0.0), (2.0, 2.0), (4.0, 4.0)]), collection(vec![point(2.0, 2.0)]))] + #[case::multilinestring(multilinestring(vec![vec![(0.0, 0.0), (4.0, 4.0)]]), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipolygon(two_part_multipolygon(), rect_polygon(1.0, 1.0, 3.0, 3.0).into())] + #[case::geometrycollection(collection(vec![rect_polygon(0.0, 0.0, 8.0, 8.0).into()]), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::rect(rect_geometry(0.0, 0.0, 8.0, 8.0), line(vec![(2.0, 2.0), (4.0, 4.0)]))] + #[case::triangle(triangle_geometry(), rect_polygon(1.0, 1.0, 2.0, 2.0).into())] + fn routes_agree_with_geo_for_every_container(#[case] a: Geometry, #[case] b: Geometry) { + let expected = a.contains(&b); + assert!( + expected, + "route cases must be containments geo answers true, or every route agrees vacuously", + ); + + let arrangements = [ + (None, None), + (Some(PreparedOperand::new(&a)), None), + (None, Some(PreparedOperand::new(&b))), + ( + Some(PreparedOperand::new(&a)), + Some(PreparedOperand::new(&b)), + ), + ]; + + for (index, (const_a, const_b)) in arrangements.into_iter().enumerate() { + let operands = ConstOperands { + a: const_a, + b: const_b, + }; + assert_eq!( + contains_row_prepared(&operands, &a, &b), + expected, + "arrangement {index} disagrees with geo's own contains", + ); + } + } + + /// Constant arrangements agree with expanded columns across the routes the prepared kernel + /// distinguishes: forward relate (polygon, linestring and multipoint containers), reversed + /// relate (multipolygon containers), and the direct pairings (a point on either side, + /// multipoint over multipoint, polygon over multipoint), including boundary contact, + /// crossing, disjoint and empty cases. + #[rstest] + #[case::polygon_nested_polygon(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::polygon_touching_from_inside(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(0.0, 2.0, 2.0, 4.0).into())] + #[case::polygon_overlapping_polygon(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(2.0, 2.0, 6.0, 6.0).into())] + #[case::polygon_disjoint_polygon(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::polygon_x_point_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(2.0, 2.0))] + #[case::polygon_x_point_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(0.0, 2.0))] + #[case::polygon_x_point_outside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(20.0, 20.0))] + #[case::polygon_x_nan_point(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(f64::NAN, 2.0))] + #[case::polygon_x_linestring_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon_x_linestring_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(0.0, 1.0), (0.0, 3.0)]))] + #[case::polygon_x_linestring_crossing(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(-2.0, 2.0), (2.0, 2.0)]))] + #[case::polygon_x_empty_linestring(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![]))] + #[case::polygon_x_multipoint_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon_x_multipoint_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), multipoint(vec![(0.0, 1.0), (0.0, 3.0)]))] + #[case::linestring_x_multipoint_on_line(line(vec![(0.0, 0.0), (4.0, 4.0)]), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipoint_x_multipoint_subset(multipoint(vec![(0.0, 0.0), (2.0, 2.0), (4.0, 4.0)]), multipoint(vec![(2.0, 2.0)]))] + #[case::multipoint_x_linestring_between_points(multipoint(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipolygon_x_polygon_in_one_part(two_part_multipolygon(), rect_polygon(1.0, 1.0, 3.0, 3.0).into())] + #[case::multipolygon_x_polygon_straddling(two_part_multipolygon(), rect_polygon(3.0, 3.0, 11.0, 11.0).into())] + #[case::multipolygon_x_polygon_disjoint(two_part_multipolygon(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::multipolygon_x_point_inside(two_part_multipolygon(), point(11.0, 11.0))] + #[case::point_x_point_equal(point(1.0, 1.0), point(1.0, 1.0))] + #[case::point_x_polygon(point(2.0, 2.0), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + fn constant_operands_agree_with_columns( + #[case] a: Geometry, + #[case] b: Geometry, + ) -> VortexResult<()> { + assert_prepared_agrees_with_columns( + SpatialContains::try_new_array, + geometry_constant(&a, 3)?, + geometry_constant(&b, 3)?, + ) + } } diff --git a/vortex-spatial/src/scalar_fn/distance.rs b/vortex-spatial/src/scalar_fn/distance.rs index dfd3d09ed23..e8de808bd32 100644 --- a/vortex-spatial/src/scalar_fn/distance.rs +++ b/vortex-spatial/src/scalar_fn/distance.rs @@ -6,43 +6,19 @@ use geo::Distance; use geo::Euclidean; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Distance`. -fn validate_distance_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: distance requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: distance operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; /// Planar (Euclidean) `ST_Distance` (no geodesic correction) between two native geometry /// operands, each a column or a constant literal. @@ -60,66 +36,45 @@ impl SpatialDistance { } } -impl ScalarFnVTable for SpatialDistance { +impl RowFn for SpatialDistance { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.distance"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { - Ok(EmptyOptions) - } - - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) - } - - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("distance has exactly two children"), - } - } - - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_distance_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Primitive(PType::F64, nullability)) - } - - fn execute( + fn deserialize( &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Distance is a value, not a verdict: no bounding-rect test can decide it. - execute_binary_geo_types(&a, &b, |x, y| Euclidean.distance(x, y), None, ctx) + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn validity( + /// Deliberately uses unit preparation: a batch-constant operand offers nothing + /// sound to hoist. geo computes linestring and polygon distances through its private + /// `nearest_neighbour_distance`, which builds the R*-trees for *both* sides inside each call, + /// and the point pairings are single expressions; reusing a tree across rows would mean + /// reimplementing geo's internals. A batch where both operands are constant already folds to + /// a single-row execution before the row loop. + fn dispatch( &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) - } - - fn is_strict(&self, _: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _: &Self::Options) -> bool { - false + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |_| (), + |&(), (a, b), output| *output = Euclidean.distance(a, b), + ) } } @@ -196,8 +151,9 @@ mod tests { Ok(()) } - /// Distance passes no bounding-rect rejection: a point far outside a constant polygon's - /// bounding rect still gets its true distance, alongside an inside point at distance zero. + /// Distance is a value rather than a verdict, so no bounding-rect rejection may fire for it: a + /// point far outside a constant polygon's rect still gets its true distance. Carried over from + /// #9076, which added the rejection to the predicates but deliberately not to this function. #[test] fn distance_to_constant_polygon_is_exact() -> VortexResult<()> { let session = vortex_array::array_session(); diff --git a/vortex-spatial/src/scalar_fn/execute.rs b/vortex-spatial/src/scalar_fn/execute.rs index ca5b4018249..836577ec26e 100644 --- a/vortex-spatial/src/scalar_fn/execute.rs +++ b/vortex-spatial/src/scalar_fn/execute.rs @@ -1,24 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Shared execution for native geometry scalar functions. -//! -//! [`dispatch_unary`] and the binary dispatcher handle constant/column operands and strict null -//! propagation without prescribing how a kernel represents geometries or builds its output. -//! Native columnar kernels such as `ST_Envelope` use the unary dispatcher directly. -//! -//! [`execute_binary_geo_types`] adapts row-oriented algorithms from the `geo` ecosystem. It decodes -//! valid inputs into `geo_types::Geometry`; the final output is still a Vortex [`ArrayRef`], such -//! as an `f64` or boolean array. +//! Shared unary execution for native geometry scalar functions. -mod binary; -mod geo_types; mod unary; -pub(crate) use binary::execute_binary_geo_types; pub(crate) use unary::dispatch_unary; use vortex_array::ArrayRef; -use vortex_array::dtype::Nullability; use vortex_array::scalar::Scalar; use vortex_mask::Mask; @@ -31,9 +19,6 @@ pub(crate) enum Operand { } /// Shared batch state presented to a null-propagating geometry kernel with `N` operands. -/// -/// Binary kernels use the default materialized [`Mask`]. Unary columnar kernels can instead -/// retain a lazy [`vortex_array::validity::Validity`] until they need row-wise access. pub(crate) struct Execution { /// Constant/column shape of each operand. pub(crate) operands: [Operand; N], @@ -41,6 +26,4 @@ pub(crate) struct Execution { pub(crate) valid: V, /// Number of output rows. pub(crate) len: usize, - /// Output nullability from the scalar function's return dtype. - pub(crate) nullability: Nullability, } diff --git a/vortex-spatial/src/scalar_fn/execute/binary.rs b/vortex-spatial/src/scalar_fn/execute/binary.rs deleted file mode 100644 index f2c03bd1beb..00000000000 --- a/vortex-spatial/src/scalar_fn/execute/binary.rs +++ /dev/null @@ -1,334 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Binary pairwise dispatch, plus an adapter for row-oriented `geo_types` kernels. - -use geo::BoundingRect; -use geo_types::Geometry; -use geo_types::Rect; -use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::Constant; -use vortex_array::arrays::ConstantArray; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::scalar::Scalar; -use vortex_error::VortexResult; -use vortex_mask::Mask; - -use super::Execution; -use super::Operand; -use super::geo_types::GeoTypesOutput; -use super::geo_types::eval_column; -use super::geo_types::eval_column_pair; -use crate::extension::single_geometry; - -/// Dispatch a binary strict geometry kernel over constants and columns. -/// -/// A null constant or an empty combined validity mask short-circuits to an all-null constant -/// output. Otherwise, `kernel` receives both operand shapes and the mask of rows where both are -/// valid. Two columns are always paired by row index. The kernel remains responsible for physical -/// input interpretation and Vortex output construction. -pub(crate) fn dispatch_binary( - left: &ArrayRef, - right: &ArrayRef, - output_dtype: DType, - kernel: K, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - K: FnOnce(Execution<2>, &mut ExecutionCtx) -> VortexResult, -{ - let len = left.len(); - for operand in [left, right] { - if operand - .as_opt::() - .is_some_and(|constant| constant.scalar().is_null()) - { - return Ok(ConstantArray::new(Scalar::null(output_dtype), len).into_array()); - } - } - - let (left, right, valid) = match (left.as_opt::(), right.as_opt::()) { - (Some(left), Some(right)) => ( - Operand::Constant(left.scalar().clone()), - Operand::Constant(right.scalar().clone()), - Mask::new_true(len), - ), - (Some(left), None) => ( - Operand::Constant(left.scalar().clone()), - Operand::Column(right.clone()), - right.validity()?.execute_mask(len, ctx)?, - ), - (None, Some(right)) => ( - Operand::Column(left.clone()), - Operand::Constant(right.scalar().clone()), - left.validity()?.execute_mask(len, ctx)?, - ), - (None, None) => { - let left_valid = left.validity()?.execute_mask(len, ctx)?; - let right_valid = right.validity()?.execute_mask(len, ctx)?; - ( - Operand::Column(left.clone()), - Operand::Column(right.clone()), - &left_valid & &right_valid, - ) - } - }; - - if len != 0 && valid.all_false() { - return Ok(ConstantArray::new(Scalar::null(output_dtype), len).into_array()); - } - kernel( - Execution { - operands: [left, right], - valid, - len, - nullability: output_dtype.nullability(), - }, - ctx, - ) -} - -/// A bounding-rectangle pre-check for [`execute_binary_geo_types`]'s one-constant paths. -/// -/// Called per row with rectangles in operand order, it returns `Some(result)` when they prove the -/// result and `None` when the exact kernel must run. -pub(crate) type BboxPrecheck = fn(&Rect, &Rect) -> Option; - -/// Run a binary row-oriented kernel whose inputs are decoded to `geo_types::Geometry`. -/// -/// The `geo_types` name describes the values passed to `compute`, not the output. `T` is converted -/// into a Vortex array before this function returns. Nulls propagate from either operand. With -/// exactly one constant operand, `bbox_precheck` may prove a result from the fixed constant -/// bounding rectangle and the current row's rectangle before the exact kernel runs. -pub(crate) fn execute_binary_geo_types( - left: &ArrayRef, - right: &ArrayRef, - compute: F, - bbox_precheck: Option>, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry, &Geometry) -> T + Copy, -{ - let nullability = Nullability::from(left.dtype().is_nullable() || right.dtype().is_nullable()); - dispatch_binary( - left, - right, - T::dtype(nullability), - |execution, ctx| match execution.operands { - [Operand::Constant(left), Operand::Constant(right)] => { - let left = single_geometry(&left, ctx)?; - let right = single_geometry(&right, ctx)?; - Ok(ConstantArray::new( - compute(&left, &right).into_scalar(execution.nullability), - execution.len, - ) - .into_array()) - } - [Operand::Constant(left), Operand::Column(right)] => { - let left = single_geometry(&left, ctx)?; - let prescreen = bbox_precheck.zip(left.bounding_rect()); - eval_column( - &right, - &execution.valid, - |right| { - prescreen - .and_then(|(precheck, fixed)| precheck(&fixed, &right.bounding_rect()?)) - .unwrap_or_else(|| compute(&left, right)) - }, - execution.nullability, - ctx, - ) - } - [Operand::Column(left), Operand::Constant(right)] => { - let right = single_geometry(&right, ctx)?; - let prescreen = bbox_precheck.zip(right.bounding_rect()); - eval_column( - &left, - &execution.valid, - |left| { - prescreen - .and_then(|(precheck, fixed)| precheck(&left.bounding_rect()?, &fixed)) - .unwrap_or_else(|| compute(left, &right)) - }, - execution.nullability, - ctx, - ) - } - [Operand::Column(left), Operand::Column(right)] => eval_column_pair( - &left, - &right, - &execution.valid, - compute, - execution.nullability, - ctx, - ), - }, - ctx, - ) -} - -#[cfg(test)] -mod tests { - use std::cell::Cell; - - use geo::Contains; - use geo::Intersects; - use geo_types::Geometry; - use vortex_array::ArrayRef; - use vortex_array::ExecutionCtx; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::BoolArray; - use vortex_array::arrays::ConstantArray; - use vortex_array::assert_arrays_eq; - use vortex_array::validity::Validity; - use vortex_buffer::BitBuffer; - use vortex_error::VortexResult; - - use super::BboxPrecheck; - use super::execute_binary_geo_types; - use crate::test_harness::linestring_column; - use crate::test_harness::nullable_point_column; - use crate::test_harness::point_column; - use crate::test_harness::polygon_column; - - const DISJOINT_PRECHECK: BboxPrecheck = - |left, right| (!left.intersects(right)).then_some(false); - - fn triangle_constant(len: usize, ctx: &mut ExecutionCtx) -> VortexResult { - let ring = vec![(0.0, 0.0), (10.0, 0.0), (0.0, 10.0), (0.0, 0.0)]; - let scalar = polygon_column(vec![vec![ring]])?.execute_scalar(0, ctx)?; - Ok(ConstantArray::new(scalar, len).into_array()) - } - - fn counting_intersects( - counter: &Cell, - ) -> impl Fn(&Geometry, &Geometry) -> bool + Copy { - move |left, right| { - counter.set(counter.get() + 1); - left.intersects(right) - } - } - - #[test] - fn bbox_precheck_skips_exact_test() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(3, &mut ctx)?; - let probes = point_column(vec![50.0, 8.0, 2.0], vec![50.0, 8.0, 2.0])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &triangle, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false, true]), &mut ctx); - assert_eq!(exact_runs.get(), 2); - Ok(()) - } - - #[test] - fn bbox_precheck_leaves_nulls_alone() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(3, &mut ctx)?; - let probes = nullable_point_column(vec![Some((50.0, 50.0)), None, Some((2.0, 2.0))])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &triangle, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - let expected = BoolArray::new( - BitBuffer::from_iter([false, false, true]), - Validity::from_iter([true, false, true]), - ) - .into_array(); - - assert_arrays_eq!(result, expected, &mut ctx); - assert_eq!(exact_runs.get(), 1); - Ok(()) - } - - #[test] - fn bbox_precheck_sees_rects_in_operand_order() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let probes = point_column(vec![2.0, 50.0], vec![2.0, 50.0])?; - let triangle = triangle_constant(2, &mut ctx)?; - let exact_runs = Cell::new(0); - let counted = |left: &Geometry, right: &Geometry| { - exact_runs.set(exact_runs.get() + 1); - left.contains(right) - }; - - let result = execute_binary_geo_types( - &probes, - &triangle, - counted, - Some(|left, right| (!left.contains(right)).then_some(false)), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false]), &mut ctx); - assert_eq!(exact_runs.get(), 0); - Ok(()) - } - - #[test] - fn empty_constant_falls_through_to_exact() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let scalar = linestring_column(vec![vec![]])?.execute_scalar(0, &mut ctx)?; - let empty = ConstantArray::new(scalar, 2).into_array(); - let probes = point_column(vec![2.0, 50.0], vec![2.0, 50.0])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &empty, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false]), &mut ctx); - assert_eq!(exact_runs.get(), 2); - Ok(()) - } - - #[test] - fn bbox_precheck_matches_exact_results() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(6, &mut ctx)?; - let probes = nullable_point_column(vec![ - Some((50.0, 50.0)), - Some((8.0, 8.0)), - Some((2.0, 2.0)), - None, - Some((0.0, 0.0)), - Some((10.0, 0.0)), - ])?; - let exact = |left: &Geometry, right: &Geometry| left.intersects(right); - - let with_precheck = - execute_binary_geo_types(&triangle, &probes, exact, Some(DISJOINT_PRECHECK), &mut ctx)?; - let exact_only = execute_binary_geo_types(&triangle, &probes, exact, None, &mut ctx)?; - - assert_arrays_eq!(with_precheck, exact_only, &mut ctx); - Ok(()) - } -} diff --git a/vortex-spatial/src/scalar_fn/execute/geo_types.rs b/vortex-spatial/src/scalar_fn/execute/geo_types.rs deleted file mode 100644 index 038aca46502..00000000000 --- a/vortex-spatial/src/scalar_fn/execute/geo_types.rs +++ /dev/null @@ -1,144 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Shared input decoding and Vortex output construction for `geo_types` kernels. -//! -//! `geo_types` is the row representation consumed by the kernel. These helpers always construct -//! and return Vortex arrays; they do not expose `geo_types` values as scalar-function outputs. - -use geo_types::Geometry; -use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::BoolArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::scalar::Scalar; -use vortex_array::validity::Validity; -use vortex_buffer::BitBuffer; -use vortex_error::VortexResult; -use vortex_mask::AllOr; -use vortex_mask::Mask; - -use crate::extension::geometries; - -/// A primitive result produced after kernel inputs are decoded to `geo_types`. -pub(crate) trait GeoTypesOutput: Copy { - /// The Vortex dtype used to represent this output. - fn dtype(nullability: Nullability) -> DType; - - /// Convert one computed value into a Vortex scalar for constant output. - fn into_scalar(self, nullability: Nullability) -> Scalar; - - /// Scatter values computed for valid rows into a full-length output array. - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef; -} - -impl GeoTypesOutput for f64 { - fn dtype(nullability: Nullability) -> DType { - DType::Primitive(PType::F64, nullability) - } - - fn into_scalar(self, nullability: Nullability) -> Scalar { - Scalar::primitive(self, nullability) - } - - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef { - let validity = Validity::from_mask(valid.clone(), nullability); - match valid.indices() { - AllOr::All => PrimitiveArray::new(values, validity).into_array(), - AllOr::None => PrimitiveArray::new(vec![0.0f64; len], validity).into_array(), - AllOr::Some(rows) => { - let mut data = vec![0.0f64; len]; - for (&row, value) in rows.iter().zip(values) { - data[row] = value; - } - PrimitiveArray::new(data, validity).into_array() - } - } - } -} - -impl GeoTypesOutput for bool { - fn dtype(nullability: Nullability) -> DType { - DType::Bool(nullability) - } - - fn into_scalar(self, nullability: Nullability) -> Scalar { - Scalar::bool(self, nullability) - } - - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef { - let validity = Validity::from_mask(valid.clone(), nullability); - match valid.indices() { - AllOr::All => BoolArray::new(BitBuffer::from_iter(values), validity).into_array(), - AllOr::None => BoolArray::new(BitBuffer::new_unset(len), validity).into_array(), - AllOr::Some(rows) => { - let mut data = vec![false; len]; - for (&row, value) in rows.iter().zip(values) { - data[row] = value; - } - BoolArray::new(BitBuffer::from_iter(data), validity).into_array() - } - } - } -} - -/// Evaluate a decoded kernel over each valid row of one geometry column. -pub(super) fn eval_column( - column: &ArrayRef, - valid: &Mask, - compute: F, - nullability: Nullability, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry) -> T, -{ - let len = column.len(); - let decoded = geometries(&column.filter(valid.clone())?, ctx)?; - let values = decoded.iter().map(compute).collect(); - Ok(T::build_array(len, valid, values, nullability)) -} - -/// Evaluate a decoded kernel over rows where both geometry columns are valid. -pub(super) fn eval_column_pair( - left: &ArrayRef, - right: &ArrayRef, - valid: &Mask, - compute: F, - nullability: Nullability, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry, &Geometry) -> T, -{ - let len = left.len(); - let left = geometries(&left.filter(valid.clone())?, ctx)?; - let right = geometries(&right.filter(valid.clone())?, ctx)?; - let values = left - .iter() - .zip(&right) - .map(|(left, right)| compute(left, right)) - .collect(); - Ok(T::build_array(len, valid, values, nullability)) -} diff --git a/vortex-spatial/src/scalar_fn/execute/unary.rs b/vortex-spatial/src/scalar_fn/execute/unary.rs index bdbbd0b33ac..478c62eef50 100644 --- a/vortex-spatial/src/scalar_fn/execute/unary.rs +++ b/vortex-spatial/src/scalar_fn/execute/unary.rs @@ -40,7 +40,6 @@ where operands: [Operand::Constant(constant.scalar().clone())], valid: Validity::AllValid, len, - nullability: output_dtype.nullability(), }, ctx, ); @@ -55,7 +54,6 @@ where operands: [Operand::Column(array.clone())], valid, len, - nullability: output_dtype.nullability(), }, ctx, ) diff --git a/vortex-spatial/src/scalar_fn/intersects.rs b/vortex-spatial/src/scalar_fn/intersects.rs index bdabd2b9967..3694303e954 100644 --- a/vortex-spatial/src/scalar_fn/intersects.rs +++ b/vortex-spatial/src/scalar_fn/intersects.rs @@ -3,44 +3,26 @@ //! `ST_Intersects`: OGC intersection test between two native geometries. +use geo::BoundingRect; use geo::Intersects; +use geo_types::Geometry; +use geo_types::Rect; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Intersects`. -fn validate_intersects_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: intersects requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: intersects operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; +#[cfg(test)] +use crate::scalar_fn::row::probe; /// OGC `ST_Intersects` (not disjoint; boundary contact counts) between two native geometry /// operands, each a column or a constant literal. @@ -58,74 +40,97 @@ impl SpatialIntersects { } } -impl ScalarFnVTable for SpatialIntersects { +impl RowFn for SpatialIntersects { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.intersects"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { Ok(EmptyOptions) } - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) - } - - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("intersects has exactly two children"), - } - } - - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_intersects_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Bool(nullability)) - } - - fn execute( + fn dispatch( &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Disjoint bounding rects prove the geometries disjoint; rect contact (closed test) - // falls through to the exact test. - execute_binary_geo_types( - &a, - &b, - |x, y| x.intersects(y), - Some(|ra, rb| (!ra.intersects(rb)).then_some(false)), - ctx, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |(a, b)| { + #[cfg(test)] + probe::record(a.is_some(), b.is_some()); + ConstBboxes::new(a, b) + }, + |bboxes, (a, b), output| *output = intersects_row_prepared(bboxes, a, b), ) } +} - fn validity( - &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) - } +/// Per-batch state for the intersects row kernel: the bounding rect of each operand that is +/// constant for the batch. +/// +/// geo opens many intersects pairings with `has_disjoint_bboxes`, an early-out that folds +/// [`bounding_rect`] over both operands. For a batch-constant operand that fold recomputes the +/// same rect every row, so it is hoisted here and [`intersects_row_prepared`] replays the +/// comparison with the hoisted value. `None` marks an operand that varies by row or has no +/// bounding rect (an empty geometry); both mean no early-out, exactly as `has_disjoint_bboxes` +/// treats a missing rect. +/// +/// [`bounding_rect`]: BoundingRect::bounding_rect +struct ConstBboxes { + /// The bounding rect of operand `a` when it is batch-constant. + a: Option>, + + /// The bounding rect of operand `b` when it is batch-constant. + b: Option>, +} - fn is_strict(&self, _: &Self::Options) -> bool { - true +impl ConstBboxes { + fn new(a: Option<&Geometry>, b: Option<&Geometry>) -> Self { + Self { + a: a.and_then(BoundingRect::bounding_rect), + b: b.and_then(BoundingRect::bounding_rect), + } } +} - fn is_fallible(&self, _: &Self::Options) -> bool { - false - } +/// Computes one row of intersects, spending any bounding rect hoisted into `bboxes`. +/// +/// Disjoint bounding rectangles conservatively prove that the geometries do not intersect. The +/// fall-through delegates to the unchanged `a.intersects(b)`, which refolds both rects internally, +/// so a batch where every row overlaps pays one extra `bounding_rect` fold over the row operand; +/// the win concentrates where most rows are disjoint, the usual spatial-filter shape. +fn intersects_row_prepared(bboxes: &ConstBboxes, a: &Geometry, b: &Geometry) -> bool { + let disjoint = match (bboxes.a, bboxes.b) { + (None, None) => false, + (Some(bbox_a), Some(bbox_b)) => !bbox_a.intersects(&bbox_b), + (Some(bbox_a), None) => b + .bounding_rect() + .is_some_and(|bbox_b| !bbox_a.intersects(&bbox_b)), + (None, Some(bbox_b)) => a + .bounding_rect() + .is_some_and(|bbox_a| !bbox_a.intersects(&bbox_b)), + }; + + if disjoint { + return false; + } + + a.intersects(b) } #[cfg(test)] @@ -133,7 +138,9 @@ mod tests { use geo_types::Coord; use geo_types::Geometry; use geo_types::LineString; + use geo_types::MultiPoint; use geo_types::MultiPolygon; + use geo_types::Point; use geo_types::Polygon; use rstest::rstest; use vortex_array::ArrayRef; @@ -157,8 +164,10 @@ mod tests { use wkb::writer::WriteOptions; use super::SpatialIntersects; + use crate::scalar_fn::row::probe::assert_prepared_agrees_with_columns; use crate::test_harness::nullable_point_column; use crate::test_harness::point_column; + use crate::test_harness::rect_column; /// A rectangle polygon with corners `(x0, y0)` and `(x1, y1)`, no holes. fn rect_polygon(x0: f64, y0: f64, x1: f64, y1: f64) -> Polygon { @@ -439,4 +448,85 @@ mod tests { assert!(result.is_err()); Ok(()) } + + // The prepared-vs-expanded agreement grid: every constant arrangement of a pairing must + // return exactly what the fully expanded columns return. + + /// A point geometry. + fn point(x: f64, y: f64) -> Geometry { + Geometry::Point(Point::new(x, y)) + } + + /// A linestring geometry through `coords`. + fn line(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::LineString(LineString::from(coords)) + } + + /// A multipoint geometry over `coords`. + fn multipoint(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::MultiPoint(MultiPoint::from(coords)) + } + + /// Constant arrangements agree with expanded columns across the pairing classes the prepared + /// kernel treats differently: bbox-prechecked pairs (polygon x polygon, linestring x + /// anything, multipolygon blankets), direct pairs (points), the excluded `MultiPoint` route, + /// and an empty geometry whose bounding rect does not exist. + #[rstest] + #[case::polygons_overlapping(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(2.0, 2.0, 6.0, 6.0).into())] + #[case::polygons_touching_edge(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(4.0, 0.0, 8.0, 4.0).into())] + #[case::polygons_touching_corner(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(4.0, 4.0, 8.0, 8.0).into())] + #[case::polygons_disjoint(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::polygons_nested(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::polygon_x_point_inside(donut(), point(2.0, 2.0))] + #[case::polygon_x_point_on_boundary(donut(), point(0.0, 5.0))] + #[case::polygon_x_point_in_hole(donut(), point(5.0, 5.0))] + #[case::point_outside_x_polygon(point(20.0, 20.0), donut())] + #[case::nan_point_x_polygon(point(f64::NAN, 2.0), donut())] + #[case::polygon_x_nan_point(donut(), point(f64::NAN, 2.0))] + #[case::points_equal(point(1.0, 1.0), point(1.0, 1.0))] + #[case::points_distinct(point(1.0, 1.0), point(2.0, 1.0))] + #[case::linestring_crossing_polygon(line(vec![(-2.0, -2.0), (2.0, 2.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::linestring_disjoint_polygon(line(vec![(-2.0, -2.0), (-6.0, -6.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::linestring_in_polygon_hole(line(vec![(4.5, 4.5), (5.5, 5.5)]), donut())] + #[case::linestrings_crossing(line(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(0.0, 4.0), (4.0, 0.0)]))] + #[case::linestrings_disjoint(line(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(10.0, 10.0), (14.0, 14.0)]))] + #[case::empty_linestring_x_polygon(line(vec![]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipoint_straddling_polygon(multipoint(vec![(2.0, 2.0), (20.0, 20.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipoint_outside_polygon(multipoint(vec![(20.0, 20.0), (30.0, 30.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipolygon_disjoint_polygon( + Geometry::MultiPolygon(MultiPolygon::new(vec![ + rect_polygon(0.0, 0.0, 2.0, 2.0), + rect_polygon(10.0, 10.0, 12.0, 12.0), + ])), + rect_polygon(20.0, 20.0, 24.0, 24.0).into() + )] + fn constant_operands_agree_with_columns( + #[case] a: Geometry, + #[case] b: Geometry, + ) -> VortexResult<()> { + assert_prepared_agrees_with_columns( + SpatialIntersects::try_new_array, + geometry_constant(&a, 3)?, + geometry_constant(&b, 3)?, + ) + } + + /// `Rect` has no WKB form, so its constant comes from a one-row rect column; its conservative + /// bbox early-out and exact fall-through must agree with the expanded form like the rest. + #[test] + fn rect_operand_agrees_with_columns() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + let rect_scalar = rect_column(vec![(0.0, 0.0, 4.0, 4.0)])?.execute_scalar(0, &mut ctx)?; + let rect_constant = ConstantArray::new(rect_scalar, 3).into_array(); + let polygon_constant = + geometry_constant(&Geometry::Polygon(rect_polygon(2.0, 2.0, 6.0, 6.0)), 3)?; + + assert_prepared_agrees_with_columns( + SpatialIntersects::try_new_array, + rect_constant, + polygon_constant, + ) + } } diff --git a/vortex-spatial/src/scalar_fn/mod.rs b/vortex-spatial/src/scalar_fn/mod.rs index e6770be4fff..bcdb15e51e6 100644 --- a/vortex-spatial/src/scalar_fn/mod.rs +++ b/vortex-spatial/src/scalar_fn/mod.rs @@ -8,3 +8,4 @@ pub mod distance; pub mod envelope; mod execute; pub mod intersects; +pub(crate) mod row; diff --git a/vortex-spatial/src/scalar_fn/row.rs b/vortex-spatial/src/scalar_fn/row.rs new file mode 100644 index 00000000000..ce7c34072ff --- /dev/null +++ b/vortex-spatial/src/scalar_fn/row.rs @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What the geo scalar functions add to the row-function machinery: an element type that decodes a +//! native geometry column into `geo_types` geometries. + +use geo_types::Geometry; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::dtype::DType; +use vortex_array::scalar_fn::InputElement; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +use crate::extension::geometries; +use crate::extension::geometries_null_tolerant; +use crate::extension::is_native_geometry; + +/// Marker for native geometry input elements: accepts any native geometry column and presents each +/// row as a decoded `geo_types` geometry. +/// +/// The two operands of a binary geo function need not share a geometry type, since distance, +/// containment and intersection across types are all meaningful, so this validates only that the +/// column is *some* native geometry. +pub struct GeometryRow; + +impl InputElement for GeometryRow { + type Column = Vec>; + type Varying<'a> = &'a [Geometry]; + type Elem<'a> = &'a Geometry; + + // A geometry row is decoded from its coordinate storage, which behind a null row holds arbitrary + // coordinates that need not describe a well-formed geometry. + const DENSE_SAFE: bool = false; + // Decoding builds a geometry from stored coordinates, and a malformed one in a *valid* row is a + // domain error rather than an infrastructural failure. + const DECODE_FALLIBLE: bool = true; + // Decoding arrow-exports the column and parses one geometry per row, so filtering the column + // first shrinks the decode itself, not just the row loop. + const FILTERED_DECODE_COST: usize = 1; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + is_native_geometry(dtype), + "spatial: operand {dtype} is not a native geometry type" + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + geometries(&array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> &Geometry { + &column[index] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column.as_slice() + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a Geometry + where + Self: 'a, + { + &column[index] + } + + /// Null rows decode to a placeholder geometry that the branch-and-skip row loop never reads. + /// `Point` and `Polygon` columns are covered; other geometry types return `Ok(None)` and the + /// batch falls back to the filter strategy. + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + geometries_null_tolerant(&array, ctx) + } +} + +/// Test-only support for the prepared geo row kernels: a probe recording which operands a +/// `prepare` step saw as batch-constant, and the shared prepared-vs-expanded agreement check +/// built on it. +#[cfg(test)] +pub(crate) mod probe { + use std::cell::Cell; + + use vortex_array::ArrayRef; + use vortex_array::Canonical; + use vortex_array::ExecutionCtx; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::MaskedArray; + use vortex_array::arrays::ScalarFnArray; + use vortex_array::assert_arrays_eq; + use vortex_array::validity::Validity; + use vortex_error::VortexResult; + + thread_local! { + /// Which operands the last `prepare` saw as constant, as a bitmask (bit 0 for `a`, bit 1 + /// for `b`). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + pub(crate) static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; + } + + /// Record which operands `prepare` saw as constant. + pub(crate) fn record(a_constant: bool, b_constant: bool) { + SEEN_CONSTANTS.set(u8::from(a_constant) | (u8::from(b_constant) << 1)); + } + + /// Execute `build(a, b)` and assert that `prepare` saw exactly `expect_seen` as its constant + /// operands, so the test knows which decode path the inputs took. + fn run_probed( + build: &impl Fn(ArrayRef, ArrayRef) -> VortexResult, + a: ArrayRef, + b: ArrayRef, + expect_seen: u8, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + SEEN_CONSTANTS.set(u8::MAX); + let result = build(a, b)? + .into_array() + .execute::(ctx)? + .into_array(); + + assert_eq!( + SEEN_CONSTANTS.get(), + expect_seen, + "prepare saw the wrong constant operands", + ); + Ok(result) + } + + /// Assert that every constant-operand arrangement of `build(a, b)` returns exactly what the + /// fully expanded columns return, and that each arrangement's constness really reached + /// `prepare` (so the constants exercised the stride-0 path rather than a decoded column). + /// + /// Arrangements: `a` constant, `b` constant, and both constant with `a` masked. A plain + /// constant pair folds to a single-row execution before the row loop, so masking one side is + /// what drives the both-hoisted arm across rows; that run is compared against the same mask + /// over the expanded column. + pub(crate) fn assert_prepared_agrees_with_columns( + build: impl Fn(ArrayRef, ArrayRef) -> VortexResult, + const_a: ArrayRef, + const_b: ArrayRef, + ) -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + let col_a = const_a.clone().execute::(&mut ctx)?.into_array(); + let col_b = const_b.clone().execute::(&mut ctx)?.into_array(); + + let baseline = run_probed(&build, col_a.clone(), col_b.clone(), 0b00, &mut ctx)?; + let a_hoisted = run_probed(&build, const_a.clone(), col_b.clone(), 0b01, &mut ctx)?; + let b_hoisted = run_probed(&build, col_a.clone(), const_b.clone(), 0b10, &mut ctx)?; + assert_arrays_eq!(a_hoisted, baseline, &mut ctx); + assert_arrays_eq!(b_hoisted, baseline, &mut ctx); + + let validity = Validity::from_iter((0..col_a.len()).map(|row| row != 1)); + let masked_const_a = MaskedArray::try_new(const_a, validity.clone())?.into_array(); + let masked_col_a = MaskedArray::try_new(col_a, validity)?.into_array(); + let both_hoisted = run_probed(&build, masked_const_a, const_b, 0b11, &mut ctx)?; + let masked_baseline = run_probed(&build, masked_col_a, col_b, 0b00, &mut ctx)?; + assert_arrays_eq!(both_hoisted, masked_baseline, &mut ctx); + + Ok(()) + } +} diff --git a/vortex-spatial/src/test_harness.rs b/vortex-spatial/src/test_harness.rs index 7b471bdf2c4..d8175d14f53 100644 --- a/vortex-spatial/src/test_harness.rs +++ b/vortex-spatial/src/test_harness.rs @@ -251,7 +251,7 @@ pub fn nullable_rect_column(boxes: Vec>) -> VortexR Ok(ExtensionArray::try_new(ext.erased(), storage)?.into_array()) } -/// Decode a [`Coordinate`] from an extension-typed point scalar (unwrapped to its coordinate +/// Decode a `Coordinate` from an extension-typed point scalar (unwrapped to its coordinate /// storage) or a bare coordinate `Struct` scalar — used to read back a single point in assertions. pub fn coordinate_from_scalar(scalar: &Scalar) -> VortexResult { match scalar.as_extension_opt() { From b541d4bdcb1ab684316766e2b961e39d694546bf Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:03:43 +0000 Subject: [PATCH 06/12] Record the row scalar function research and handoff notes Progress towards #9128. `STRICT_SCALAR_FN_RESEARCH.md` holds the null-strategy measurements behind the per-batch selection rule, the three verdicts and the crossover, and the record of why `vortex.byte_length`, `vortex.not`, and `vortex.list.sum` stay on `ScalarFnVTable`. `SCALAR_FN_HANDOFF.md` records the current state of the work, including which API proposals from the review were backed out and why. `NUMERIC_ROWFN_PLAN.md` is the plan and measured outcome for the numeric port. `docs/strictness-and-validity-pushdown.typ` writes up strictness and validity pushdown, which is the property the whole derivation rests on. These are working notes rather than published documentation, and they are separated here so they are easy to drop before this ships. Signed-off-by: Connor Tsui Co-authored-by: Claude --- NUMERIC_ROWFN_PLAN.md | 256 +++ SCALAR_FN_HANDOFF.md | 443 +++++ STRICT_SCALAR_FN_RESEARCH.md | 1790 +++++++++++++++++++++ docs/strictness-and-validity-pushdown.typ | 243 +++ 4 files changed, 2732 insertions(+) create mode 100644 NUMERIC_ROWFN_PLAN.md create mode 100644 SCALAR_FN_HANDOFF.md create mode 100644 STRICT_SCALAR_FN_RESEARCH.md create mode 100644 docs/strictness-and-validity-pushdown.typ diff --git a/NUMERIC_ROWFN_PLAN.md b/NUMERIC_ROWFN_PLAN.md new file mode 100644 index 00000000000..2f69b708b01 --- /dev/null +++ b/NUMERIC_ROWFN_PLAN.md @@ -0,0 +1,256 @@ + + + +# Plan: fit the numeric binary operators onto `RowFn` + +Working note, branch-only, like `SCALAR_FN_HANDOFF.md`. Written so this survives a conversation +compaction: everything needed to start is here, and nothing below depends on chat history. + +## Where things stand + +Branch `ct/row-fn`, at `4becc863ae` after the final API +simplification. Issues #9128, #9129, and #9130 match the implementation. The public-path benchmark +baseline from #9136 is now in the repository. + +This document preserves the original spike plan and the measurements that answered it. The current +API has no witnesses, persistence is function-owned, executor-only helper traits are sealed, and +filtered decode cost is additive per input. Read the outcome and final API/codegen sections before +following an earlier step literally. + +`byte_length` is no longer a row function, and `Bytes`/`BytesLen` are deleted. It measured 7.6-7.7x +slower than develop and is the case #9128 already excludes. + +## Goal of this spike + +Prove, or disprove, that the four arithmetic operators can move onto `RowFn` without changing the +`RowFn` API, without a second scalar function ID, and without touching serialization. Doing this +first is deliberate: it is the change most likely to force an API change, and discovering that after +tensor and geo are ported would mean reworking them. + +## The design + +`Binary` keeps everything and delegates only execution: + +```rust +Operator::Add => ScalarFnVTable::execute(&NumericBinary, &NumericOperator::Add, args, ctx), +``` + +`NumericBinary` is a `RowFn` with `Options = NumericOperator` and `FALLIBLE = true`. It is **not** +registered as a public scalar function, so it needs no ID in the registry and appears in no +serialized expression. It is reached through the `ScalarFnVTable::execute` that the blanket impl +already provides. + +Why this works, and each of these was verified against the code rather than assumed: + +- **Nothing is lost.** `BooleanKernel` and `CompareKernel` exist with per-encoding pushdown; there is + no `NumericKernel`. Unlike `not`, a numeric port gives up no encoding fast path. +- **The seam is already numeric-only.** All four arithmetic arms of `Binary::execute` funnel into + `execute_numeric(lhs, rhs, NumericOperator, ctx)`, and `NumericOperator` is already its own enum in + `crate::scalar`, so it is a ready-made `RowFn::Options`. +- **Fallibility is uniform.** `Binary::is_fallible` is false for the six comparisons plus `And`/`Or` + and true for exactly the four arithmetic operators, so `FALLIBLE = true` on a numeric-only `RowFn` + is exactly right. The options-independence of `RowFn::is_fallible` only bites when one function + spans both families. +- **Strictness stays where it belongs.** `Binary::is_strict` is `!matches!(op, And | Or)` because + Kleene `false AND null` is a valid `false`. `Binary` keeps owning that; `NumericBinary` never sees + a boolean operator. +- **Decimal fits.** `OutputSink::sink_dtype(args)` sees the input dtypes, which is what + `numeric_op_result_decimal_dtype(decimal_dtype, op)` needs. + +## Steps + +1. **Primitive path only, `Add` only.** A `NumericBinary` `RowFn` over `(T, T)` for one integer + width, with a deferred-error sink that writes the wrapping sum and ORs an overflow bit. Delegate + only `Operator::Add` from `Binary::execute` and leave the other three on `execute_numeric`. + Success is: the existing `binary/numeric/tests.rs` suite passes unchanged. +2. **Widen to every primitive ptype**, through `match_each_native_ptype!` in `dispatch`. Confirm the + compile-time witness check tolerates it, as it does for tensor widths. +3. **Add `Sub`, `Mul`, `Div`.** `Div` is the awkward one: see the risk below. +4. **Decide decimal.** Either a decimal input element plus a sink that carries the result precision + and scale, or leave `DType::Decimal` on `execute_numeric` and delegate only the primitive path. + Leaving it is a legitimate outcome for the spike and possibly for the first PR. +5. **Delete the replaced code** only once benchmarks agree, not before. + +## Risks, in the order they are likely to bite + +- **`Div` already has a per-type strategy.** `primitive.rs` carries `CHECKED_VALUE_LOOP` and + `DIV_CHECKS_IN_VALUE_LOOP`, set per type, so division checking is not uniform. A single row closure + may not express it, and `Div` may have to stay behind. +- **The existing implementation is tuned, not naive.** `checked.rs` has `checked_lanes` and + `checked_apply_lanes` taking a `valid_rows: &Mask` and returning `Result, usize>` with the + failing index. The port is replacing real engineering, so parity is not a given. This is the reason + the CodSpeed gate on the `binary_ops` names from #9136 matters. +- **Two declarations of the result dtype must agree.** `Binary::return_dtype` is what the expression + layer uses, while `reconcile_return` checks the kernel output against `NumericBinary`'s + sink-derived dtype. Cover every operator and dtype pair with a test that asserts they match. +- **Error messages are part of the contract.** `primitive.rs` defines `ERROR` per operator, such as + `"integer overflow in checked add"`, and `numeric/tests.rs` asserts on failures. The deferred-error + sink reports once from `finish`, so the message must be preserved and the error must still be + raised for the same inputs. +- **Overflow behind a null row must stay invisible.** `numeric/tests.rs` has + `test_decimal_overflow_on_null_lane_ignored`. The lifting's deferred-error retry over valid rows is + exactly this behavior, so the test should pass, but it is the first thing to check. + +## Verification + +```bash +cargo nextest run -p vortex-array +cargo clippy --all-targets --all-features -p vortex-array +cargo +nightly fmt --all +cargo test --doc -p vortex-array +``` + +The numeric suite specifically: + +```bash +cargo nextest run -p vortex-array scalar_fn::fns::binary +``` + +Performance gate is CodSpeed on the stable `binary_ops` names from #9136. Locally, use +`cargo bench -p vortex-array --bench binary_ops` with two runs, fastest and median, machine stated. + +## What this spike is not + +Not a PR. Not a deletion of `execute_numeric`. Not decimal support unless step 4 turns out easy. The +output is an answer to "does this fit cleanly", plus whatever the answer implies for #9129's API. + +## Outcome + +It fits, with no change to the `RowFn` API and one change to the machinery. + +Steps 1 through 3 landed together rather than in sequence: once the sink existed, widening it through +`match_each_native_ptype!` and adding the other three operators was the same code. Step 4 leaves +decimal on `execute_numeric_decimal`, which the delegation makes easy since `execute_numeric` still +owns the dtype split. Step 5 deleted the replaced primitive execution, which the measurements below +justify. + +### What the design turned out to be + +`Binary::execute` is untouched. `execute_numeric` keeps its validation, its error messages, its empty +short circuit, and its primitive/decimal split, and only `execute_numeric_primitive` changed: it +builds a `VecExecutionArgs` and calls `ScalarFnVTable::execute(&NumericBinary, &op, ..)`. Everything +the old implementation did around the arithmetic (decoding, the constant-operand collapse, the +all-constant fold, the null-constant short circuit, output allocation, nullability widening, masking, +and the valid-row retry after an overflow behind a null) is now the lifting's. + +`NumericOperator` became the options type. `NumericBinary` is unregistered and deliberately has no +serialization implementation. Persistence now belongs to each `RowFn`, so reusing an options type +does not silently assign the helper a wire contract. `Binary` retains its existing ID and options +serialization, and only primitive execution delegates to `NumericBinary`. + +Three things the old code carried are gone because the row framework removes the distinction they +existed for: + +- `CHECKED_VALUE_LOOP` and `DIV_CHECKS_IN_VALUE_LOOP` chose between a split value/error scan and a + one-pass early-exit kernel, because for integer division the split loop only added a second scan. + A row kernel produces the value and the error bit in the same pass, so there is one loop shape and + no choice to make. `div_i64` got 1.11x faster. +- `checked_apply_lanes` had no caller left. `checked_lanes` stays for decimal. +- `PrimitiveOperand` moved to `compare/primitive.rs`, its only remaining user. + +### The machinery change: the reduction is a word the kernel chooses + +`SinkResult` gained `Accumulated`, the word the executor OR-reduces in a loop-local. Two properties +of that reduction are load-bearing, and each was got wrong once before the numbers made it obvious. + +- **Width no greater than the element.** `DeferredError` held an `i64`, which bounds how many rows a + vector of the reduction covers whatever the element width. That cost `Mul` 3.5x at `i8`, 2.05x at + `i16` and 1.28x at `i32`, and nothing at `i64` where the widths already agree. +- **It lives in a loop-local, not in the sink.** Holding the accumulator as a sink field, reached + through a `&mut` for every row, is a loop-carried memory dependence. It cost the boolean kernels + 2.5x to 10x while leaving the three unsigned multiply kernels untouched. + +Naming the word is also what lets multiplication report the discarded high half of its product +rather than a comparison, which is what recovers its vectorization. `OutputSink` is unchanged and no +sink names the word. + +### Results + +Against the hand-written kernels, divan medians, best of two runs, 65536 rows, Apple M4 Max, with +the decimal, boolean and comparison benchmarks held as controls and moving under 2%: + +| benchmark | hand-written | row framework | | +| --- | --- | --- | --- | +| `mul_u8_nonnull` | 22.91 us | 1.854 us | 12.4x faster | +| `mul_u16_nonnull` | 22.20 us | 3.791 us | 5.9x faster | +| `mul_u32_nonnull` | 24.62 us | 7.124 us | 3.5x faster | +| `div_i64_nonnull` | 40.41 us | 34.83 us | 1.16x faster | +| `mul_i64_nonnull` | 27.37 us | 28.66 us | 1.05x slower | +| `mul_i32_constant` | 7.583 us | 8.041 us | 1.06x slower | + +Everything else lands within 3%, which is inside this host's drift between sessions. The unsigned +multiply win is not attributable to the port: the same defect exists in the hand-written kernels and +is fixed for `develop` separately in vortex-data/vortex#9210, stacked on vortex-data/vortex#9211. +Re-measure the port against `develop` once that lands, because the comparison above flatters it. + +### Measured dead ends + +Recorded so they are not retried. All of these are in vortex-data/vortex#9130 as well. + +- Bounds-check elimination in the row loop is not available. Narrowing the varying view to the row + count buys nothing, and `get_unchecked` is not uniformly a win: about 10% on `mul_u16` and + `mul_u32`, and 22% slower on `mul_u8`. +- A per-argument row source that keeps the `Varying` view when another argument is batch-constant is + 4x slower than the `ArgColumn` branch it replaces, which already vectorizes. +- A batch-constant operand therefore still demotes its neighbours off the slice path. Closing that + needs the row loop monomorphized over which arguments are constant. Revisit when `Compare` moves + onto `RowFn`, since `col < literal` is exactly this shape. + +`mul_i32_constant` is the one regression that survives, and it is inside this host's drift. Let +CodSpeed settle whether it is real. + +### What this implies for #9129 and #9130 + +- The `RowFn` API needed nothing. No new visit method, no options-aware `sink_dtype`, no return + witness. `NumericBinary::FALLIBLE = true` is conservative for every dispatch arm, and each + concrete result type supplies the precise loop behavior. +- `SinkResult::Accumulated` and its two constraints belong in #9130, and are recorded there. +- On kernels this close to the vectorizer's decision boundary, the emitted IR is the reliable gate + and wall clock on one host is not. Two separate interventions here moved a benchmark the wrong + way, and host drift between sessions exceeded the effects under measurement. + +### Final API cleanup and generated code + +The later simplification did not add numeric-specific surface: + +- `NumericBinary` declares `ARG_NAMES = &["lhs", "rhs"]` instead of repeating an argument witness. +- Its `Options = NumericOperator` has no persistence bound or implementation. The registered + `Binary` function remains the sole owner of the serialized `vortex.binary` contract. +- The selected input tuple carries arity, dense-safety, decode fallibility, and filtered-decode + cost. The selected sink and `SinkResult` carry output and deferred-error facts. +- `SinkResult` is sealed, but a numeric function does not need to implement it. It chooses the + supplied unsigned evidence width that matches the primitive element width. +- `OutputSink` remains one abstraction. A later numeric function with multiple logical outputs + should put both builders in one sink rather than add a pair-of-sinks framework type. + +The final cleanup was checked against its parent by cross-compiling the optimized +`row_fn_executor` benchmark for `x86_64-apple-darwin` with `target-cpu=x86-64-v3`. After normalizing +symbol names and metadata, the vector/reduction block for checked `i64` add matched exactly. It +retains `<4 x i64>` loads and adds, vector overflow detection through xor/and/compare operations, +`<4 x i1>` OR accumulation, and a reduction after the loop. The vector body has no call or panic +path, and the scalar tail is unchanged. + +The ordinary `ElementSink` and custom-sink wrapping-add monomorphs also matched exactly. Native +Apple M4 Max measurements over 65,536 rows found RowFn median changes between 1.11% faster and 0.94% +slower, with fastest changes within about 0.17%. Specialized controls drifted more than the RowFn +arms, so there is no measurable native regression from the cleanup. + +This is not an x86 runtime result. It proves that the API edits preserved the optimized x86_64-v3 +loop shape. Runtime confirmation for numeric changes should use the stable public benchmark names +from #9136 on the target host. + +The next session will run on x86 and must perform that confirmation. The #9136 `binary_ops` +benchmark is on `develop` at `9a482c0230`, so compare this branch with the latest +`origin/develop` using the same public benchmark names. Record both exact commits and run each +revision at least twice in alternating order. If possible, pin one core. Report fastest and median +values with the CPU and timer configuration. If a stable case regresses, compare its optimized LLVM +IR before changing the row API or restoring hand-written execution. + +### Verification + +The whole of `binary/numeric/tests.rs` passed unchanged, including +`test_decimal_overflow_on_null_lane_ignored` and the integer-error tests that pin the valid-row +retry. Decimal is untouched and stays on `execute_numeric_decimal`. The final API state also +recorded 67 focused RowFn tests, 179 tensor tests, 230 geo tests, nightly formatting, and full +workspace clippy. Clippy needed `PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1` because the host +Python is 3.9 while the workspace targets the Python 3.11 stable ABI. diff --git a/SCALAR_FN_HANDOFF.md b/SCALAR_FN_HANDOFF.md new file mode 100644 index 00000000000..f99f4cb0eaa --- /dev/null +++ b/SCALAR_FN_HANDOFF.md @@ -0,0 +1,443 @@ + + + +# Handoff: the row scalar-function framework + +This is the concise source of truth for the branch. `STRICT_SCALAR_FN_RESEARCH.md` keeps the full +design history, rejected alternatives, measurements, and generated-code evidence. +`NUMERIC_ROWFN_PLAN.md` records the numeric-binary migration and its narrower performance boundary. +All three are branch-only working notes for agents. They are not intended to land with the API. + +The public design lives in these tracking issues, which now match the implementation: + +- [#9128, Row-oriented scalar functions](https://github.com/vortex-data/vortex/issues/9128) +- [#9129, Define the `RowFn` API](https://github.com/vortex-data/vortex/issues/9129) +- [#9130, Execute `RowFn` over Vortex arrays](https://github.com/vortex-data/vortex/issues/9130) + +The branch is `ct/row-fn`. It is publicly linked from #9128, so do +not rewrite or delete its history. Commit `4becc863ae` contains the final API simplification. Push +only when explicitly requested. + +## Next action: rerun the benchmarks on x86 + +The next session will run on an x86 machine. Rerun the performance comparison there before treating +the implementation as complete. Do not reuse the Apple timings as the final runtime result. + +The production benchmark baseline from #9136 is on `develop` at `9a482c0230`. Fetch the latest +`origin/develop`, record the exact baseline and candidate commits, and run the same public benchmark +binaries at both revisions: + +```bash +cargo bench -p vortex-array --bench binary_ops +cargo bench -p vortex-array --bench like +cargo bench -p vortex-tensor --bench l2_norm +cargo bench -p vortex-tensor --bench inner_product +cargo bench -p vortex-tensor --bench cosine_similarity +cargo bench -p vortex-tensor --bench normalized +cargo bench -p vortex-geo --bench binary_predicates +cargo bench -p vortex-geo --bench distance +cargo bench -p vortex-geo --bench envelope +cargo bench -p vortex-geo --bench predicate_bbox +``` + +Run each revision at least twice in alternating order. If the host allows it, pin the process to one +core. Record the timer and CPU configuration, and compare both fastest and median values. The +benchmark binaries and public names are now shared with `develop`, so the comparison no longer +needs a frozen benchmark-local implementation as its primary control. + +Also run the branch-only `vortex-geo` `null_strategies` diagnostic. It forces branch-and-skip and +filter-and-scatter for the measured nullable geometry shapes. Confirm that automatic selection uses +the faster mechanism for one costly decode at 50% survivors and for two costly decodes at about 81% +survivors. This is the x86 runtime check that remains after the LLVM comparison. + +```bash +cargo bench -p vortex-geo --bench null_strategies +``` + +If a stable benchmark regresses, inspect optimized LLVM IR again. The previous cross-compile proves +that the API cleanup preserved the x86_64-v3 loop shape. The x86 run must confirm runtime effects +from the revised null selector and the target CPU's vectorizer and branch predictor. + +## The API in one screen + +`RowFn` is the author-facing function trait. A function gives the framework its exact argument +names, a conservative fallibility declaration, function-owned persistence, and a value-blind +dispatch over concrete input and sink types: + +```rust +impl RowFn for Example { + type Options = ExampleOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.example"); + *ID + } + + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + Ok(Some(encode(options)?)) + } + + fn deserialize( + &self, + metadata: &[u8], + session: &VortexSession, + ) -> VortexResult { + decode(metadata, session) + } + + fn dispatch( + &self, + options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + validate_options(options, args)?; + visitor.visit_prepared_into::<(InputA, InputB), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| { + *output = compute(lhs, rhs); + }, + ) + } +} +``` + +There are no argument or return witness types. The dispatched tuple is the argument declaration, +the sink owns the output representation, and the row result names the error behavior. Planning +runs the same dispatch as execution and checks the selected types against the function constants. + +## The extension boundary + +The framework is deliberately not sealed wholesale. Function authors need to add decode and output +primitives for their own scalar functions. Only the executor mechanics are closed. + +| API | Boundary | Why | +| --- | --- | --- | +| `RowFn` | open | Defines a scalar function and selects concrete execution types. | +| `InputElement` | open | Adds a new scalar decode primitive, including crate-local domain types. | +| `OutputElement` | open | Adds an ordinary one-value-per-row output primitive. | +| `OutputSink` | open | Adds a custom output representation or builder. | +| `RowVisitor` | sealed | Executor-owned dispatch mechanism with one supported implementation. | +| `ElementTuple` | sealed | Executor-owned tuple recursion, with built-ins through arity 12. | +| `SinkResult` | sealed | Executor-owned loop and error facts trusted by the blanket vtable. | + +`ElementTuple` being sealed does not prevent a function from adding a decode primitive. Implement +`InputElement` and use it inside one of the supplied tuples. Likewise, a function with two logical +outputs should define one `OutputSink` whose state has two fields. The framework does not need a +second tuple or composite-sink abstraction. + +The supplied `SinkResult` forms are: + +- `()` for infallible rows; +- `VortexResult<()>` for an error that must stop immediately; and +- `bool`, `u8`, `u16`, `u32`, or `u64` for error evidence OR-reduced after the loop. + +The unsigned evidence widths let each kernel choose a word no wider than its element type. That is +load-bearing for vectorization, particularly for checked unsigned multiplication. + +## Function-owned persistence + +Persistence belongs to the function ID, not to the Rust options type. `RowFn::Options` has no +serialization supertrait. The `RowFn::serialize` and `RowFn::deserialize` hooks have conservative +defaults, and registered functions override them when their existing wire contract requires it. + +This has three useful consequences: + +- two functions may reuse an options type while choosing different formats; +- a function may deliberately be nonserializable even if another function serializes the same + options type; and +- an unregistered helper such as `NumericBinary` needs no dummy persistence implementation. + +Tensor and geo functions keep their explicit existing formats. Do not introduce a blanket options +wire format or infer serializability from `Options`. + +## One sink abstraction + +`OutputSink` is the complete output contract. It owns the output dtype, allocation, row storage, +row lookup, length proof, and final array construction. `ElementSink` covers the common case. Its +row type is `&mut T`, so the closure writes with ordinary assignment. + +Custom sinks remain available for a real output shape that cannot use `ElementSink`. The unused +public `TensorSink` was removed. No current tensor row function returns tensor-valued rows, and a +90-line public runtime-shaped sink was not justified without a user. Add a custom sink when a real +function needs one, using one sink struct even when it owns several builders. + +Every current sink produces an all-valid child column. The blanket vtable can therefore derive the +function result validity from the input validities. Nullable row outputs remain out of scope. A +sink that emits its own nulls must change that derivation in the same change. + +`OutputSink::sink_dtype` must return a non-nullable dtype. `SUPPORTS_SKIPPED_ROWS` says whether +branch-and-skip may leave placeholder rows behind the result validity. `ERRORS_ARE_DEFERRED` says +whether the sink accepts accumulated error evidence at `finish`. + +## Dispatch and fallibility + +`dispatch` must be pure in `(options, args)`. It sees dtypes, not array values. Planning and +execution both call it, so value-dependent preparation belongs inside `visit_prepared_into`. + +The executor statically checks each dispatched visit: + +- the tuple arity equals `ARG_NAMES.len()`; +- a fallible decoder, early-error result, or deferred result implies `RowFn::FALLIBLE`; +- deferred evidence requires both `RowFn::FALLIBLE` and a sink with + `ERRORS_ARE_DEFERRED = true`; and +- the sink and result agree about their error contract. + +The implications are intentionally one-way. `FALLIBLE = true` is a conservative function-level +claim, while a particular dtype dispatch arm may be infallible. + +`prepare` must not be load-bearing for validation. Empty batches may bypass value preparation, and +the executor needs its safety and fallibility facts before it runs the closure. + +## Null execution policy + +The old public `NullHandling` enum and argument witness were removed. Authors do not select an +execution mechanism. The executor derives a private row policy from the dispatched input and result +types: + +- `Dense` may execute over garbage behind nulls and masks afterward; +- `DenseWithRetry` may execute densely, then retry valid rows when deferred evidence reports an + error; and +- `ValidOnly { filtered_decode_cost }` guarantees that the row closure sees only valid rows. + +An early-failing row or a decoder that is not dense-safe must use valid-only execution. A deferred +kernel may use dense execution because it writes a legal provisional value for every row. If only +garbage behind nulls reports an error, the valid-row retry discards it. + +Valid-only execution has two mechanisms. Filter-and-scatter shrinks inputs before decoding. +Branch-and-skip decodes the original batch and visits set bits from the conjoined validity mask. A +sink that does not support skipped rows automatically falls back to filter-and-scatter. + +The selector needs more than a boolean "decode shrinks" flag. Every `InputElement` declares an +additive `FILTERED_DECODE_COST`, defaulting to zero. `ElementTuple` sums the costs across arguments: + +- cost 0 always prefers branch-and-skip; +- cost 1 prefers branch-and-skip at 50% or more surviving rows; and +- cost 2 or greater prefers branch-and-skip at 85% or more surviving rows. + +This distinction comes from the x86 measurement in #9128. One nullable geometry input at 50% nulls +favored branching, while two independently nullable geometry inputs at 10% nulls each, about 81% +survivors, favored filtering. OR-ing a per-argument flag loses exactly that distinction. + +The values are still a coarse heuristic. There is no evidence yet to separate cost 2 from cost 3, +and the batch-size crossover has not been measured. `NullStrategy` remains only as a test-harness +seam for forcing a mechanism. Do not expose the private row policy as an author contract. + +## Performance and generated-code evidence + +The older Ryzen 9 7950X AVX-512 measurements remain the production-performance record in the +[#9128 follow-up](https://github.com/vortex-data/vortex/issues/9128#issuecomment-5151831802). They +also supplied the per-argument null-selection evidence above. + +The final API cleanup was checked separately against its parent, `53c51d803c`, by cross-compiling +the optimized `row_fn_executor` benchmark for `x86_64-apple-darwin` with `target-cpu=x86-64-v3`. +After normalizing symbol names and metadata, the vector and reduction blocks were identical for all +three executor shapes: + +- ordinary wrapping add through `ElementSink`; +- checked add with deferred evidence; and +- wrapping add through a custom sink. + +The wrapping loops retain 256-bit `<4 x i64>` loads, adds, and stores. The checked loop retains the +same vector loads and adds, derives overflow with vector xor/and/compare operations, accumulates +`<4 x i1>` with vector OR, and reduces after the loop. None of the vector bodies contains a call or +panic path. Scalar tails are unchanged. + +The production tensor benchmarks were also cross-compiled before and after the cleanup. Normalized +arithmetic sequences and counts match for `l2_norm`, inner product, and cosine similarity. Their +ordered floating-point reductions are scalar-unrolled in both revisions because LLVM preserves the +strict reduction order. The cleanup did not remove vectorization because those reductions were not +vectorized before it. + +Native Apple M4 Max timings used 65,536 rows, two alternating before/after runs, 100 samples, and a +0.5-second minimum per arm. RowFn median deltas ranged from 1.11% faster to 0.94% slower. Fastest +deltas stayed within about 0.17%, while specialized controls drifted by as much as 3.7% in their +medians. There is no measurable native regression from the API cleanup. + +This does not replace the required x86 runtime run above. Cross-target IR proves that the hot loop +shape survived, not that the revised null selector has the expected branch-predictor behavior on +x86. + +## Current implementation and checks + +The implementation includes production users in `vortex-array`, `vortex-tensor`, and `vortex-geo`. +`NumericBinary` is an unregistered `RowFn` used only for primitive arithmetic execution. Decimal +arithmetic keeps its existing path. The stable public-path benchmark baseline landed as #9136. + +The checks recorded for the final API state are: + +- 67 focused RowFn tests; +- 179 `vortex-tensor` tests; +- 230 `vortex-geo` tests; +- `cargo +nightly fmt --all`; and +- full workspace clippy, with `PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1` because the host + `/usr/bin/python3` is 3.9 while the workspace requires the Python 3.11 stable ABI. + +The generated-code comparison and native timing evidence are described above and in the final +section of `STRICT_SCALAR_FN_RESEARCH.md`. + +## Review pass: what changed and what was deliberately left + +A review of the three parts (API, execution, implementations). **The author-facing API is +unchanged**: every proposal that would have altered it was backed out, for the reasons below, and +what landed is cleanup, corrected documentation, and test coverage. The emitted IR of every +`visit_prepared_into` monomorph is identical to the pre-review commit. + +API: + +- `InputElement::decode_null_tolerant` overrides that only restated the default were deleted from + the primitive, bool and `TensorRow` elements. `GeometryRow`'s override is the only real one. The + doc now says a dense-safe element should *not* override. +- `ElementTuple` now records why it carries arities past the widest function in tree: it is sealed, + so a downstream crate cannot add the one it needs, and an uninstantiated arity costs only its own + macro expansion. + +Execution: + +- `execute_filtered` and the forced-strategy test seam now share `resolve_validity`, so the mask + materialization and the all-true/all-false shortcuts cannot drift apart between them. +- The dense-retry path's comment was wrong and is corrected. It filters unconditionally because + `execute_dense` is not handed the `branch` closure, **not** because a deferred sink cannot skip + rows: `ERRORS_ARE_DEFERRED` and `SUPPORTS_SKIPPED_ROWS` are independent consts and a sink may + legally set both. + +Implementations: + +- `l2_norm_row` had two copies, in `l2_norm.rs` and `cosine_similarity.rs`. Cosine's prepared and + per-row arms must agree bit for bit, which only holds while both accumulate in the same order, so + the duplicate was an invitation to break exactly the property the comments defend. One copy now + lives in `utils.rs` beside the other shared tensor helpers. +- `CosineSimilarity::reduce_encoded` zips its three slices instead of indexing `0..len` three times + per row, and documents why it materializes where `InnerProduct::reduce_encoded` stays lazy (the + zero-norm guard is a conditional, not an arithmetic factor). +- `IndexedSourceExt::map_checked_into` was deleted from vortex-compute. `CheckedSink` replaced the + split value/evidence pass it served, and it had no caller left. +- `contains_route` and the workspace `geo` dependency both record that the table transcribes geo's + `impl_contains_from_relate!` and must be re-verified on a version bump. `geo` is pinned to + `=0.31.0`: a caret requirement would admit 0.31.x patches, which `cargo update` (or automated + lockfile maintenance) takes with no diff to review, and a patch is free to reshuffle the dispatch + without any API change. The agreement tests stay green wherever relate and the direct algorithm + agree, so the pin, not the suite, is what makes the coupling break only deliberately. + +Split out onto `develop` instead of landing here: + +- **The checked-arithmetic macro collapse.** `primitive.rs` on this branch and on `develop` both + carry four near-identical `CheckedArithmetic` bodies that differ only in `mul_failure`, so the + collapse into one `impl_checked_integer!` belongs on `develop` where every caller benefits. It is + on `claude/collapse-checked-arith-macros`. This branch's `primitive.rs` keeps its four bodies + until `develop` is merged, at which point the collapse arrives with it and the merge conflict is + a member deletion rather than two competing macro structures. +- **The `mul_failure` kernel tests.** The exhaustive 8-bit sweep and the 64-bit probe grid already + exist on `develop` from vortex-data/vortex#9210 and arrive with the same merge. + +Deliberately **not** done: + +- **No `DeferredElementSink`.** `CheckedSink` exists largely because `ElementSink` cannot name an + error at `finish`. A framework sink combining an element output with a type-level message would + remove ~100 lines per function, but there is exactly one deferred-error function. Build it when a + second appears, rather than copying `CheckedSink`. +- **No change to `reduce_encoded`'s probe semantics.** Hoisting the probe out of the strategy paths + and masking a full-length result looks like a simplification and is not one: + `normalized_readthrough_survives_null_rows` pins that a filtered input is no longer `Normalized`, + so which arrays reach `reduce_encoded` is load-bearing and differs per strategy. +- **No PR split.** Recommended landing order, each step individually revertible and separately + benchmarkable: (1) API + lifting with dense/filter only; (2) branch-and-skip + adaptive selection + + its benchmarks; (3) `NumericBinary`; (4) tensor; (5) geo. The seam already supports this split + and no API changes between steps. + +### Three API changes proposed, and why none of them landed + +All three were implemented, run against the suite, and backed out. None prevents a bug, and this +branch's open work is *settling* the API rather than churning it, so they belong in #9129 as +questions decided alongside the rest of the surface: + +- **Should `reduce_encoded` take an explicit `row_count`?** The filtered-count requirement is real + and easy to miss, but `args` are filtered to match, so `args[0].len()` is already both the natural + thing to write and correct. The parameter is documentation, and it costs every implementor a + signature change. What survived is the test: + `reduce_encoded_is_probed_before_and_after_filtering` pins that the rewrite is offered the + original arrays at full length and then the filtered ones at the surviving count. +- **Should `OutputSink::row_count_matches` become `rows_len`?** A length reads cleaner and lets the + executor name what it found. Against that, `row_count_matches` lets a sink fold in its own + invariants, which `SpreadSink` uses for its width check; narrowing it turns that into a panic. + Neither spelling prevents a bug. +- **Should the nullary path go?** A function with no inputs has no validity to lift, which is the + lifting's whole job. But `RowFn` would still give it sink allocation and dtype derivation, so + `random()` or `now()` is not obviously better hand-written, and the path is ~70 lines and tested. + +Trimming `ElementTuple` to arity four was proposed on the same reasoning and backed out for a +stronger one: the trait is sealed, so the arities are the only ones a downstream crate can ever +have. + +### Two changes this pass made and then reverted + +Both were proposed, implemented, reviewed, and backed out on evidence. They are recorded because +each is an attractive idea that a later reader will have again. + +**Making `CheckedSink` safe with `BufferMut::zeroed` costs 1.65 to 1.71x.** Replacing the +`MaybeUninit` storage removes an `unsafe set_len` and reads as a clear win, and `ElementSink`'s own +comment appears to bless it by routing a zeroable placeholder to `alloc_zeroed`. Measured, it is +not: allocate-zeroed-then-fill against allocate-then-fill, interleaved in one process over `u64` +outputs, ran **1.221x** slower at 8 KiB, **1.71x** at 64 KiB, **1.66x** at 512 KiB and **1.71x** at +2 MiB, stable to within 2% across two runs. `alloc_zeroed` does not avoid the write: below glibc's +mmap threshold `calloc` recycles a dirty chunk and memsets it, and above it every fresh page faults +on first touch. The row loop overwrites every slot regardless, so this is a duplicated pass over +the output of the hottest kernel in the system. + +Note the corollary, which is a real optimization nobody has taken: `ElementSink::with_capacity` +pays exactly this on every batch, and only branch-and-skip ever reads a placeholder back. A sink +that allocated uninitialized on the dense and filter paths would recover it. + +**Hoisting `OutputSink::SUPPORTS_SKIPPED_ROWS` into the plan is not sound as an optimization.** +#9130 records "avoid probing `reduce_encoded` twice when branch execution is unsupported" as a +follow-up. It reads as free, and is not, because the branch path probes `reduce_encoded` against +the _original_ arrays before it consults the sink, and that is the only probe that ever sees them +still encoded. Skipping the path early leaves such a function with only the filtered probe, whose +canonical arrays match no encoding fast path. For a function whose reduction is _defined_ to answer +differently from its row loop, which is exactly what `L2Norm` over `Normalized` is, that is a wrong +answer rather than a slow one. Nothing in tree is reachable today only because every `ValidOnly` +dispatch happens to use `ElementSink`. **#9130's follow-up should be struck, not implemented.** +`reduce_encoded_is_probed_before_and_after_filtering` now pins the two probes and their row +counts. + +### On measurement, and what the IR gate does and does not cover + +Wall-clock benchmarking of the row loops was attempted first and abandoned on evidence. Two runs of +the *same* baseline binary, pinned with `taskset -c 2`, 100 samples, disagreed by up to 4x +(`row_wrapping_add_nullable`: 198.8 us then 52.9 us median; `specialized_checked_add`: 185.5 us then +34.4 us). The 4-vCPU shared VM drifts more within a session than any effect being measured, which is +the same conclusion this branch already reached on a dedicated 7950X. + +The gate used instead is the emitted optimized IR of every `visit_prepared_into` monomorph in +`vortex-array`, profiled by vector width, reduction count, overflow-intrinsic survival and bounds +checks, then compared as a multiset before and after. Reproduce with: + +```bash +RUSTFLAGS="--emit=llvm-ir -C codegen-units=1" cargo rustc -p vortex-array --release --lib +``` + +**Its blind spot is worth stating, because it nearly landed a regression.** The IR of a row loop +cannot show an allocator call outside it, so the `BufferMut::zeroed` substitution above passed this +gate cleanly while costing 1.7x. An allocation-strategy change needs its own targeted A/B, which is +cheap to write and immune to the host drift above because both arms run interleaved in one process. +Use the IR gate for loop shape and a focused microbenchmark for anything the loop does not contain. + +## Remaining boundaries + +- Complete the required x86 production and forced-null-strategy benchmark run above before treating + the thresholds or overall performance as settled. +- Keep nullable outputs separate until the first real function can define the validity contract. +- Do not add another sink composition abstraction. Put multiple builders in one custom sink. +- Do not add a general runtime-shaped sink until a production function needs one. +- Keep pattern compilation and other state shared across rows outside `RowFn` when it cannot be + represented as batch preparation. +- Use emitted optimized IR as a gate for numeric changes near LLVM's vectorization boundary, then + use the stable #9136 benchmark names for runtime confirmation. + +## Repository rules for the next agent + +Follow `AGENTS.md`. Keep public APIs small, run narrow checks before workspace-wide checks, and +report blocked checks separately from passing ones. Preserve unrelated working-tree and staging +state. Every commit must include the required `Signed-off-by` trailer. diff --git a/STRICT_SCALAR_FN_RESEARCH.md b/STRICT_SCALAR_FN_RESEARCH.md new file mode 100644 index 00000000000..1c873108345 --- /dev/null +++ b/STRICT_SCALAR_FN_RESEARCH.md @@ -0,0 +1,1790 @@ + + + +# A layered authoring API for strict scalar functions + +**Status: historical design record, with the final API review recorded at the end.** This document +keeps the experiments in the order they happened, including APIs and ports that were later removed. +The current architecture is one `RowFn` authoring trait, private lifting, one sink-backed +`RowVisitor::visit_prepared_into` primitive, and a deliberately open input/output vocabulary. Read +[`SCALAR_FN_HANDOFF.md`](SCALAR_FN_HANDOFF.md) for orientation, then the final section here before +using an earlier sketch. + +> **Later architecture decisions:** `StrictScalarFnVTable`, the columnar ports, returning visits, +> both witness types, `PersistableOptions`, the public `NullHandling`, the aggregate decode-shrinks +> flag, and the unused `TensorSink` were deleted. Framework-only visitor, tuple, and result traits +> are sealed. `InputElement`, `OutputElement`, and `OutputSink` remain open so functions can add +> their own decode and output primitives. Sections below remain the evidence that led to those +> decisions, not the API to implement. + +--- + +## Current benchmark and codegen record + +The authoritative current comparison is the +[x86 AVX-512 re-measurement on issue #9128](https://github.com/vortex-data/vortex/issues/9128#issuecomment-5151831802). +It records the machine, exact refs, stabilized governor, two-run fastest and median results, control +limitations, geo fix, adaptive-null diagnostics, and native LLVM IR/assembly in folded sections. +It supersedes every older shared-VM or pre-#9076 figure in these notes for claims about the current +branch versus `develop`. + +The run used candidate `d293d3cdd59e` plus the recorded geo bbox widening, baseline +`876996fe7846`, and a Ryzen 9 7950X pinned to CPU 4 with the TSC timer and performance governor. +The conclusions that survive into the implementation plan are: + +- sink-only checked add is 1.018-1.226x faster by median than its benchmark-local specialized + control, depending on constant and null shape; +- cosine is 1.40-30.13x faster than current develop; +- prepared overlapping `contains` is 8.60-8.77x faster by median, while the widened bbox gate + restores disjoint polygons to parity with #9076; +- point/constant geo still has real 8.6-14.2% and 10.9-13.2% median regressions; +- `BytesLen` is 1.410-1.411x faster by median on long strings and 1.097x on short strings; +- the global 75% survivor threshold mispredicts both one-input/50%-null and + two-input/10%-null geo cases, so adaptive selection needs element/arity-aware cost data; +- checked-add codegen has AVX-512 error-word accumulation and post-loop vector reduction with no + per-row error branch; current `l2_norm` remains a strict-order scalar reduction. + +The stabilized cosine median ratios preserve the shape and width dependence instead of collapsing +the result into one headline range: + +| shape | width 2 | width 32 | width 256 | +| --- | ---: | ---: | ---: | +| column x column | 5.77-5.79x | 1.93-2.19x | 2.54-2.58x | +| column x constant | 12.44-12.48x | 28.87-28.96x | 30.08-30.13x | +| column x extension constant | 3.05x | 1.40-1.41x | 1.77-1.78x | + +The final geo median ratios, including the bbox patch, are: + +| predicate and shape | develop / row branch | +| --- | ---: | +| contains, column x column points | 0.951-0.964x | +| contains, column x column polygons | 0.999-1.003x | +| contains, constant x points | 0.876-0.921x | +| contains, disjoint polygons | 0.993-0.997x | +| contains, overlapping polygons | 8.60-8.77x | +| intersects, column x column polygons | 0.983-0.995x | +| intersects, points x constant | 0.884-0.902x | +| intersects, disjoint polygons | 1.011-1.019x | +| intersects, overlapping polygons | 1.011-1.023x | + +Here, as above, ratios greater than 1x favor the row branch. The issue comment contains the paired +fastest and median observations rather than only these compact ranges. + +The historical measurements below remain because they explain design decisions and experiments made +while building the prototype; they are not the current before/after performance record. + +--- + +## The design in one screen + +```text +RowFn ──────────blanket──▶ StrictScalarFnVTable ──────blanket──▶ ScalarFnVTable +(row at a time, types (null / constant / validity (full control) + chosen per batch) lifting for a columnar kernel) +``` + +Two authoring traits, one for each axis a strict function actually varies on, plus a third axis (*how +a row is typed, and how its output is delivered*) factored into an open element and sink vocabulary that +neither trait mentions. + +### `StrictScalarFnVTable`, the null/validity lifting + +Write the structural metadata plus one **columnar** kernel that ignores validity. A blanket impl +derives: + +- `is_strict = true`, and a mirrored `validity` a kernel can answer with the conjunction of its child + validities when it never turns a wholly non-null row into a null (see + [Strictness is not totality](#strictness-is-not-totality)), so the planner knows which rows are null + without executing the function. +- `return_dtype` = `return_element_dtype` widened to nullable iff any input is nullable, so the + strictness dtype contract holds by construction rather than per function. +- `execute` = the shared cases before the kernel runs: a null-constant input short-circuits to an + all-null constant, all-constant inputs evaluate one row and broadcast, and partially-null inputs + are handled per `NullHandling` (`Dense` masks after a full pass, `Filter` filters then scatters). +- Options serde, from `PersistableOptions` on the options type. + +This is the layer for a function whose kernel is columnar rather than row-at-a-time: `not` (one `!` +per 64-bit word), `list_length` (a difference of offset buffers), `list_sum` (a grouped accumulator over +the elements child). See [Why three concepts and not fewer](#why-three-concepts-and-not-fewer) for why it +cannot be folded away. + +### `RowFn`, one row with element types chosen per batch + +Name a witness argument tuple and return type, then in `dispatch` pick the concrete element types for +a batch and hand the framework a row closure through a rank-2 visitor. A blanket impl derives the +whole `StrictScalarFnVTable` from it. When the element types are fixed, `dispatch` is a single +`visit` at those types. When one ID spans several widths (`l2_norm` accepts f16/f32/f64), `dispatch` +matches on the input dtypes and visits at the chosen width. + +Everything structural follows from the argument tuple and return type: arity, per-argument dtype +validation, the output dtype, null handling, and fallibility. There is nothing for an implementor to +declare twice or get wrong, because the framework reads it off the types (see +[Properties, not conventions](#properties-not-conventions)). A constant operand is decoded once and +read at stride 0, so a broadcast argument costs one decode rather than one per row. + +Output takes one of two forms, chosen per visit. `visit` takes a closure that **returns** an +`OutputElement`, one owned value per row whose dtype is a property of its Rust type. `visit_into` takes +one that **writes** into an `OutputSink`, allocated once per batch knowing the output dtype and handing +out a place to write. Orthogonally, `visit_prepared` runs a once-per-batch prepare step over the +element values of whichever operands are constant for the batch, and threads its result to every row +by shared reference; plain `visit` is that with unit state (see +[Constant compute](#constant-compute-the-last-quadrant-of-the-lifting)). The sink carries what an owned per-row value cannot: `l2_denorm` writes each row +into a slice of one flat buffer, so its output width comes from the arguments and it allocates once +rather than per row. The executor holds the sink and passes the handle in, so the closure stays `Fn` +and the returning path pays nothing. + +Note that `RowFn` does not *require* totality, it just cannot currently express its absence: both output +forms build an all-valid column, so a row kernel has no way to say "this row is null". An +`impl OutputElement for Option`, or a sink that can push a null, would lift that, at the cost of +revisiting the `validity` law that reads the output validity off the inputs. No function needs it yet, so +it is not there. + +### The element vocabulary, how a row is typed + +`InputElement`, `OutputElement` and `OutputSink` are open traits. A `NativePType`, `bool`, `Bytes` (a +resolved `&[u8]`), and `BytesLen` (a length read from a view without resolving it) ship in the framework, +and `vortex-tensor` adds `TensorRow`, reaching through the extension wrapper into flat storage, plus +`TensorSink` on the output side, in its own crate. Adding `&str`, decimals, or a list row is one impl +that every row function gains, with no framework change. + +--- + +## Why three concepts and not fewer + +The standard applied here: every trait, and every member of every trait, has to have a purpose +nothing else can provide. Testing each against that standard is what the bulk of this research was. + +### `RowFn` and the witnesses are forced, not chosen + +A scalar function's *signature*, meaning its arity and fallibility, is a property of +`(function, options)` with **no input dtypes**: `ScalarFnVTable::arity(&self, options)` and +`is_fallible(&self, options)`, and `ScalarFnSignature` above them, take none. So any framework that +derives arity and fallibility from element types has to be able to name element types *without seeing +dtypes*, which is exactly what `ArgsWitness` / `RetWitness` are. Because `dispatch` *does* see dtypes +and could choose otherwise, some check has to tie the two together, which is the compile-time witness +check below. This cost is not a consequence of the rank-2 encoding: **any** design that derives a +dtype-free signature from per-batch types pays it. + +A previous iteration made the width choice a generic-associated-type family generated by a +`row_family!` macro. Rust cannot abstract over a GAT's bound (`type Args` is +rejected), so that approach needed a trait *and* an adapter per width class, hand-written or +macro-stamped. The rank-2 visitor sidesteps the limit rather than writing around it: the kernel owns +the width `match`, where `T: Float` appears literally inside a `match_each_*_ptype!` arm, and the +framework method `RowVisitor::visit` is generic only over bounds it +owns. The macro, its family traits, and its generated adapters are all deleted. Note that `dispatch` +is not even per-*width*: it can pick different element *kinds* per dtype, which no +bound-parameterized family could. + +### `ElementwiseFn` was not forced, so it is gone + +An earlier revision had a third trait, `ElementwiseFn`, for the fixed-element-type case: name `Args` +and `Ret`, write `apply`. It read cleanly, but it failed the standard. `RowFn` already covers the +fixed case (the dispatch is a single constant `visit`), so `ElementwiseFn` bought roughly seven lines +on exactly one production function (`byte_length`) at the cost of 114 framework lines and a third +link in the blanket-impl chain. The probes settled it: of the functions examined, `not` and `list_sum` +turned out not to be row functions at all, and `list_length` needed the encoding-aware +`reduce_encoded` hook that `ElementwiseFn` never exposed. So the constituency I expected it to have +never materialized, and it is deleted. `byte_length` writes a two-line `dispatch` instead. + +The one-trait-with-defaults alternative (a single `RowFn` with `dispatch` defaulted to visit the +witnesses and `apply` defaulted to `unimplemented!()`) was rejected because it converts a compile +error into a runtime panic: a type implementing neither method compiles, registers, and answers +signature queries with a plausible shape, then panics on first execution. `dispatch` is therefore +required. + +### `StrictScalarFnVTable` cannot be folded into `RowFn` + +`RowFn`'s type surface is *closed*. The output dtype is `OutputElement::element_dtype()`, drawn from +the finite set of `OutputElement` impls, `ElementTuple` exists only for arities 1 to 3, and the loop +is one `apply` per row. Three whole classes of strict function are therefore inexpressible as a +`RowFn` at any cost: + +- **Output dtype outside the element set.** `ext_storage`'s output is an extension array's storage + dtype, so `vortex.geo.box` is a struct and `vortex.uuid` is a `FixedSizeList(u8,16)`. `vortex-geo`'s + zone-map pruning calls `ext_storage` on a `geo.box` statistic, and a row-function port breaks it at + plan time. +- **Variadic arity.** `merge` and `select` take an unbounded number of children, while `RowFn` fixes + `Arity::Exact(n <= 3)`. +- **Sub-row-granular kernels.** `not` negates one 64-bit word at a time, so a row loop over `bool` is + ~64x the memory traffic and, measured, 406x slower at a 64Ki batch (see + [Measurements](#measurements)). + +So the middle layer has a genuine, disjoint constituency: `not`, `list_length`, `list_sum`, and +prospectively `select`, `merge`, `json_to_variant`. "Just a visitor" collapses three concepts to two +rather than to one. + +### Every remaining member earns its place + +A member-by-member audit, with call sites found by grep rather than by guess, turned up nothing +deletable. The non-obvious cases are worth recording: + +- **`RowVisitor::Out`** is what lets one `dispatch` `match` serve both plan time (`Out = DType`, + validate and name the output dtype) and run time (`Out = ArrayRef`, decode and run the loop). The + alternatives, a `{DType, ArrayRef}` enum unwrapped at each site or two separate dispatch hooks, + either add unwrap-panics or duplicate the width `match` in every width-polymorphic function with no + compiler check that the two copies agree. +- **A plan-time visit is unavoidable.** `l2_norm` declares `RetWitness = f64` but dispatches over + f16/f32/f64, so the output dtype read off the witness would be wrong for two of three widths. Also + `TensorRow::validate` rejects an `f32` column against an `f64` witness, and the visit is what + gives cross-argument uniformity for free (`int_max(i16_col, i64_col)` is rejected by + `(T, T)::validate`, not by any `dispatch` body, which only inspects `args[0]`). +- **`ApplyResult` distinct from `OutputElement`** is what lets one trait serve both infallible + (`Ret = f64`) and fallible (`Ret = VortexResult`) kernels without a wrapper. `f64` cannot be + simultaneously fallible and infallible, so the fallibility bit lives on the return *shape* rather + than on the element. + +--- + +## Properties, not conventions + +The framework's real value beyond line count is that two invariants an implementor used to have to +get right are now derived from the types, so an unsound combination cannot be written. + +### Null handling follows from the arguments and the return type + +`NullHandling::Dense` runs the kernel over every row including those behind nulls, then masks. It is +cheaper than filtering and the only option that leaves inputs at their original encoding, so it is +right whenever it is sound. Soundness needs two things, every argument readable behind a null row and +an infallible computation, and both are already visible in the types: + +```rust +const fn row_null_handling() -> NullHandling { + if A::DENSE_SAFE && !row_is_fallible::() { NullHandling::Dense } else { NullHandling::Filter } +} +``` + +Whether a dense read is safe is a property of the *element*, not of the function: reading a whole +value out of a flat buffer is safe (`NativePType`, `bool`, `TensorRow`, `BytesLen`), while following a +stored offset into a data buffer is not (`Bytes`), because arrays only validate the views of their +*valid* rows. This caught a real bug in this branch's own `byte_length`, see +[Problems to extract](#problems-to-extract-onto-develop). + +### Fallibility comes from the return type *and* the element decode + +A function is fallible if its computation can fail (`Ret = VortexResult`) **or** if decoding an +argument can fail on legal data. The second source is real and was missing: `geo_distance`'s row +computation cannot fail, but parsing WKB bytes into a geometry can, for a *valid* row holding +malformed bytes. So `InputElement` carries `DECODE_FALLIBLE`, and fallibility is the disjunction: + +```rust +const fn row_is_fallible() -> bool { A::DECODE_FALLIBLE || R::FALLIBLE } +``` + +`is_fallible` gates dict-value pushdown (`arrays/dict/compute/rules.rs`), which speculatively +evaluates a function over *unreferenced* dictionary values, so a function that under-reports +fallibility fails a query on rows it never needed. + +### The witness is checked at compile time + +Arity, dense-safety and fallibility must not vary between the choices `dispatch` makes, because the +framework acts on them before dispatching. Since (with `ElementwiseFn` gone) *every* function names +its element tuple twice, once as `ArgsWitness` and once in the `visit`, the check that the two agree +is load-bearing, and it is a compile-time `const` assert inside each visit: + +```rust +const fn assert_witness_agrees() { + assert!(A::ARITY == ::ARITY, "…"); + assert!(A::DENSE_SAFE == ::DENSE_SAFE, "…"); + assert!(row_is_fallible::() == row_is_fallible::(), "…"); +} +``` + +Monomorphizing any dispatch arm evaluates it, so even a `match` arm that never runs at a given width +is checked, and a disagreement fails the build pointing at the exact `visit::<…>` call. It compares +the raw arity/dense-safety/fallibility rather than the derived `NullHandling`, which collapses +dense-safety and fallibility together and would miss an arm that flipped both. A `compile_fail` +doctest pins that a lying witness does not compile. This replaced a runtime check that ran three +times per array (plan, execute, deserialize). + +--- + +## Strictness is not totality + +This is the finding that decides what the middle layer may derive. Note that +[#9033](https://github.com/vortex-data/vortex/pull/9033) reached the same conclusion independently and +has since landed, so this section is no longer the argument for the finding, only for the API that +follows from it. + +Before #9033, the `is_strict` documentation stated the validity-equivariance law, +`f(…, mask(aⱼ, m), …) == mask(f(…, aⱼ, …), m)`, and then asserted as "consequence 1" that output +validity is the conjunction of input validities. **Consequence 1 does not follow from the law.** It +needs an extra premise: that the kernel never turns a wholly non-null row into a null. #9033 replaced +that equality with a one-sided bound, `valid(f(a₁, …, aₖ)) ⊆ valid(a₁) ∧ … ∧ valid(aₖ)`, which is the +vocabulary this branch uses. `docs/strictness-and-validity-pushdown.typ` proves the law and the +null-propagation reading are the same property, and separates what does not follow from either. + +`list_sum` is the counterexample. Summing a valid *empty* list yields null. It still satisfies the law +(a null it introduces at a valid row appears identically on both sides of the equation and cancels), +so it is genuinely strict, but its output validity is *narrower* than its input validity. + +Two properties, then, not one: + +| property | what needs it | +| --- | --- | +| **strict** (null propagation, equivalently validity equivariance) | every validity push-down, the thing we actually want | +| **total** (non-null in implies non-null out) | upgrading the `⊆` bound to `=`, so validity is precomputable | + +The old blanket impl derived `validity = union_child_validities` for *every* implementor, which needs +totality while the trait only requires strictness. Every current implementor happens to be total, so +nothing was broken, but a partial function joining the layer would get a `validity` that contradicts +what it computes: `arr.validity()` would report all-valid while `arr.execute()` yields the null, since +`ValidityVTable::validity` evaluates the derived expression. `list_sum` was about to be +exactly that, and is now ported onto the layer as the first non-total member. + +#9033 says a function satisfying the stronger equality "can advertise that through +`ScalarFnVTable::validity`". That is the same idea as `is_total`, moved from a hand-written method to a +boolean, because a blanket impl cannot hand-write `validity` per function: it needs the property as +data in order to decide whether to derive one. + +The fix needs no new property. `validity` is mirrored on `StrictScalarFnVTable` alongside `reduce`, +defaulting to `None`, and a kernel that satisfies the equality answers it with +`union_child_validities`. The unsound direction is the one that now takes work, and the safe default is +what a function gets for free. + +An earlier revision of this branch instead added an `is_total` method and derived `validity` from it. +That was strictly worse: it introduced a concept the codebase did not have, in order to compute +something a function can just say directly. It is gone. The `RowFn` blanket impl answers `validity` +for every row function, justified by its own output vocabulary (no `OutputElement` is nullable, so no +row kernel can introduce a null), which keeps the row layer at zero boilerplate. + +Note that strictness rather than totality gates membership either way: `is_null` is total but +disqualified, because it inspects validity and so does not propagate nulls. That is also why the trait +is not called `TotalFnVTable`. + +> **A related latent issue, deliberately not fixed here.** Four functions declare `is_strict = true` +> and are strict-but-not-total: `get_item` (a nullable field under a non-null struct), `mask`, +> `variant_get`, `geo_envelope`. None is broken today, since `get_item` leaves `validity` at the +> default and `mask` overrides it correctly, but any that grows a conjunction-shaped `validity` +> derivation would be wrong. This predates the branch and belongs in its own investigation. + +--- + +## Problems to extract onto develop + +The framework surfaced three problems that are not really about the framework. Each is filed +separately and I think each should land as its own PR rather than riding in on this one. Note that +none of them is a live miscompute on `develop` today, which is worth saying plainly, because the +branch's own commit messages describe fixes to *this branch's* code. + +1. **Strict-but-non-total validity derivation ([#9091]).** The `is_strict` documentation presents + totality as a consequence of strictness when it is an independent premise (see above). Nothing + derives validity from `is_strict` automatically, so nothing is wrong today, but the doc invites the + next strict-but-partial function to write `validity: union_child_validities` and be silently wrong. + **Superseded by [#9033], which lands the documentation correction on `develop`.** This branch needs + nothing beyond that, since it now mirrors `validity` rather than deriving it from a property. + +2. **Views behind null rows are unvalidated ([#9090]).** `VarBinViewArray::validate_views` only + validates the views of *valid* rows, so a legal array can hold a view behind a null row naming a + buffer that does not exist, and resolving it densely panics (`index out of bounds: the len is 1 but + the index is 9`). On this branch, expressing byte length as "a function of the row's bytes" quietly + changed *what gets decoded* and hit that panic. The fix here reads the length out of the view + (`BytesLen`) and never resolves the row, and + `test_byte_length_ignores_unresolvable_views_behind_nulls` pins it (verified to panic without the + fix). `develop`'s `byte_length` was already immune, since it also read `view.len()`, so the + extraction is that regression test rather than a code change. The doc half is also covered by + [#9033], which deletes the dense-evaluation "consequence 2" outright rather than narrowing it. That + leaves `InputElement::DENSE_SAFE` as the only place the licence is written down, per element rather + than as a blanket claim, which is where it belongs. + +3. **Bit-at-a-time bool packing ([#9092]).** `OutputElement for bool` used `BitBuffer::from_iter`, + where the `Vec` is already owned and contiguous so `BitBuffer::from` routes to the + multiversioned SIMD packer. Measured **6.6 to 7.9x faster** on the packing step, for every + bool-returning row function. Note that `OutputElement` only exists on this branch, so the + develop-side instance of the same pattern is a different call site: + `encodings/sequence/src/compute/compare.rs` builds an n-bit result with a per-row predicate when it + already knows the single set index. I have not benchmarked that site. + +[#9033]: https://github.com/vortex-data/vortex/pull/9033 +[#9090]: https://github.com/vortex-data/vortex/issues/9090 +[#9091]: https://github.com/vortex-data/vortex/issues/9091 +[#9092]: https://github.com/vortex-data/vortex/issues/9092 + +--- + +## Audit: can the four `StrictScalarFnVTable` impls really not be `RowFn`? + +There were exactly four in production when this audit ran. Auditing each against the two questions that +matter, rather than repeating the earlier verdicts, **not one of them was structurally impossible**. Every +"cannot" in this document was really "cannot with the trait signed as it is today". One of the four, +`l2_denorm`, has since moved onto `RowFn`, so three remain. Recording the distinction because it is the +difference between a limit and a decision. + +| function | signature expressible? | kernel row-shaped? | what it would take | +| --- | --- | --- | --- | +| `not` | **yes**, `(bool,) -> bool`, both elements exist | **no** | nothing. It can be a `RowFn` today and should not be: `!bits` is one `!` per 64-bit word, in place when unshared, against 16k closure calls and a `Vec` repack | +| `list_length` | output is a fixed `U64`; input needs a `ListLen` element | **no** | one new element. Still should not: the answer is a child array or one constant | +| `list_sum` | output is one number per row, so nearly: only the *nullability* is unexpressible | **no** | `impl OutputElement for Option` and a list element, but the kernel is the real blocker | +| `l2_denorm` | **yes, now**: an `OutputSink` names its dtype from the arguments | yes, per-row scaling | **done**, see below | + +**A varying output dtype was already supported, and listing it as a blocker was wrong.** `dispatch` +chooses element types per batch and `return_element_dtype` routes through it, so `R::Out::element_dtype()` +is already answered per dispatch arm. `l2_norm` relies on this today, visiting `::<(TensorRow,), T>` +with `T` ranging over the float widths. The compile-time witness check pins only arity, dense-safety and +fallibility, deliberately leaving the output type free to vary. What `l2_denorm` needed was different and +narrower: its output dtype depends on the input *dtype* in a way no choice of element type can express, +because the extension dtype carries a shape. That is what `OutputSink::sink_dtype(args)` supplies. + +**`list_sum`'s output side is the easy part; its kernel is not.** One number per row means it needs only +a nullable output element, no write-into-buffer machinery. But `execute_strict` is not a per-row sum: it +builds a `GroupedAccumulator` over `Sum`, calls `accumulate_list`, and then `mask_empty_lists` computes +per-group emptiness with `count_range` popcounts, with all-true and all-none fast paths and an early +return when nothing needs masking. Porting it to a row loop would hand-roll the shared aggregate +framework, lose the overflow modes that `NumericalAggregateOpts` selects, and trade SIMD popcounts for +per-row checks. That puts it in the same category as `not`: expressible, and worse. + +So `l2_denorm` was the only one of the four whose kernel actually wants to be a row loop, which is why it +was the right first target despite needing the larger output-side change. + +Two readings follow. + +**The honest framing is "can, and here is whether it is worth it."** For `not` and `list_length` the +answer is a flat no on performance grounds, and those are settled. For `list_sum` the answer is +yes-with-changes, and the change it wants is a nullable output, which the sink could supply but which the +`validity` law argues against (see below). + +**`l2_denorm` was the one worth doing, and it is done.** Its kernel genuinely is per-row scaling, and +it carried the `unsafe` the other three tensor ports removed. What it needed was a second visit method +whose closure *writes* its row instead of returning it, generalized to an `OutputSink` rather than +hardcoding `&mut [T]`, because the same mechanism covers three gaps recorded separately in these notes: + +- **runtime-shaped output**: the sink is a preallocated flat buffer and the per-row handle a + `&mut [T]` slice of it, so `l2_denorm` allocates once per batch rather than once per row. This is what + shipped. +- **`str -> str` without the double copy**: the sink is one growing byte buffer plus views, and + `upper`/`lower`/`replace` push into it. Strictly better than the `Cow` output element considered + above, which still copies each row once. Not built, but the trait admits it unchanged. +- **nullable output**: a sink *could* push a null, which would remove the need for + `impl OutputElement for Option` as a separate patch. Deliberately **not** taken: both output forms + build an all-valid column today, and that is exactly what lets the blanket `validity` return + `union_child_validities`. Adding nulls to either form has to come with that law being revisited. + +### What shipped + +```rust +pub trait OutputSink: 'static + Sized { + type Row<'a> where Self: 'a; + fn sink_dtype(args: &[DType]) -> VortexResult; + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult; + fn row(&mut self, index: usize) -> Self::Row<'_>; + fn finish(self) -> VortexResult; +} + +fn visit_into( + self, + apply: impl Fn(A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult; +``` + +**The executor threads the sink, not the closure**, so `apply` stays `Fn` and the existing `visit` pays +nothing. That was the design constraint, not an accident: relaxing `visit` itself to `FnMut` measured at +8 to 11% (see the `like` discussion), and a handle passed in per row avoids captured mutable state +entirely. Measured after the fact, `l2_norm` is unchanged at 69.05 µs against the 69.44 µs recorded +before the sink landed. + +**Step 1 of the earlier plan turned out to be unnecessary.** The plan called for widening +`OutputElement::element_dtype()` to take `args`. It never happened, because `sink_dtype(args)` puts the +argument-dependence on the *sink* instead, leaving all three existing `OutputElement` impls untouched. +That is the better split: an element's dtype genuinely is a property of its Rust type, and only the +thing that needs the arguments asks for them. + +**The `RetWitness` split resolved as predicted.** It carried two roles, *what dtype* and *is it +fallible*, and only the second is readable before `dispatch` picks a form. So `RowResult` now holds just +`const FALLIBLE`, with `ApplyResult: RowResult` adding the output element and `SinkResult: RowResult` +adding nothing but the error, and `RowFn::RetWitness` is bounded by `RowResult`. A returning dispatch +names `f64` or `VortexResult`; a writing one names `()` or `VortexResult<()>`. Coherence permits +this: `impl RowResult for ()` does not overlap `impl RowResult for T` because +`(): OutputElement` does not hold and no downstream crate can make it hold, the same negative reasoning +the pre-existing `ApplyResult` impls already relied on. + +**A new limit, worth naming.** `sink_dtype` sees the input dtypes but **not** the function's options, +because `OutputSink` does not know the `RowFn`'s `Options` type. A function whose output dtype depends +on an option value therefore still drops to `StrictScalarFnVTable`, whose `return_element_dtype` sees +both. Nothing in the repository needs it, and threading options through later is additive. + +### Results + +`unsafe` in `l2_denorm.rs` went from 8 blocks to 6. The two removed are the memory-safety ones on the +kernel path: `FixedSizeListArray::new_unchecked` in the constant-norms path, now `try_new` (the norm is +cast to the element dtype first, so the product stays non-nullable and the check passes), and +`PrimitiveArray::new_unchecked` in `build_tensor_array`, now `new`. That second one is an independent +cleanup rather than something the port forced. + +The 6 remaining are not of that kind and are not the row layer's business: four are calls to +`L2Denorm::new_array_unchecked`, an `unsafe fn` whose contract is the *semantic* unit-norm invariant and +not memory safety, and two are buffer pushes inside `normalize_as_l2_denorm`, a helper that builds the +normalized child and is not a scalar function at all. + +**Performance: the sink is faster than the kernel it replaced**, which was not the expected outcome. +`vortex-tensor/benches/l2_denorm.rs`, `fastest` column, both configurations run twice, 16384 rows, +non-nullable. The control implements `StrictScalarFnVTable` with the pre-port body, so it shares the +strict lifting and the gap is the row layer alone: + +| width | sink | pre-port kernel | ratio | +| --- | --- | --- | --- | +| 2 | 88.02 / 88.16 µs | 60.19 / 60.45 µs | sink 1.46x slower *(since fixed, see below)* | +| 32 | 482.0 / 515.5 µs | 1.175 / 1.014 ms | sink **2.1x faster** | +| 256 | 10.23 / 10.43 ms | 20.41 / 22.48 ms | sink **2.0x faster** | + +The likely cause of the win is that the pre-port kernel collected a `flat_map` over rows into a fresh +`Buffer`, and `flat_map` is not `TrustedLen`, so that `collect` grew the buffer with a capacity check +per element. The sink allocates once with `BufferMut::zeroed` and each row writes a slice of it, which +vectorizes. The zeroing is not a separate pass at these sizes, since large allocations come back zeroed +from the allocator. This is a hypothesis consistent with the width scaling rather than something +profiled. + +Width 2 showed the same regression as `l2_norm`'s, and for the same reason: both read tensor rows through +`TensorRow`, whose `get` re-derived a typed slice per row. Typing the column at decode time took +`l2_denorm` from 88.0 µs to **48.9 µs** at width 2, ahead of this control rather than behind it. See +[the like-for-like comparison](#the-like-for-like-comparison-and-the-per-row-cost-that-was-hiding-in-it) +for the measurement and for the wrong diagnosis it corrects. + +The constant-norms fast path moved to `reduce_encoded`, which sees the argument arrays before the row +loop. It keeps both of its cases (unit norms return the normalized child untouched, any other constant +rewrites the storage elements through one multiply), and it still fires for a filtered batch because +filtering a constant yields a constant. + +**Two visit methods do not cover everything, and it is worth being precise about the residue.** They +cover every function whose output is *computed* per row, returned or written. What stays columnar is +output that *aliases* its input, since `trim` and `substring` want to keep the input's data buffer and +rewrite only views, copying nothing, and a sink still copies bytes into itself. Likewise kernels whose +natural unit is not a row (`not`'s word-at-a-time negation, `binary`'s slice kernels) gain nothing. + +The sink is also what a `str -> str` string library needs. After reclassifying `L2Denorm` as an +encoding, that string library becomes the prospective first production user rather than a second +one. The experiment still demonstrates that the generic sink can carry runtime-shaped and +builder-backed outputs without making the returning path pay, but it should not be stabilized from +the tensor experiment alone. + +--- + +## Constant compute: the last quadrant of the lifting + +The lifting's constant handling was complete on the data side and absent on the compute side. A +null-constant input short-circuits, all-constant inputs fold to one row, and a constant operand is +decoded once and read at stride 0. What nothing owned was kernel computation that depends only on a +constant argument: `cosine_similarity(rows, query)` with a broadcast query re-accumulated +`norm(query)`, an O(width) pass plus a sqrt, once per row, and the geo predicates rebuilt the +constant side's topology graph, R-tree, or bounding box once per row. `cosine_similarity` escaped +partially by hand-writing a `reduce_encoded` rewrite, and the survey found that rewrite already +wrong for the literal shape, which is the argument for framework ownership stated as a correctness +fact: one hand-written constant path per function is one place per function to rot on +encoding-normalization details. + +### Where the hook can live, and where it cannot + +The hoist needs three things at once: knowing which arguments are constant, having their decoded +values, and a typed place for the function to compute from them. Constness is a per-batch value +fact (a RunEnd slice landing inside one run, a per-chunk compression decision), so: + +- **`dispatch` cannot see it.** It runs at plan time and run time and must choose identical element + types at both; values do not exist at plan time. +- **Element types cannot encode it.** A `Const` wrapper element would need value-aware dispatch + to be chosen, splitting plan/run monomorphizations in exactly the way the witness deliberately + does not pin, and costing 2^arity dispatch arms. The salvageable half of the idea, + framework-internal value-driven specialization, already exists as the stride-0 `ArgColumn`. +- **The closure cannot memoize it.** An `unsync::OnceCell` capture compiles under `Fn`, but without + constness information it is wrong (it would cache row 0 of a varying operand), and with that + information it saves nothing over a prepare step while planting an unhoistable load inside the + loop. + +That leaves one point: inside the visit, after decode, where `ArgColumn` already knows each +column's stride. `ElementTuple` gains `ConstElems<'a>`, the element tuple with every slot wrapped +in `Option` (`Some` iff that operand is batch-constant), and the visitor gains: + +```rust +fn visit_prepared( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> R, +) -> VortexResult; +``` + +`prepare` runs once per batch; its result reaches every row by `&P`, so `apply` stays `Fn` and the +loop keeps the shape the FnMut measurement forbids changing. `P` names no column lifetime, so +prepared state provably cannot alias the columns the loop reads. Plain `visit` is now a *provided* +method, `visit_prepared` with unit state: the ZST erases under monomorphization (measured, l2_norm +non_nullable at 33.38 us against the 32.83 us hand-written control, parity), the duplicate row loop +is deleted, and the visitor's method count grows with genuine axes (how output is delivered) rather +than with feature combinations. + +`prepare` is infallible in v1: it refines values the row loop could compute itself, and fallibility +is read off the witnesses before dispatch, so a failing prepare would have nowhere to be declared. +The extension (prepare returning `VortexResult

`, riding the existing fallibility axis) is +documented next to the method and deliberately unbuilt, because no adopter needs it. + +Three boundary facts worth stating because they will bite someone: + +- **Prepare must never be load-bearing for validation.** An empty batch decodes every operand as + non-constant (there is no row 0 to slice), so a prepare that validated its constant would + silently not run. Validation belongs to `validate` and the dtype rules. +- **What counts as a batch constant is wider than the constant encoding.** The stride-0 decode sees + one level through two wrappers that spell "the same value in every row" without being it: + `MaskedArray(ConstantArray)`, how the compressor spells an all-same-with-nulls chunk (sound + because the lifting owns validity entirely, so the value the loop reads behind a null row is + unobservable), and `Extension` over constant storage, the shape extension builders produce before + `ExtensionConstantRule` normalizes it. +- **`P` having no `Send`/`Sync` bound is load-bearing.** geo's `PreparedGeometry` carries + `Rc`/`RefCell` and could not be prepared state otherwise. The flip side, recorded so it is a + decision rather than a surprise: adding such bounds later (a parallel row loop, say) is a + breaking change to real adopters, not a relaxation. + +### What it bought, measured + +**cosine_similarity, and a lesson in ILP.** The closure accumulated the rhs norm per element and +sqrt'd it per row, a third of the arithmetic plus one of two sqrts. Hoisting it moved the benchmark +by only ~5% at width 32 and ~3% at 256 (16384 rows, fastest column), far under the flop count, +because the loop is latency-bound on the serial dot-product FMA chain (FP reassociation is illegal) +and the removed accumulation was executing in the chain's spare ILP slots. The measurable saving is +the hoisted sqrt. The row is bit-identical either way, each arm accumulating in the same order as +the unprepared kernel. + +The lesson generalizes and is the honest scoping of the feature: **"removes an O(width) pass per +row" is not "saves time" when that pass rides in ILP slack.** The work that collects the full +saving is work that extends the dependency chain: parses, tree builds, prepared structures. Which +is exactly what the geo numbers then showed. + +**The geo predicates, where the win lives.** `contains` substitutes an owned +`PreparedGeometry<'static>` of the constant operand (r-tree plus self-noded topology, built lazily +inside `P` through a `OnceCell` so point-row batches never pay for it) into relate exactly where +geo routes `Contains` through relate, argument order preserved including the `MultiPolygon` +reversal; direct pairings keep geo's own algorithms untouched. `intersects` hoists the constant +side's `bounding_rect` and replays geo's own disjoint-bboxes early-out, gated to fire only where +geo makes exactly that comparison first. `distance` was investigated and left alone: geo builds +R-trees for both sides inside a private helper on every call, so there is no seam to reuse one, and +the finding is recorded as a doc comment on its dispatch. 16384 rows, fastest column, two runs: + +| arm | before | after | change | +| --- | --- | --- | --- | +| contains, constant x polygons, overlapping | 457.5 / 458.0 ms | 50.88 / 50.00 ms | **9.1x** | +| contains, constant x polygons, disjoint | 7.04 / 7.05 ms | 3.97 / 3.74 ms | **1.9x** | +| contains, constant x points (direct route) | 3.15 / 3.08 ms | 3.22 / 3.15 ms | unchanged | +| contains, column x column | 3.56 / 6.29 ms | 3.68 / 6.40 ms | unchanged | +| intersects, polygons disjoint x constant | 6.81 / 6.72 ms | 3.20 / 3.14 ms | **2.1x** | +| intersects, polygons overlapping x constant | 9.57 / 9.48 ms | 9.87 / 9.63 ms | 1-3% slower, accepted | +| intersects, points and column x column arms | 3.20 / 5.98 ms | 3.21 / 5.92 ms | unchanged | + +The overlapping-intersects arm is the disclosed tradeoff: the hoisted bbox check is an early-out, +so where it rarely fires the row pays for it. The port was an out-of-sample test of the API and +passed it: **zero framework changes were needed**, matching the element vocabulary's earlier record +(`TensorRow`, `GeometryRow`, `TensorSink`, each added in its own crate). + +**Deleting the hand-written path made its shape faster.** With `Extension`-over-constant visible to +the stride-0 decode, cosine's `reduce_encoded` constant routing (manufacture an `L2Denorm` from a +constant operand, answer through the denorm paths) became deletable. Its shape then sped up: + +| width | through the deleted rewrite | through the row loop + prepare | +| --- | --- | --- | +| 2 | 118.8 us | **63.08 us** | +| 32 | 554.0 us | **377.9 us** | +| 256 | 5.159 ms | **3.007 ms** | + +Both constant spellings now measure identically (63.08 vs 62.72 us at width 2). The hand-written +fast path was 1.5-1.9x slower than the framework path that replaced it, on top of having missed the +literal shape entirely. That is the dedup argument in its strongest form: not fewer lines, but +fewer wrong ones. + +### The one unenforceable thing + +The design's benefit rests on LLVM treating the per-row branch on the prepared `Option` as +loop-invariant. Three outcomes exist per call site: unswitched (intended), if-converted (both arms +computed, the hoist silently evaporates while staying correct), or retained (a branch in a cheap +scalar kernel can block vectorization). For every real adopter the hoisted work is a loop or a +parse, which cannot be speculated, so the worst case degrades to one predicted branch per row, the +same cost class as the bounds check kept over `unsafe`. It is still a hope rather than a contract, +and the convention that polices it is stated in the trait-choice guide: every adopter lands with a +constant/non-constant benchmark pair, and the non-constant arm must not move. + +### Rejected alongside + +- **`Const` wrapper elements**: needs value-aware dispatch; splits plan/run; 2^arity dispatch + arms. Dead on the purity invariant. +- **Closure-internal `OnceCell` memoization**: wrong without constness plumbing, redundant with it. + Distinct from the `OnceCell` *inside `P`* that contains uses, which is constness-aware and only + defers an expensive build. +- **Plan-time currying through `reduce`** (folding a Literal into Options as a compiled variant): + the only design that amortizes across batches, deferred because `PersistableOptions` admits only + the source value, it misses every run-time-only constant, and re-currying bifurcates function + identity, silently detaching encoding kernels keyed on the original function. Revisit only if + per-batch prepare cost ever measures as material. +- **`visit_prepared_into`** (sink plus prepare): no user. `l2_denorm`'s constant case is a bulk + answer in `reduce_encoded`, not a prepared loop. The asymmetry is deliberate and cheap to fix + when a user appears. + +--- + +## Is there anything left to port? + +Asked directly: could the remaining hand-written vtables move onto `RowFn` if the element vocabulary +covered more types? Classifying all ~30 of them says no, and says the vocabulary is not what is +stopping them. + +| blocker | count | members | +| --- | --- | --- | +| **Not strict.** `RowFn` implies strict, so these cannot reach it at all. | 12 | `between`, `case_when`, `cast`, `dynamic`, `fill_null`, `is_null`, `is_not_null`, `list_contains`, `pack`, `stat`, `row_size`, `zip` | +| **The answer already exists in bulk.** Zero-copy child projection, a metadata field, or a vectorized slice kernel. A row loop would be strictly slower. | 12 | `not`, `list_length`, `binary`, `mask`, `ext_storage`, `get_item`, `select`, `merge`, `variant_get`, `geo.envelope`, `json_to_variant`, `row_encode` | +| **No element rows to read.** Zero-arity, or a type-erasure adapter. | 5 | `literal`, `root`, `row_idx`, `row_count`, `ForeignScalarFnVTable` | +| **Output side.** Nullable output, or an output dtype that depends on runtime data. | 2 | `list_sum`, `geo.envelope` | +| **Value-dependent per-batch setup.** | 1 | `like` | + +`geo.envelope` is the one function counted twice: its output is a struct-of-four extension type *and* +its fast paths hand back existing child arrays untouched. + +`binary` deserves a note, since on strictness alone it looks portable: only its Kleene `And`/`Or` are +non-strict, and `is_strict` already varies by operator, so comparison and arithmetic go through the +strict lifting today. What keeps it columnar is the kernel. `collect_zip_bits` and `LaneZip` run over +`as_slice()` pairs as tight vectorizable loops, with a separate constant-operand path +(`collect_bits(lhs, |a| a.is_eq(rhs))`). Routing that through a per-row closure and `ArgColumn::get` +would give up the slice-level vectorization for nothing. + +Three things follow. + +**The porting well is dry.** The eight functions on `RowFn` (`byte_length`, the four tensor kernels, the +three geo kernels) are the complete set in this repository that wants a row loop. Every remaining one is +blocked, and forcing any of them onto `RowFn` would cost performance rather than save lines. `l2_denorm` +was the last one the vocabulary was actually keeping out, and the sink let it in. + +**Missing elements are not the constraint.** Only `list_contains` would need new input vocabulary, and +it is independently blocked by non-strictness, so a list element would not unblock a single function +today. A list *input* element is nonetheless easy (`Bytes` already proves the shape: `Elem<'a>` is a +GAT, so `&'a [T]` works), and `list_length` could even be a `RowFn` given a `ListLen` element in the +style of `BytesLen`. It should not be, because its answer is a child array or one constant. + +**`like` is a new gap, and the sharpest one.** It is strict, infallible, `(Utf8, Utf8) -> Bool`: on +signature alone it is the ideal `RowFn`. Two things block it, and measuring both is what settled where +it belongs. + +Its constant-pattern path is fine. `reduce_encoded` already sees the argument arrays before the row +loop, so compiling the pattern once and evaluating in bulk has a home, and a constant operand stays +constant even through a filtered batch. No new hook needed for that case. + +Its *per-row* pattern path is what blocks it. That path memoizes the compiled pattern across +consecutive rows carrying the same one, and a `RowFn` closure is `impl Fn`, so it can hold no such +state. Defeating the cache costs **5.7x** (`like_per_row_distinct_patterns` 249.1 µs against +`like_per_row_patterns` 44.03 µs, 2048 rows, same matching work in both), which is the same shape of +regression the constant-operand stride fixed for geo. + +Relaxing the closure to `impl FnMut` would restore the cache, and it compiles as a one-word change. +It is not free. Measured on `byte_length_element`, `fastest` column, both configurations run twice: + +| case | `Fn` | `FnMut` | delta | +| --- | --- | --- | --- | +| `long_strings_bytes_len` 4096 | 11.15 µs | 12.08 µs | +8.3% | +| `long_strings_bytes_len` 65536 | 166.4 µs | 181.7 µs | +9.2% | +| `long_strings_bytes_slice` 4096 | 14.75 µs | 15.97 µs | +8.3% | +| `short_strings_bytes_len` 65536 | 166.2 µs | 180.4 µs | +8.5% | +| `short_strings_bytes_slice` 65536 | 180.9 µs | 200.3 µs | +10.7% | + +Capturing the closure by `&mut` inhibits the vectorization the shared capture allows, so `FnMut` +taxes every row function 8 to 11% to enable state that one function wants. Keep `visit` on `Fn`. + +The conclusion is that `like` does not want a row loop at all: its general path needs cross-row state, +and its fast path is bulk. What it wants is to declare `(Utf8, Utf8) -> Bool` through the element +vocabulary and keep its own kernel, which is the missing cell below. A per-batch setup hook would not +have been enough on its own, since the state `like` needs is mutable *across* rows rather than fixed +before them. + +A second, smaller thing blocks `like` too: it renders custom SQL through `fmt_sql`, and neither +`StrictScalarFnVTable` nor `RowFn` forwards that, so today porting any function with bespoke SQL +rendering would silently lose it. + +--- + +## Known gaps and future work + +Found by the porting probes, left unfixed here because each is a larger change with its own review +surface. Recorded so they are decisions rather than surprises. + +- **~~No constant-operand affordance.~~ Fixed twice over.** A partially-constant call used to decode + the constant column in full, so a broadcast operand cost one decode per row (measured: a broadcast + query vector cost the same as a genuine column, 234 ms vs 226 ms at 50k x 256). That was what kept + the geo functions off `RowFn`. Each decoded column now carries a stride, 0 for a constant, and the + geo functions are row functions. Constant *compute* was the remaining half, closed by + `visit_prepared` (see [Constant compute](#constant-compute-the-last-quadrant-of-the-lifting)). +- **`NullHandling::Dense` is chosen on safety alone, with no cost input.** For a fixed-width element + (`TensorRow`) dense is unambiguously cheaper. For an unbounded-width row (a nested list) the garbage + behind a null row need only be *in bounds*, so it can span the whole elements array, which is + pathologically O(nulls x elements). No current function hits this, but the choice should consider + width. +- **`OutputElement::build(Vec)` forces materialization.** A row function's output is always a + freshly built `Vec` turned into a `PrimitiveArray`, so it cannot return a `ConstantArray` or a lazy + child. This is why `list_length` is a columnar `StrictScalarFnVTable` rather than a `RowFn`, since a + row port would materialize one `u64` per row and lose the `FixedSizeList` constant. A columnar output + escape that stays inside the framework ("given the decoded columns, can you produce the whole output + at once?") would let `list_length`, `byte_length` and `not` share one abstraction. +- **The missing cell.** The two authoring traits cover *declare-signature-once + row-loop* (`RowFn`) + and *hand-write-signature + own-kernel* (`StrictScalarFnVTable`). The cell for + *declare-signature-once + own-kernel* is empty, so a columnar function hand-writes five signature + methods (`arity`, `child_name`, `return_element_dtype`, `null_handling`, `is_fallible`) that are all + mechanically derivable from an element tuple. + + **It is buildable.** The obvious worry is coherence, since `RowFn` already blanket-impls + `StrictScalarFnVTable` and a second blanket impl of the same trait is a hard E0119 conflict. The way + through is to layer rather than branch, putting the new trait *between* the two: + + ```text + StrictScalarFnVTable <-blanket- StrictSignature <-blanket- RowFn + ``` + + One blanket impl per edge, so nothing overlaps, and a columnar function hand-writes `StrictSignature` + while a row function reaches it through `RowFn`. Compiling the shape confirms a hand-written impl + coexists with the blanket one, including from a *downstream* crate, because within the crate that owns + the type rustc can see the blanket impl's bound does not hold. This is not a new trick here: + `impl ScalarFnVTable for V` already coexists with `Like`'s and `Between`'s + hand-written `ScalarFnVTable` impls the same way. + + **The user count is 3, not 12, and 2 of those need an element first.** Being in the columnar category + is not enough: the function's *signature* has to be expressible in the vocabulary, and + `element_dtype()` taking no arguments rules out every function whose return dtype is derived from its + input at runtime. That is most of them: `mask` returns `arg_dtypes[0].as_nullable()`, `ext_storage` + returns `ext_dtype.storage_dtype()`, `get_item` and `select` a projection of the input struct, + `variant_get` an options-derived dtype, `binary` a width negotiated between operands. What is left is + `not` (`(bool,) -> bool`, usable today), `like` (`(Bytes, Bytes) -> bool`, usable today once `fmt_sql` + forwards), and `list_length` (needs a `ListLen` element in the style of `BytesLen`). + + So this is worth building *after* the elements that give it a third user, not before. Against ~140 + lines of new trait and blanket impl it would save roughly 20 lines per function, which at one usable + caller is a wrapper with one impl. The cheap interim is to make `validate_row_args`, + `row_null_handling` and `row_is_fallible` public, which turns each hand-written signature method into + a one-liner and removes the *logic* duplication (each function currently rolling its own dtype check + and asserting rather than deriving its null handling) without adding a layer. +- **No nullable output element, so no non-total `RowFn`.** `OutputElement::build` always produces an + all-valid column, so a row kernel cannot return a null from a valid row. `impl OutputElement for + Option` is the whole fix. Left out because nothing needs it *yet*: `list_sum` would need it, but + is columnar for independent reasons too (the grouped-accumulator path and the `FixedSizeList` + constant). +- **No borrowed output element, so no zero-copy row function.** A row closure returns an + `ApplyResult`, which is `'static`, so its result cannot borrow from the input columns. Note the + asymmetry with the input side, where `InputElement::Elem<'a>` is a GAT and borrows freely. Every + `str -> str` function therefore copies: `OutputElement for String` allocates one `String` per row + and then rebuilds views from them. A string library would hit this on its first `upper`. Two + distinct fixes, of increasing scope: + - `upper`, `lower` and `replace` genuinely allocate, and want a `Cow<'a, str>` output element. That + needs `OutputElement` to grow its own lifetime GAT and `build` to take an iterator rather than a + `Vec`, so a borrowed row passes through without a copy and an owned one is built in place. + - `trim`, `substring`, `left` and `right` want more than a `Cow` can give. Their result is a + *slice* of the input, so the right kernel keeps the input's data buffer entirely and rewrites + only the views, copying no bytes. That stays columnar whatever the output element can express. + + Predicates and measurements (`starts_with`, `contains`, `byte_length`) have none of this problem + and are already the best case for `RowFn`, so the split for a string library falls along the return + type rather than the argument type. + + **A plain higher-ranked bound does not get there,** which is worth recording because it looks like + it should. Writing the visit as `impl for<'a> Fn(A::Elems<'a>) -> R::Elem<'a>` fails with + [E0582]: the `Fn` sugar puts `R::Elem<'a>` in an `Output` binding, and rustc requires the bound + lifetime to appear *structurally* in the trait's input types before a binding may reference it. An + opaque projection `A::Elems<'a>` does not count, even though it plainly mentions `'a`. Three routes + around it, measured by compiling each: + + | route | works | cost | + | --- | --- | --- | + | concrete input type instead of `A::Elems<'a>` | yes | gives up the element abstraction | + | custom callable trait with a generic `apply` method | yes | callers write a struct per kernel, not a closure, and the impl must spell `::Elem<'a>` rather than `&'a str`, or hit [E0195] | + | pass a zero-sized `Row<'a>(PhantomData<&'a ()>)` token beside the row | yes | closures survive, but every row closure grows an ignored parameter | + + The third is the one to build on: the token makes `'a` appear structurally in the `Fn`'s inputs, + which satisfies E0582 and lets the `Output` binding reference it, and plain closures still infer. + The ignored parameter is a tax on *every* row function though, so the shape to prefer is a second + visit method for lending kernels, leaving today's `visit` untouched for the `'static` majority. + + **Still open, and not what `visit_into` is.** The sink method added since is a second visit method, but + for a closure that *writes* rather than one that *lends*: its output is owned by the sink, not borrowed + from the row. A lending visit would still need the `Row<'a>` token. The precedent it sets is that + adding a third visit method costs the existing ones nothing, which is the same additive shape. + + [E0582]: https://doc.rust-lang.org/error_codes/E0582.html + [E0195]: https://doc.rust-lang.org/error_codes/E0195.html +- **~~`OutputElement::element_dtype()` takes no arguments,~~ Resolved, and not the way this predicted.** + An element's output dtype is a property of its Rust type and cannot depend on runtime data, which is + what kept `l2_denorm` columnar: it returns whole tensor rows, and a tensor's dtype carries its shape. + + Calling that a law was wrong, and the fix was recorded here as "widen `element_dtype` to take `args`". + That is *not* what shipped, and the shipped version is better. `OutputSink::sink_dtype(args)` puts the + argument-dependence on the sink, so all three `OutputElement` impls keep their no-argument + `element_dtype()` and only the thing that needs the arguments asks for them. + + This gap also named the real blocker correctly: `build(values: Vec)` with `Self = Vec` means + one heap allocation per row and then a flatten, against a columnar kernel that scales the flat storage + buffer in a single pass. At 16k rows that is 16k allocations versus zero, and no amount of dtype + plumbing fixes it. The prescription it drew, "an output element that writes into a preallocated flat + buffer (`fn apply(row, out: &mut [T])`)", is exactly what `OutputSink` is, generalized past `&mut [T]` + so a byte buffer works too. See + [the audit](#audit-can-the-four-strictscalarfnvtable-impls-really-not-be-rowfn) for what it cost and + bought. + + Note also what *not* to do on the input side: replacing the generic `TensorRow` with a + non-generic element whose `Elem<'a>` is an enum over `f16`/`f32`/`f64` would move the width choice + from monomorphization into a branch inside the row loop. That is precisely what + `match_each_float_ptype!` plus a generic element exists to avoid, so it would cost every tensor + kernel its inner-loop specialization. +- **~~The witness carries four scalars through two associated types.~~ Not a gap.** This looked like + the framework's weakest joint, since `ArgsWitness` and `RetWitness` are read *only* for `ARITY`, + `DENSE_SAFE`, `DECODE_FALLIBLE` and `FALLIBLE`, and for a multi-dispatch function the witness names + an arbitrary representative (`L2Norm` says `f64` for no reason a reader can see). The plan was to + collapse them into three consts. + + Checking the signatures says no. `arity`, `null_handling` and `is_fallible` on + `StrictScalarFnVTable` all take *only* the options, with no input dtypes, while `dispatch` needs + dtypes to choose. So those three answers **must** be dtype-independent, which means they cannot be + read off whatever element types a batch picks, which is exactly why a separate declaration has to + exist. The witness is not redundant bookkeeping; it is the only place those facts can live. + + Given that, types beat consts. With types, dense-safety and fallibility are *derived* from the + element types, so the only available mistake is a witness that disagrees with the dispatch, and that + is a build error. With three hand-written consts an implementor could state a fact wrongly *and* + visit consistently with their mistake. Converting would be a notation change that removes a + derivation, not a fragility fix. Left alone, with the reason now recorded on `ArgsWitness` so the + next reader does not re-open it. + + What is left of the original complaint is presentational: the arbitrary representative reads oddly. + A doc line on each multi-dispatch implementor saying why the width shown is arbitrary is the whole + fix. +- **`InputElement` is an open trait with required consts.** Adding `DECODE_FALLIBLE` broke every + out-of-crate element (`TensorRow`) until updated. If elements are a real extension point for other + crates, `DENSE_SAFE` / `DECODE_FALLIBLE` should carry conservative defaults. +- **`DENSE_SAFE`'s doc guidance is subtly wrong for lists.** It says `false` for "any element that + follows an offset," but a list element *is* dense-safe, because list arrays validate + `offsets[i] + sizes[i] <= elements.len()` for every row including nulls. Following the doc literally + would put `list_length` on `Filter` and lose its encoding fast paths. + +--- + +## What the ports bought + +**Not line count.** That was the first justification I reached for and it does not hold up: `row/` is +514 code lines and `strict/` is 269, against roughly 470 lines saved across six kernels. Near +break-even. Nor is it bug fixes, since none of the three extracted problems is a live miscompute on +`develop`. + +**It is `unsafe`.** Every hand-written kernel in `vortex-tensor` ended the same way: + +```rust +// SAFETY: The buffer length equals `len`, which matches the source validity length. +Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) +``` + +A kernel that computes its own values *and* carries its input's validity has to assert that the two +lengths agree, and the only tool for that is `new_unchecked`. The framework never pairs them: +[`OutputElement::build`] returns a non-nullable column, and the strict lifting applies validity +afterwards by masking. The invariant stops being asserted and becomes unrepresentable. + +Counting production `unsafe` blocks, test modules excluded: + +| function | layer it moved to | `unsafe` on `develop` | `unsafe` now | +| --- | --- | --- | --- | +| `l2_norm` | `RowFn` | 1 | 0 | +| `inner_product` | `RowFn` | 3 | 0 | +| `cosine_similarity` | `RowFn` | 3 | 0 | +| `l2_denorm` | `RowFn` (was `StrictScalarFnVTable`) | 8 | 6 | + +**This started as a controlled experiment and the control has since been ported, so read it in two +stages.** For most of this branch's life `l2_denorm` stayed on `StrictScalarFnVTable` and held all 8 of +its blocks while the three functions that moved onto the row layer lost all of theirs. Same crate, same +reviewers, same standards, so the row layer was what removed them rather than the strict lifting or the +port itself. That is the inference the control bought, and it is still the argument. + +`l2_denorm` then moved onto the row layer too, via `OutputSink`, and dropped to 6. The two it lost are +exactly the memory-safety ones on its kernel path, which is the pattern the other three showed. Of those +two, one (`FixedSizeListArray::new_unchecked` in the constant-norms path) is attributable to the port and +one (`PrimitiveArray::new_unchecked` in `build_tensor_array`) is an independent cleanup noticed along the +way. Its 6 remaining blocks are a different kind and are not the row layer's business: four call +`L2Denorm::new_array_unchecked`, an `unsafe fn` guarding the *semantic* unit-norm invariant rather than +memory safety, and two are buffer pushes in `normalize_as_l2_denorm`, a helper that is not a scalar +function. + +`develop`'s `l2_norm` also hand-rolled a 25-line constant-array fast path that the strict lifting now +does generically for every function, and computed its output nullability by hand. + +This is the justification to carry onto a clean branch. It also bounds the claim: a `vortex-tensor` +local helper owning the same invariant would remove the same `unsafe`, so what earns the *generic* +placement in `vortex-array` is that `vortex-geo`'s three predicates and `byte_length` use it too, +over three different element types. Two downstream crates plus core is the second-caller test met, not +anticipated. + +### What it costs + +Removing that `unsafe` is not free, because `new_unchecked` was buying something: the old kernel paired +its freshly built buffer with the input's validity in one step, so a nullable input cost it nothing +extra. The framework builds a non-nullable column and the lifting applies validity afterwards, which +for `Validity::Array` means materializing a mask and running a separate pass. + +That pass is `O(rows)` while the kernel is `O(rows * width)`, so width amortizes it. Measured on +`vortex-tensor/benches/l2_norm.rs`, 16384 rows, `fastest` column: + +| width | non-nullable | nullable | cost of the extra pass | +| --- | --- | --- | --- | +| 2 | 68.87 µs | 70.44 µs | +2.3% | +| 32 | 241.4 µs | 243.9 µs | +1.0% | +| 256 | 2.513 ms | 2.529 ms | +0.6% | + +So 1 to 2% on nullable input, worst at the narrowest vector anyone would store, and nothing at all on +non-nullable input where no mask is applied. Trading that for eight memory-safety `unsafe` blocks is the +right side of the deal. + +These figures are near this machine's noise floor and should be re-confirmed on quieter hardware before +being quoted. The larger measurements in these notes (the 5.7x `like` cache loss, the 8 to 11% `FnMut` +tax, the 2x width-2 per-row cost and its removal, the 2x `l2_denorm` sink win) are well clear of it. + +### The like-for-like comparison, and the per-row cost that was hiding in it + +The table above compares the framework against itself, so it isolates the masking pass but says nothing +about the rest of the machinery. `PrePortL2Norm` in the same benchmark closes that: a bench-local +`ScalarFnVTable` running the identical arithmetic, indexing the flat slice directly into a `Buffer` and +attaching validity in one step. + +This measurement found a real defect in the tensor element, and the diagnosis recorded here first was +wrong in a way worth keeping visible. + +**What was measured, and the wrong inference.** `fastest` column, non-nullable, 16384 rows: + +| width | framework | pre-port | delta | +| --- | --- | --- | --- | +| 2 | 68.85 µs | 32.85 µs | **2.10x slower** | +| 32 | 266.6 µs | 255.5 µs | +4% | +| 256 | 2.564 ms | 2.512 ms | +2% | + +The gap in absolute terms is 36 µs at width 2 and 11 µs at 32, and the conclusion drawn was "a cost that +shrinks as total work grows is a constant being amortized, so the framework carries tens of microseconds +of fixed per-batch setup." That reasoning does not hold. 36 µs over 16384 rows is 2.2 ns/row, which is a +*per-row* cost; it stops showing at width 32 because the kernel there is memory-bound and absorbs extra +CPU work in its stalls. Reading "shrinks with width" as "fixed per batch" skipped dividing by the row +count. + +**The actual cause was one per-row accessor, in the tensor element.** `TensorRow::get` called +`FlatElements::row::(i)`, which per row re-derived its typed slice: a ptype comparison against the +stored `PType`, a host-buffer downcast out of the buffer handle, a length division, and then two range +indexings with a bounds check each. All of it loop-invariant except the offset. This is exactly the +hidden-cost-accessor pattern the repository guidelines warn about, and it was written into the element +rather than found in the framework. + +The fix types the column at decode time instead of per row. `TensorRow` is already generic over `T`, +so its `Column` can be a `Buffer` plus a stride, and `get` becomes one multiply and one range index +into a typed slice. `FlatElements` keeps its untyped `row` for the callers that read a handful of rows. + +**After, same bench, same run:** + +| width | framework | pre-port | delta | +| --- | --- | --- | --- | +| 2 | **33.32 µs** | 32.83 µs | **parity, 1.01x** | +| 32 | **227.4 µs** | 258.9 µs | framework **1.14x faster** | +| 256 | **2.422 ms** | 2.522 ms | framework **1.04x faster** | + +The pre-port column is stable across both runs (32.85 then 32.83 µs at width 2), which is what makes +this comparison trustworthy; only the framework side moved. `l2_denorm` gained the same way, from +88.0 µs to 48.9 µs at width 2, since it reads its tensor argument through the same element. + +Three things follow. + +**The row layer was never the cost.** The 2x was one accessor in one element implementation, and the +generic machinery around it (the visitor, the witness, the strict lifting's bookkeeping, `reduce_encoded`'s +probe, the dispatch width match) does not measurably show up at 16384 rows. The planned decomposition +into "strict lifting versus row layer" is moot: neither was it. + +**An element is a performance-critical surface, and nothing in the framework says so.** `InputElement::get` +is documented as needing to be `O(1)`, which `FlatElements::row` technically was. `O(1)` is the wrong +contract; the right one is that `get` must not repeat work that is constant across the batch, because it +is the one function called once per row. `decode` exists precisely to hold that work, and the element +vocabulary's whole promise (anyone can add an element in their own crate) means this trap is now +available to every future implementor. + +**The framework being generic is what let one fix pay out twice.** `l2_norm`, `inner_product`, +`cosine_similarity` and `l2_denorm` all read tensor rows through this element, so a single change moved +all four. That is the case for the shared layer stated in performance terms rather than in line counts. + +### What the harness actually costs, from the optimized IR + +The measurements above say the harness is free at 16384 rows. Reading the post-optimization LLVM IR says +*why*, and settles whether more `#[inline]` would buy anything. Emitted with +`cargo rustc --release -p vortex-tensor --lib -- --emit=llvm-ir -Cdebuginfo=0`, reading the `l2_norm` f64 +arm. + +**The whole stack is already one function.** `execute_row_loop`, `ElementTuple::get` and the row closure +have no `define` of their own anywhere in the module. They survive only as basic-block *labels* carrying +`.exit.i.i.i…` suffixes about sixteen `.i` deep, which is inline-depth notation: the engine's +`ScalarFnVTable::execute`, `execute_dense`, `execute_strict`, `dispatch`, `RowVisitor::visit`, +`execute_row_loop`, `A::get` and the closure are all inlined into a single body. Adding `#[inline]` +anywhere on that path cannot help, because nothing on it is still a call. + +**Per batch the harness leaves five calls**, each correctly placed outside the loop: one +`ArgColumn::decode` per argument, one `tensor_element_ptype` for the width match, one `reduce_encoded`, +one `OutputElement::build` after the loop exits, and the output allocation. + +**Per row it leaves this, and nothing else:** + +```llvm +%row = phi i64 [ 0, %preheader ], [ %next, %loop_latch ] +%next = add nuw i64 %row, 1 +%start = mul i64 %row, %stride ; ArgColumn's stride, fused with list_size +%end = add i64 %start, %list_size +%ovf = icmp ult i64 %end, %start ; the two halves of one slice range check +%oob = icmp ugt i64 %end, %len +br i1 (or %ovf, %oob), label %slice_index_fail, label %body ; cold side out of line +%rowp = getelementptr inbounds nuw double, ptr %elements, i64 %start +%endp = getelementptr inbounds nuw i8, ptr %rowp, i64 %list_size_bytes +... ; element loop, 8x unrolled +%out = getelementptr inbounds nuw double, ptr %values, i64 %row +store double %result, ptr %out +``` + +About ten integer ops and one always-taken branch. The element loop underneath is 8x unrolled with a +serial `fadd` chain (LLVM correctly refuses to reassociate the float sum) terminating on `icmp eq ptr` +against `%endp`, which is what a hand-written `iter().map(|x| x * x).sum().sqrt()` compiles to: the +`Elem<'a> = &'a [T]` GAT is fully scalar-replaced, and the slice iterator becomes pointer bumping at +fixed byte offsets. + +**The one removable cost is not worth removing.** The surviving per-row branch is the range check on +`&elements.as_slice()[start..start + list_size]`. LLVM cannot hoist it because nothing tells it +`len == rows * list_size`. Eliminating it means `get_unchecked`, and this framework's stated value is +removing `unsafe` from kernels, so buying back a perfectly-predicted branch with an unchecked index is +the wrong direction. It is also already hidden: at width 2 the row's `sqrt` alone has longer latency than +the whole index computation. + +LLVM also unswitched the row loop on `list_size == 0` and emitted a zero-width specialization that stores +`0.0` per row. Harmless, and a sign the loop was simple enough to reason about completely. + + + +[`OutputElement::build`]: vortex-array/src/scalar_fn/row/element/mod.rs + +Production lines, before and after: + +| function | layer | before | after | +| --- | --- | --- | --- | +| `byte_length` | `RowFn` (fixed) | n/a | 23 (impl) | +| `list_length` | `StrictScalarFnVTable` | 189 | 143 | +| `not` | `StrictScalarFnVTable` | 76 (impl) | 53 (impl) | +| `list_sum` | `StrictScalarFnVTable` | 78 (impl) | 56 (impl) | +| `l2_norm` | `RowFn` (width) | 254 | 96 | +| `inner_product` | `RowFn` (width) | 277 | 112 | +| `cosine_similarity` | `RowFn` (width) | 309 | 203 | +| `l2_denorm` | `RowFn` (width, sink) | 731 | 618 | +| geo x 3 | `RowFn` (fixed) | 51 each (impl) | 15 each (impl), plus one shared element | + +Nothing outside the functions' own crates changed: the `L2DenormScheme` compressor and every +`ExactScalarFn` matcher are untouched, because the encoding-aware push-downs key off the function +*type* rather than its vtable layer. + +The line-count case does not close on its own. The framework is ~1670 production lines (up from ~1510 +before the sink, which added `result.rs`, `sink.rs` and a second visit path) and removes ~870 across the +ported functions, so **net this branch adds lines**, amortizing around the fourteenth function against +~20 strict candidates in the tree. To be honest, the case for merging is the marginal +cost of the *next* function (~15 lines, and the invariants above enforced rather than reviewed), plus +the correctness the type-derived properties buy, rather than the diff. + +--- + +## Measurements + +`vortex-array/benches/byte_length_element.rs`, element choice for `byte_length`, whole-execution +medians: + +| input | `BytesLen` | `Bytes` | | +| --- | --- | --- | --- | +| 64Ki non-inlined rows | **206 µs** | 256 µs | 24% faster | +| 64Ki inlined rows | **207 µs** | 215 µs | 4% faster | + +`vortex-array/benches/strict_validity.rs`, how the `Dense` path applies validity, same kernel in both +arms: + +| | `lazy` | `eager` | | +| --- | --- | --- | --- | +| 64Ki, one call | **9.0 µs** | 75.3 µs | 8.3x faster | +| 1Mi, one call | 1.357 ms | 1.357 ms | parity | +| 64Ki, chain of 3 | **28.3 µs** | 30.6 µs | 7% faster | + +`Validity::and` is already lazy, so the conjunction is never materialized to be applied. Only +`NullHandling::Filter` needs positions, and only it pays for them. + +`not`, word-wise kernel against the row loop it would have if it were a `RowFn` (release, identical +outputs asserted): + +| len | word-wise `!` | row loop + `bool::build` | +| --- | --- | --- | +| 64Ki | 927 ns | 376 µs (**406x**) | +| 1Mi | 10.3 µs | 5.83 ms (**569x**) | + +This is why `not` is a columnar `StrictScalarFnVTable` rather than a row function. + +--- + +## Rejected alternatives + +- **A wrapper type instead of a blanket impl** (`Strict`): forces churn at every call site, + meaning matchers, kernel registrations, and expression constructors. The blanket impl means a port + edits only the function's own impl block. +- **A `row_family!` macro, a per-crate GAT family, or a framework GAT family**: three encodings of + "element types as a function of the width," all paying for the same limit (the width bound has to + appear literally in a GAT), so each width class needed its own trait *and* adapter. The rank-2 + visitor replaces the whole lineage with one non-generic trait method and no generated code. +- **`ElementwiseFn` as a third trait**: subsumed by `RowFn` with a constant dispatch, see above. +- **One `RowFn` with defaulted `dispatch` and `apply`**: converts "define nothing" from a compile + error into a runtime panic. +- **Renaming `StrictScalarFnVTable` to `TotalFnVTable`**: the trait admits non-total members on + purpose, so the name would be wrong. +- **An `is_total` method feeding a derived `validity`**: a new concept to compute what a function can + state directly. Mirroring `validity` with a `None` default makes the unsound answer the one that + takes work. +- **Macro-generated per-type constructors**: a bespoke API per function, where the general + `ScalarFnFactoryExt::try_new_array` is what every other scalar function already uses. +- **A separate `FallibleElementwiseFn`**: an associated return type (`ApplyResult`) costs one line per + function instead of a whole trait and a spent coherence slot. + +## Null strategies and the non-strict frontier + +The question that opened this chapter: with the strict trait retiring into a private lifting under +`RowFn`, could the row framework also serve non-strict functions, where the kernel sees each input +as an `Option` and owns null semantics itself? The prior expectation was "probably not useful or +performant, but worth establishing why." The answer splits into three verdicts, one per axis, and +the investigation surfaced a fourth result nobody asked for that is worth more than the question. + +Method: a survey of every non-strict `ScalarFnVTable` impl in the workspace plus every consumer of +`is_strict` and `validity()`, and a working prototype (worktree branch `proto/null-strategies`, +2,034-line diff, not for merging) that implemented both a branch-and-skip execution strategy and a +`Nullable` input element, benchmarked on 65,536-row batches at null densities from 0% to 90%. +All 435 vortex-array scalar_fn tests and 223 vortex-geo tests pass with the prototype strategy both +off and on, including new hostile tests (out-of-bounds views and poison divisors behind null rows) +proving the kernel never runs behind a null. + +### Verdict 1: null-visible inputs have no customer, and now we know the price + +The survey found 15 non-strict functions. Thirteen are cheap columnar mask algebra or pure +structure. The canonical case is Kleene `AND`: a fused kernel computing values and validity +together at roughly six bitwise ops per 64 rows, with validity `(lv & rv) | (lv & !l) | (rv & !r)`. +The prototype measured a row-function Kleene `AND` over `(Nullable, Nullable)` against +it: **250x to 1,030x slower** depending on density. That is the honest price of spelling bitwise +logic one row at a time, and no framework design recovers it. + +The remaining two, `RowEncode` and `RowSize` in vortex-row, are the only genuinely expensive +null-visible per-row kernels in the tree, and they are excluded by something the Option tier does +not touch: they are variadic over heterogeneous column types with a shared per-row write cursor, +which the fixed-arity tuple witness cannot express. Null-visible inputs alone unlock nothing. + +Four functions (Kleene `AND`/`OR`, `zip`, `case_when`, `list_contains`) have **value-dependent +output validity**: `false AND null` is a *valid* `false`. For these no validity expression over +child validities exists even in principle, so the lifting's derivations (validity expression, mask +motion, dictionary push-down eligibility) are unavailable by definition rather than by +implementation gap. Any future Option-input tier must let the kernel author value and validity +together, which is to say it must be a different trait, not a mode of this one. + +What `is_strict = false` forfeits is exactly enumerable: the dictionary values push-down +(`arrays/dict/compute/rules.rs`), the dict-layout below-decode push-down +(`vortex-layout/src/layouts/dict/reader.rs`), and, when `validity()` is also `None`, lazy validity +on an unexecuted `ScalarFnArray` degrades to executing the kernel to read its nulls. Nothing in +vortex-scan, vortex-file, or the engine integrations consumes strictness. + +Mechanically, `Nullable` works exactly as sketched: `Elem<'a> = Option>`, decode +materializes the validity mask once, `get(i)` consults it, `DENSE_SAFE = true` by construction. +Niche packing is free for every by-reference element (`Option<&[u8]>`, `Option<&str>`, +`Option<&[T]>`, `Option<&Geometry>`, `Option` all compile-time asserted same-size) and +doubles every by-value primitive, which are precisely the elements that were already dense-safe +and never needed a strategy. The prototype's geo `contains` over `(Nullable, const)` +tracked branch-and-skip within 2-8%, so the shape is viable for a kernel that wants null +visibility for semantic reasons. Nothing in the tree does. **Do not build it; keep the survey's +constraint list for whenever a real variadic or null-visible demand shows up.** + +### Verdict 2: Option outputs inside the strict tier are the real demand + +Strictness is a subset bound, `valid(out) ⊆ valid(in)`, so a kernel that turns a valid row into a +null is still strict, and the strict lifting already keeps kernel-produced nulls, unioned with the +lifted ones. What excludes such functions from `RowFn` today is only the all-valid-output rule on +`OutputElement`. Two in-tree functions are shaped exactly like this: `list_sum` (a valid empty +list sums to null; the module doc names it as the canonical exclusion) and `variant_get` +(expensive per-row path traversal where a missing path yields null). The extension is small and +local: an `Option` output form whose element dtype is nullable and whose build sets validity, +`RetWitness` gaining a nullability bit alongside `FALLIBLE`, and the derived `validity()` moving +from `union_child_validities` to `None` for such functions, which costs them lazy validity but is +already the status quo for both named candidates. `is_strict` stays `true`. **This is the piece +worth building.** + +### Verdict 3: branch-and-skip, the result nobody asked for + +Today the derived null handling is binary: `Dense` (run over garbage, mask after) when every +element is dense-safe and the kernel infallible, else `Filter` (filter every input to the +conjoined-valid rows, run, scatter back). The prototype added the missing third strategy: +materialize the conjoined mask once, run over the *unfiltered* inputs visiting only set rows +word-at-a-time (`BitBuffer::for_each_set_index`), pre-fill the output with garbage, mask exactly +as Dense does. Fallible kernels stay sound because apply never runs behind a null. + +Measured against Filter at 65,536 rows (divan fastest, two runs): + +| workload | 1% nulls | 10% | 25% | 50% | 90% | +| --- | --- | --- | --- | --- | --- | +| `byte_length` at `Bytes` (cheap kernel) | branch 1.8x | 2.6x | 3.8x | 4.7x | **5.9x** | +| geo `contains`, one nullable operand | branch 1.07x | 1.11x | 1.18x | 1.11x | filter 1.38x | +| geo `contains`, two nullable operands | branch 1.06x | even | filter 1.2x | filter 1.9x | filter 11.3x | + +For the cheap kernel Filter never wins: at even 1% nulls, filtering the input plus scattering the +output costs more than the entire branch-side loop. For the expensive kernel the governing +quantity is the **surviving-row fraction**: branch pays O(n) decode regardless, Filter pays +O(survivors) decode plus filter and scatter. Geo's ablation makes the mechanism explicit: filter +plus scatter are under 4% of `contains`' total, so Filter's entire advantage at sparse validity is +the shrunken arrow-export-and-parse, while for `byte_length` those same two steps are 20-40% of +Filter's total and pure waste. Crossover lands near 50-75% surviving rows for one nullable operand +and lower with two (the conjoined fraction shrinks quadratically). + +The strategy is invisible to function authors: it slots under the existing derived null handling, +selectable per batch from `Mask::true_count`, with Filter kept for the sparse tail. **This is now +implemented on this branch** (see "Adaptive null strategy, as shipped" below); the rest of this +section records the prototype evidence that justified it. The prototype +also validated the two supporting pieces: a null-tolerant `decode_branch` on `InputElement` +(defaulting to plain decode, correct for bulk canonicalizations) and `OutputElement::garbage()` +for pre-fill. Production caveats recorded in the prototype report: `reduce_encoded` is not +consulted on the branch path, sinks fall back to Filter, the toggle must become per-execution and +cost-based, and geo's null-tolerant decode covered Point and Polygon only, still paying a +full-length arrow export that a run-slicing decode would shrink. The prototype's conclusion, since borne out: `Bytes`-element functions were paying the +Filter tax on every nullable batch, and most of it is recoverable. + +### Adjacent findings, recorded so they are not relearned + +- `Between::validity` declares the strict three-way conjunction while its fallback execute path + joins two comparisons with Kleene `AND`; with per-row nullable bounds the lazy validity and the + executed result disagree (a valid `false` reported as null). Pre-existing on develop, + independent of this work, slated-for-removal expression; deserves an issue. +- `not` is already at the optimum reachable through the current ownership model: `to_bit_buffer()` + is a handle clone, the source array keeps the buffer shared, so in-place negation (a real 19% on + uniquely owned buffers) is unreachable without redesigning `ExecutionArgs` ownership. Encoded + NOT flows through `NotReduce` (Constant, Sparse) and generic per-encoding push-down (Dictionary, + RunEnd) at 13-24x below canonical cost; `NotKernel` has no implementations and looks like dead + code. The three columnar ports of the retired strict trait revert entirely. +- The strict lifting's small-batch overhead is generic prelude bookkeeping (collect inputs, + compute the declared dtype, conjoin validity), not any single avoidable allocation; ablations + including SmallVec found nothing independently beneficial, and the earlier -10%-at-100-rows + reading did not reproduce uniformly. The row layer can eventually monomorphize the prelude over + its compile-time arity (`[ArrayRef; N]` via the tuple witness), which is the only structural + answer if small batches ever matter. + +## Adaptive null strategy, as shipped + +Branch-and-skip is implemented as a third null strategy, chosen per batch by the lifting. Nothing +about a function's definition changes: the row layer already derived `Dense` or `Filter` from the +element types, and `Filter` now names a *contract* (the kernel never sees a row null in any input) +rather than a mechanism. Two mechanisms satisfy that contract, and the lifting picks between them +where the conjoined mask is materialized. + +The selection rule needs one fact the framework cannot infer, so elements state it: +`InputElement::DECODE_SHRINKS_WHEN_FILTERED`, defaulted `false`, is `true` for an element whose +decode parses every row (geometry from coordinate storage) and `false` for a bulk canonicalization +(bytes, bools, primitives). Getting it wrong is a performance bug, never a correctness bug. +`ElementTuple` ORs it across arguments, the witness check pins it like dense-safety and +fallibility, and the rule is: + +```text +branch-and-skip, UNLESS some argument's decode shrinks when filtered + AND fewer than BRANCH_MIN_SURVIVING_FRACTION (0.75) of rows survive +``` + +Two supporting hooks: `InputElement::decode_null_tolerant` (defaults to the ordinary decode, sound +because the branch loop never resolves an unset row, so hostile bytes behind a null are never +touched) and `OutputElement::placeholder` (the pre-fill written behind nulls, masked before anyone +observes it). Geo overrides the decode for Point and Polygon; other geometry types report +unsupported and the selection falls back to Filter, which is tested rather than asserted in a +comment. Sinks stay on Dense/Filter, documented at the visitor. `reduce_encoded` runs on the +branch path over the *original* encodings, which is strictly better for encoding fast paths than +Filter's canonical copies, and its contract doc now states the row count differs per strategy. + +The original forced-filter, forced-branch and auto measurements used 65,536 rows on a shared 4-vCPU +VM: + +| workload | 1% | 5% | 10% | 25% | 50% | 90% | +| --- | --- | --- | --- | --- | --- | --- | +| `byte_length` at `Bytes`, auto over filter | 5.0x | 5.3x | 5.8x | 4.0x | 4.5x | 6.3x | +| geo `contains` x const, auto picks | branch | branch | branch | branch | filter | filter | +| geo `contains` x column, auto picks | branch | branch | branch | filter | filter | filter | + +Those historical rows justified shipping branch-and-skip, but they no longer calibrate the global +threshold. The controlled x86 AVX-512 rerun used a Ryzen 9 7950X pinned to CPU 4, TSC timing, a +performance governor, 60 samples for 2-4 seconds per arm, and two runs. Its representative medians +were: + +| workload | auto | branch | filter | verdict | +| --- | ---: | ---: | ---: | --- | +| one nullable, 50% nulls | 5.999-6.050 ms | 5.560-5.642 ms | 6.026-6.049 ms | auto filters, branch is 6-8% lower latency | +| two nullable, 10% nulls | 10.40-10.48 ms | 10.49-10.60 ms | 10.20-10.34 ms | auto branches, filter is 2.5-2.8% lower latency | +| two nullable, 25% nulls | 7.502-7.678 ms | 9.156-9.285 ms | 7.588-7.749 ms | auto correctly filters; filter is 1.21-1.22x faster than branch | +| two nullable, 90% nulls | 277.1-277.4 us | 3.232-3.253 ms | 277.7-278.5 us | auto matches filter; filter is about 11.6x faster than branch | + +The two misses point in opposite directions. A 50% surviving one-element decode still favors +branch, while an approximately 81% surviving two-element decode already favors filter. A single +threshold against the conjoined survivor fraction therefore cannot represent both decode cost and +arity. Replace it with per-element/arity inputs or a small estimated-cost comparison when this work +moves onto production branches. Batch size remains an unmeasured input to that model. + +Verified independently of the implementing agent: 3,441 tests pass across vortex-array and +vortex-geo (17 new: hostile out-of-bounds views behind nulls, a fallible kernel with poison +divisors behind nulls in one and both operands, conjoined-mask honoring, constant operands, real +errors still propagating, geo filter-versus-branch agreement, the unsupported-geometry fallback, +and six selection-rule cases), vortex-tensor's 164 pass unchanged, clippy `--all-targets +--all-features` is silent on both crates, and fmt and whitespace are clean. + +Open items, none blocking: the branch fallback probes `reduce_encoded` twice when the dispatch +turns out unsupported (cheap encoding check, no in-tree function affected since every +`reduce_encoded` implementor is a dense-path tensor function); geo's null-tolerant decode still +arrow-exports the full column, and slicing runs of valid rows would blunt Filter's sparse-validity +advantage enough to retire the threshold for geo; the fallible branch loop pays one `is_none` +check per set row after the first error because `for_each_set_index` cannot early-return. + +## The strict trait, deleted + +`StrictScalarFnVTable` is gone. Not made private: deleted, with its lifting kept as private +machinery under `vortex-array/src/scalar_fn/row/lift.rs`. The chain is now `RowFn` -> +`ScalarFnVTable`, one blanket impl, no intermediate trait. + +Three things converged on that. First, reverting the columnar ports left the trait with exactly one +implementor, the blanket impl over `RowFn`, and a trait with one impl is indirection rather than +abstraction. Second, the mirroring tax existed only because that blanket impl occupied the +`ScalarFnVTable` slot: `reduce` and `validity` were forwarded so a strict function could override +them despite being unable to implement `ScalarFnVTable` itself. `RowFn` keeps `validity`, because +all-valid outputs make it the child conjunction, and the `reduce` mirror went with the trait since +no adopter ever used it. Third, the naming objection a local review raised was real and is now +moot: `is_strict` names the semantic property `valid(f(x)) ⊆ valid(x)` that pushdown consumes, +while the trait demanded the *operational* property that a kernel may run over the garbage behind a +null row or over a filtered copy. Those are independent, `Bytes` being strict and not dense-safe, +so the trait was named for the wrong one of the two. + +What replaced each member: `execute_strict` and `execute_strict_branch` are the two closures +`Batch::execute` takes, `decode_shrinks_when_filtered` is a `Batch` field read off +`ElementTuple::DECODE_SHRINKS_WHEN_FILTERED`, `return_element_dtype` is what a visit returns before +`ScalarFnVTable::return_dtype` widens it, `null_handling` is `row_null_handling` over the witnesses, +and options serde is `RowFn::Options: PersistableOptions` delegated from the blanket impl. +`Batch` carries one batch's facts (id, arguments, collected inputs, conjoined validity, declared +return dtype, null handling, and the decode-shrinks flag) and takes the kernel as closures rather +than through a trait, which is the point: there is no second implementor to name. + +The one behaviour deliberately dropped is the runtime rejection of `Dense` paired with a fallible +kernel. `row_null_handling` derives the pairing from the same witnesses `is_fallible` reads, so the +combination cannot be constructed, and the requirement now lives in `NullHandling::Dense`'s doc +pointing at the derivation. Four tests went with the trait: three described a strict kernel that +returns nulls of its own (`list_sum`'s shape), which no `RowFn` can be until the `Option` output +form of open item 3 exists, and one pinned the `reduce` mirror. + +`PersistableOptions` survives with `EmptyOptions` as its only implementor, since every row function +in tree uses it. That is a bound on `RowFn::Options` rather than a speculative trait, and the +reverted `list_sum` port is what removed its second implementor. + +If a non-row columnar kernel ever wants the lifting, extract the trait then, named for the lifting +contract rather than for strictness, with that kernel as its first user. + +## Sink-only execution, the final prototype + +The last executor revision collapses every row function onto one primitive: + +```rust +visitor.visit_prepared_into::( + |constant_args| prepare(constant_args), + |state, args, output| write_one_row(state, args, output), +) +``` + +The ordinary case uses unit preparation and `ElementSink`. A tensor uses `TensorSink` so the +input dtype can determine the runtime row width. A future string transform can own one batch-wide +builder. These are not different executor modes, so the API no longer gives them different visit +methods. + +### Why the return witness disappeared + +A returning row closure needed a return witness before dispatch so `return_dtype` and fallibility +could be derived without knowing which dtype arm dispatch would select. Once every closure writes +through a sink, the sink already answers the output question: + +- `sink_dtype(args)` supplies the non-nullable element or runtime-shaped dtype. +- `with_capacity` allocates once for the batch. +- `rows` borrows the loop-local storage once. +- `row_count_matches` proves the output bound once. +- `row` hands one slot into the closure. +- `finish` builds the column and interprets any deferred error. + +`RowFn::ArgsWitness` remains load-bearing because arity and input decode properties are needed +before dispatch. `RowFn::FALLIBLE` remains because `ScalarFnVTable::is_fallible` is queried without +input dtypes. There is no analogous need for a return witness. + +The closure stays `Fn`, not `FnMut`. An earlier sink design captured `&mut Sink` in the closure and +measured 8 to 11% slower because the mutable capture blocked loop vectorization. The executor now +owns the sink, borrows its rows once, and passes a row slot as an ordinary argument. + +### Errors without a per-row result branch + +`SinkResult` has three implementations: + +- `()` for an infallible write. +- `VortexResult<()>` for an error that must exit immediately. +- `DeferredError` for a row that can write a legal provisional value and report failure after the + loop. + +Checked integer addition is the motivating deferred case. Its sink writes the wrapping sum, each +row returns a word whose sign bit means overflow, and the executor OR-reduces those words. `finish` +returns the overflow error only when the final word has its sign bit set. No `Result` discriminant +or conditional error branch is required per row. + +Nullable dense execution needs one extra rule. Garbage behind a null may overflow even when every +valid row succeeds. When dense execution finishes with a deferred error, the lifting materializes +the conjoined validity and retries only valid rows. A successful retry proves the first error came +only from discarded rows; a second deferred error is real. This preserves strict null propagation +without giving up the dense vector loop on the common path. + +This is deliberately narrow. Parsing, allocation, and any computation that cannot produce a legal +provisional row still returns `VortexResult<()>` and receives valid-row-only execution. + +### Skipped rows are a sink property + +`OutputSink::SUPPORTS_SKIPPED_ROWS` replaces the earlier blanket statement that sinks cannot use +branch-and-skip. `ElementSink` pre-fills `OutputElement::placeholder` and supports skipped rows. +A custom sink may do the same, or decline and let the lifting filter and scatter. The semantic +contract remains that skipped values are legal but arbitrary and are masked before the result +escapes. + +### Final executor measurements and IR + +The authoritative `row_fn_executor` run used 65,536 `i64` rows, 100 samples, a one-second minimum +per arm, TSC timing, CPU 4, and a performance governor on the Ryzen 9 7950X. Each cell is the range +across two runs as fastest / median: + +| workload | specialized | sink-only `RowFn` | specialized / `RowFn` | +| --- | ---: | ---: | ---: | +| checked add, two columns | 131.5-132.3 / 132.3-133.3 us | 128.4-129.6 / 129.5-130.9 us | 1.021-1.024x / 1.018-1.022x | +| checked add, column and constant | 16.90-16.93 / 17.10-17.21 us | 13.82-13.85 / 14.04 us | 1.222x / 1.218-1.226x | +| checked add, nullable columns | 133.8-134.8 / 136.1 us | 128.4-128.7 / 130.6-131.8 us | 1.042-1.047x / 1.033-1.042x | + +The native release IR has `<8 x i64>` vector error-word accumulators and +`llvm.vector.reduce.or.v8i64`. The two-column assembly is four-way unrolled over AVX-512 `zmm` +registers, producing 32 `i64` rows per iteration with four `vpaddq` instructions. Overflow bits +accumulate through vector xor/ternary-OR operations and reduce after the loop; there is no per-row +result discriminant or error branch. The specialized arm remains benchmark-local, and no production +deferred-error user exists yet. + +Other final diagnostic medians: + +- `strict_validity` lazy versus eager stayed within 2% across 65,536 and 1,048,576 rows, including + a chain of three calls. +- `byte_length_element` found `BytesLen` 1.410-1.411x faster by median than resolving a byte slice + for long strings and 1.097x for short/inlined strings at 65,536 rows. This justifies the element + choice but is not a production benchmark. +- `null_strategy_bytes` auto matched branch-and-skip; at 90% nulls it took 24.95 us against + 175.4 us for filter-and-scatter. +- Geo auto broadly tracks branch at dense validity and filter at sparse validity, but the controlled + x86 run found the two threshold misses recorded above. The full forced-strategy matrix remains an + implementation diagnostic, not permanent CodSpeed coverage. +- Distinct per-row LIKE patterns took 126.4 us against 26.87 us for a repeated pattern, 4.7x + slower. That is the measured reason LIKE remains a stateful columnar implementation. + +### Durable benchmark boundary + +Draft PR [#9136](https://github.com/vortex-data/vortex/pull/9136) now owns the stable production +benchmark names. At `bf814bbe02cb` it covers public-path byte length; signed and unsigned add, +including constant and nullable inputs; repeated and distinct LIKE patterns; tensor functions and +the `Normalized` encoding; and geo contains, intersects, and distance with constant and nullable +shapes. It also reduces the expensive overlapping-contains simulation to 1,024 rows and uses +vendored `mimalloc` in allocating binaries. + +Do not merge the research harnesses above into that permanent suite. They compare internal +strategies or frozen controls that do not exist on develop. Land #9136 first, then use its identical +benchmark names to gate each production implementation PR through CodSpeed's compiled amd64/AVX2 +simulation. Keep local Divan for real wall-clock diagnosis and generated IR for explaining a +regression. + +### Final API consequence + +Issue 9129's current sketch is obsolete: it still has `RetWitness`, `visit`, `visit_prepared`, and +`visit_into`. Issue 9130 still says sink-backed execution cannot branch-and-skip. Update both before +using their checklists to cut the implementation stack. The prototype to carry forward is: + +```text +RowFn + -> dispatches Args + OutputSink through visit_prepared_into + -> private Batch lifting chooses dense, branch-and-skip, or filter-and-scatter + -> ElementSink covers ordinary output + -> custom sinks cover runtime shape and deferred errors + -> ScalarFnVTable blanket impl exposes the function +``` + +Nullable outputs remain separate. A sink can build values plus validity, but doing so invalidates +the unconditional `validity() = union_child_validities` derivation. That semantic change should +land with its first strict non-total user, not inside the initial sink executor. + +--- + +## Final API simplification review + +This section supersedes every earlier API sketch in this document. In particular, do not carry +forward `ArgsWitness`, `RetWitness`, `PersistableOptions`, public `NullHandling`, +`DECODE_SHRINKS_WHEN_FILTERED`, or `TensorSink`. + +The review started from two constraints. The public API should expose only decisions a function +author can meaningfully make, and the executor should not trust facts fabricated by downstream +implementations. Applying both constraints removed more framework surface without preventing a +function from defining domain-specific rows. + +### The final extension boundary + +The framework is selectively sealed: + +- `RowFn` remains open. It names the function, options, argument names, fallibility, persistence, + and dtype-based dispatch. +- `InputElement` remains open. This is how a crate adds a new decoder for a geometry, tensor view, + byte view, or another domain scalar. +- `OutputElement` remains open for ordinary one-value-per-row outputs. +- `OutputSink` remains open for output representations that need their own builder or row state. +- `RowVisitor`, `ElementTuple`, and `SinkResult` are sealed because their implementations assert + executor facts used by the blanket vtable. + +Sealing `ElementTuple` does not seal decoding. The framework supplies tuple recursion for arities 0 +through 12, and a function places any open `InputElement` implementation inside those tuples. +Sealing `SinkResult` likewise does not seal output representation. A custom `OutputSink` selects one +of the supplied result behaviors. + +This keeps the author vocabulary extensible while avoiding public implementations that can lie +about arity, dense safety, result fallibility, deferred errors, or skipped-row support. + +### Dispatch contains its own evidence + +`RowFn` no longer has argument or return witnesses. `ARG_NAMES.len()` is the exact arity. The types +selected by `dispatch` carry the remaining evidence: + +```text +(InputElement, ...) + OutputSink + SinkResult + -> arity and decode properties + -> output representation and dtype + -> row fallibility and deferred-error word +``` + +The visitor asserts at compile time that the dispatched tuple arity matches `ARG_NAMES`, a +fallible decoder or result implies `RowFn::FALLIBLE`, and deferred evidence is accepted by the +selected sink. These are implications rather than equalities. A function may conservatively +declare `FALLIBLE = true` while selecting an infallible arm for some dtypes. + +This is enough for planning because dispatch is pure in `(options, args)`. It is also simpler than +duplicating the same tuple in a witness and every dispatch arm, then proving that the declarations +agree. + +### Persistence follows the function ID + +`Options: PersistableOptions` assigned one wire contract to a Rust type. That was the wrong owner. +Two functions may reuse an options type while choosing different encodings or serializability, and +an unregistered function should not invent persistence merely because its options type supports it. + +The final `RowFn` therefore owns `serialize` and `deserialize` hooks. Serialization defaults to +`Ok(None)`, and deserialization defaults to an error. Registered tensor and geo functions preserve +their explicit existing formats. The unregistered `NumericBinary` needs no otherwise-unused +serialization implementation for `NumericOperator`. + +### One custom sink is enough + +`OutputSink` already permits arbitrary internal state. A function that needs two builders defines +one sink with two fields rather than asking the executor to understand pairs of sinks. The same +rule applies to other composite or runtime-shaped results: express the shape inside one sink and +add framework abstraction only after two real users expose shared mechanics. + +The public `TensorSink` had no user after `l2_denorm` became the `Normalized` encoding. `l2_norm`, +inner product, and cosine similarity all return scalar rows through `ElementSink`. Removing +`TensorSink` avoids stabilizing roughly 90 lines of runtime-shaped row behavior without preventing +a future tensor-valued function from defining a private sink. + +`ElementSink` also no longer needs an `ElementRow` wrapper. Its row is `&mut T`, and a closure +writes with `*output = value`. The sink still pre-fills legal placeholders so branch-and-skip may +leave masked rows untouched. + +### Per-argument filtered-decode cost + +The aggregate `DECODE_SHRINKS_WHEN_FILTERED` flag was measurably lossy. OR-ing the flag made one +expensive decode indistinguishable from two, even though the x86 data selected opposite mechanisms: + +- one nullable geometry argument at 50% nulls favored branch-and-skip; and +- two independently nullable geometry arguments at 10% nulls, about 81% surviving rows, favored + filter-and-scatter. + +`InputElement::FILTERED_DECODE_COST` now defaults to zero, and each tuple adds the costs of all its +arguments. The batch selector uses the following coarse policy: + +- cost 0 always branches; +- cost 1 branches at 50% or more survivors; and +- cost 2 or greater branches at 85% or more survivors. + +The exact values come from the measured cases rather than a general cost model. There is not yet +enough evidence to distinguish two costly arguments from three, or to make the crossover depend on +batch size. Keep the value additive so a later selector can use that information without another +author-facing API change. + +The old public `NullHandling` enum is gone. The executor privately derives `Dense`, +`DenseWithRetry`, or `ValidOnly { filtered_decode_cost }`. Authors declare local safety and cost on +their input/result types, not a global mechanism. `NullStrategy` survives only in the test harness +to force branch-and-skip or filter-and-scatter. + +### Deferred errors stay in a loop-local word + +The numeric migration confirmed two constraints on deferred error evidence: + +- the accumulated word must be no wider than the element type; and +- the accumulator must live in the generated loop, not behind a mutable sink reference. + +The sealed `SinkResult` implementations for `bool`, `u8`, `u16`, `u32`, and `u64` preserve both. +Checked multiplication can report discarded high bits directly, LLVM can accumulate those words in +vectors, and `finish` turns the final evidence into the function error. `VortexResult<()>` remains +the separate early-exit form for a row that cannot write a legal provisional value. + +### Code generation after the simplification + +The final cleanup at `4becc863ae` was compared with parent `53c51d803c` using rustc 1.91.0 and LLVM +21.1.2. Both revisions were cross-compiled with: + +```bash +cargo rustc -p vortex-array --bench row_fn_executor --profile bench \ + --target x86_64-apple-darwin -- \ + --emit=llvm-ir -C codegen-units=1 -C target-cpu=x86-64-v3 +``` + +The optimized executor monomorphs were normalized to remove revision-specific symbol names and +metadata. Their vector/reduction block hashes matched exactly for wrapping add through +`ElementSink`, checked add with deferred evidence, and wrapping add through the custom `I64Sink`. + +The two wrapping paths retain 256-bit `<4 x i64>` loads, adds, and stores across six vector loop +bodies covering constant and varying inputs. Checked add retains `<4 x i64>` arithmetic, derives +overflow with vector xor/and/compare operations, ORs `<4 x i1>` evidence in the vector loop, and +reduces after the loop. The vector bodies have no calls or panic references. Scalar tails are +present in both revisions. + +The production tensor benchmark IR was checked separately for `l2_norm`, inner product, and cosine +similarity. After normalizing SSA and metadata, arithmetic sequences and instruction counts matched +between revisions for both `f32` and `f64`. Their ordered floating-point reductions remain +eightfold scalar-unrolled in both revisions. They were not vectorized before the cleanup, so the +API change did not cause that property. + +Native Apple M4 Max `row_fn_executor` timings used 65,536 rows, two alternating revisions, 100 +samples, and a 0.5-second minimum per arm. RowFn median deltas ranged from 1.11% faster to 0.94% +slower. Fastest deltas stayed within approximately 0.17%, while specialized controls had median +drift as high as 3.7%. That is no measurable native regression. + +This evidence is deliberately bounded. Cross-target optimized IR shows that the API cleanup did +not change the x86_64-v3 hot loops. It cannot establish the runtime effect of the new null selector +on an x86 branch predictor. Re-run the measured null shapes on x86 before changing or declaring the +50% and 85% thresholds settled. + +### Required x86 rerun + +The next session will run on an x86 machine. It must rerun the production comparison before this +performance record is considered complete. The #9136 benchmark baseline is now on `develop` at +`9a482c0230`, including the public binary, tensor, and geo benchmark binaries used by this work. +Fetch the latest `origin/develop`, record both exact revisions, and compare the branch against +`develop` with the same benchmark names. + +Run `binary_ops` and `like` from `vortex-array`. Run `l2_norm`, `inner_product`, +`cosine_similarity`, and `normalized` from `vortex-tensor`. Run `binary_predicates`, `distance`, +`envelope`, and `predicate_bbox` from `vortex-geo`. Use at least two alternating runs per revision. +If the host permits it, pin one core. Report both fastest and median values with the CPU, timer, and +governor configuration. + +The stable production binaries are the cross-revision gate because they now exist on `develop`. +The branch-only `vortex-geo` `null_strategies` benchmark remains the forced-policy diagnostic. Run +it on the same x86 host to verify both measured selector decisions: one costly decode at 50% +survivors must select the faster mechanism, and two costly decodes at about 81% survivors must do +the same. Inspect optimized LLVM IR again for any stable regression before changing the API or the +selector. + +### Final verification state + +The final API state recorded 67 focused RowFn tests, 179 tensor tests, and 230 geo tests. Nightly +formatting passed. Full workspace clippy passed with +`PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1`, required because the host `/usr/bin/python3` is +3.9 while the workspace targets the Python 3.11 stable ABI. + +Issues #9128, #9129, and #9130 were updated to this API. The durable public-path benchmark baseline +from #9136 is now in the repository. Earlier statements in this document that those issues or the +baseline still need updating are historical only. diff --git a/docs/strictness-and-validity-pushdown.typ b/docs/strictness-and-validity-pushdown.typ new file mode 100644 index 00000000000..d45d87a37c9 --- /dev/null +++ b/docs/strictness-and-validity-pushdown.typ @@ -0,0 +1,243 @@ +#set page(paper: "a4", margin: 2.2cm, numbering: "1 / 1") +#set text(font: "Libertinus Serif", size: 10.5pt) +#set par(justify: true, leading: 0.62em) +#set heading(numbering: "1.") +#show heading: it => block(above: 1.4em, below: 0.8em, it) +#show raw: it => text(font: "Noto Sans Mono", size: 0.88em, it) +#set table(stroke: 0.4pt + luma(65%), inset: 5pt) + +#let mask = math.op("mask") +#let valid = math.op("valid") +#let N = text(fill: rgb("#b03a2e"), weight: "bold", [NULL]) + +#let node(body, fill: luma(96%)) = box( + inset: (x: 7pt, y: 5pt), radius: 3pt, stroke: 0.5pt + luma(55%), fill: fill, body, +) + +#let lead(body) = block( + inset: (x: 10pt, y: 8pt), radius: 3pt, fill: luma(97%), + stroke: (left: 2pt + rgb("#2c3e50")), width: 100%, body, +) + +#align(center)[ + #text(size: 17pt, weight: "bold")[Strictness and validity push-down] + #v(-0.4em) + #text(size: 12pt)[the same value law, once partiality is accounted for] +] + +#v(1em) + +#lead[ + *Summary.* A row-local function may be pushed through an input's validity exactly when it is strict + in that argument *and* remains defined after validity masks that argument. The first condition is the + usual null-propagation meaning of `is_strict`; the second matters only for partial functions. It is + automatic for an infallible function. Return-dtype representability, totality, speculative errors, + and `Dense` safety remain separate concerns. +] + += Model + +Scalar functions are *row-local*: output row $i$ depends only on input rows $i$. They are also assumed +deterministic and insensitive to the bytes behind nulls. Equality below is therefore *logical equality* +$eq.triple$: equal length, equal validity, and equal values at valid rows. + +A mask is a non-nullable boolean column. It applies validity without changing valid values: + +$ mask(a, m)[i] = cases(#N &"if" not m[i], a[i] &"otherwise") $ + +For example, masking does not distinguish a newly nulled row from one that was already null: + +#figure( + table( + columns: 4, + align: center, + table.header([$i$], [$a$], [$m$], [$mask(a, m)$]), + [0], [10], [`true`], [10], + [1], [20], [`false`], N, + [2], N, [true], N, + ), + caption: [Rows 1 and 2 are both null after masking, for different reasons.], +) + +The function $f$ may be partial: an evaluation can error instead of returning a column. Statements +about its result are quantified only where that evaluation succeeds. + += The law and its missing premise + +Fix an argument position $j$. + +#lead[ + *$(S_j)$ Strictness.* If $f(a_1, ..., a_k)$ succeeds and $a_j[i] = #N$, its output at $i$ is #N. + + *$(C_j)$ Mask closure.* If $f(a_1, ..., a_k)$ succeeds, then + $f(a_1, ..., mask(a_j, m), ..., a_k)$ succeeds for every mask $m$. + + *$(M_j)$ Validity equivariance.* Whenever $f(a_1, ..., a_k)$ succeeds, the masked evaluation also + succeeds and + $ f(a_1, ..., mask(a_j, m), ..., a_k) eq.triple mask(f(a_1, ..., a_k), m). $ +] + +$(M_j)$ is the law used by a validity push-down: compute after masking one argument, or compute first +and mask the result. It includes definedness of both sides, rather than treating an error as a value. + +#pagebreak() + +For an ordinary addition, $(M_1)$ says the following two columns agree. The evaluation after masking +is defined, and strictness makes its second row null. + +#figure( + table( + columns: 6, + align: center, + table.header( + [$i$], [$a_1$], [$a_2$], [$m$], + [mask first, then add], [add first, then mask], + ), + [0], [1], [10], [`true`], [11], [11], + [1], [2], [20], [`false`], N, N, + [2], [3], [30], [`false`], N, N, + ), + caption: [The two orders differ only in the unobserved bytes behind null rows.], +) + +#lead[ + *Theorem.* For a row-local deterministic function, + $ (S_j) " and " (C_j) quad arrow.l.r quad (M_j). $ + Consequently, full strictness plus mask closure in every argument is exactly what licenses every + per-argument validity push-down. +] + +== Forward: strictness and closure imply the law + +Assume $(S_j)$ and $(C_j)$, and start with any successful evaluation +$f(a_1, ..., a_k)$. By closure, the left side below also succeeds. Fix a row $i$; row-locality means +there are only two cases to check: + +#figure( + table( + columns: (auto, 1fr, 1fr), + align: (center, left, left), + table.header([mask bit], [left: compute after masking], [right: mask after computing]), + [$m[i] = $ `true`], + [the input at row $i$ is unchanged, so this is $f(a_1, ..., a_k)[i]$], + [masking preserves $f(a_1, ..., a_k)[i]$], + [$m[i] = $ `false`], + [argument $j$ is #N; the successful left evaluation is #N by $(S_j)$], + [the mask makes the result #N by definition], + ), + caption: [Each row agrees, so the columns are logically equal.], +) + +This proves $(M_j)$. Notice the distinct jobs of the two premises: closure establishes that the left +evaluation exists; strictness establishes its value at masked rows. + +== Reverse (by contrapositive): the law implies strictness and closure + +$(M_j)$ explicitly includes $(C_j)$. To obtain $(S_j)$, use its contrapositive: suppose a successful +input $b$ has a null in argument $j$ at row $i$, but gives a non-null result $v$ there. This is exactly +the negation of $(S_j)$, and we will derive a contradiction with $(M_j)$. + +Choose a mask $m$ that is false only at $i$, and write +$b'_j = mask(b_j, m)$. At row $i$, $b_j[i]$ was already #N; at every other row, $m$ is true. Thus +$b'_j eq.triple b_j$. Replacing $b_j$ by $b'_j$ changes no logical input value, including at the one +row we care about. + +Now apply $(M_j)$ to the successful input $b$. Its left-hand side is precisely the evaluation with +$b'_j = mask(b_j, m)$, and it guarantees that evaluation succeeds. At row $i$, the common left-hand +side has these two incompatible values: + +$ f(b_1, ..., mask(b_j, m), ..., b_k)[i] + = f(b_1, ..., b'_j, ..., b_k)[i] + = f(b_1, ..., b_j, ..., b_k)[i] = v != #N. $ + +But $(M_j)$ also says + +$ f(b_1, ..., mask(b_j, m), ..., b_k)[i] + = mask(f(b_1, ..., b_j, ..., b_k), m)[i] = #N. $ + +The first line uses the definition of $b'_j$, then row-locality and $b'_j eq.triple b_j$; the second is +$(M_j)$ and $m[i] = $ `false`. We do not use $(S_j)$ here --- it is the fact being proved. One successful +evaluation cannot be both $v$ and #N, so the assumed counterexample cannot exist. Therefore $(M_j)$ +implies $(S_j)$. $square.stroked$ + +#pagebreak() + +The closure premise is necessary. A binary function that succeeds on $(0, 1)$, errors on $(#N, 1)$, +and otherwise returns null whenever it does evaluate with a null first argument satisfies $(S_1)$ under +the partiality convention, but not $(M_1)$: masking the first input turns a successful evaluation into +an error. Defining strictness to require a *successful* null result on every null input is an equivalent +way to build this premise into $(S_j)$. + +#figure( + table( + columns: 4, + align: center, + table.header([input], [$f$], [after masking argument 1], [$f$ after masking]), + [$(0, 1)$], [0], [$(#N, 1)$], [*error*], + ), + caption: [The function is vacuously strict at $(#N, 1)$ because it does not return a non-null value; + nevertheless, it cannot satisfy the masked-evaluation law.], +) + += What the optimizer uses + +The dictionary rule has the shape + +#align(center)[ + #grid( + columns: 3, column-gutter: 1.2em, align: horizon, + node[`f(dict(codes, values), c)`], + text(size: 13pt)[$arrow.r.long$], + node(fill: rgb("#eafaf1"))[`dict(codes, f(values, c))`], + ) +] + +A null code masks only the dictionary argument while $c$ stays live, so this requires $(M_j)$ for that +argument, not a weaker law that masks all arguments together. Kleene `AND` illustrates the difference: +`false AND NULL` is `false`, so masking only its second argument is not equivariant. + +#table( + columns: 6, + align: center, + table.header( + [$a_1$], [$a_2$], [$m$], [mask $a_2$, then `AND`], [`AND`, then mask], [result], + ), + [`false`], [`true`], [`false`], [`false`], N, [not $(M_2)$], +) + +Value equivalence is not enough for this rewrite when $f$ is fallible. It evaluates *every* dictionary +value, including values with no live code; `div(100, 0)` can then error on the rewritten side although +the original never evaluated it. Thus the dictionary rule also needs its existing no-speculative-error +condition (normally `!is_fallible`). Mask closure addresses masked input rows; it does not make dead +dictionary values safe to evaluate. + += Independent obligations + +#table( + columns: (auto, 1fr, 1fr), + align: (left, left, left), + table.header([property], [statement], [what it enables]), + [strict + mask-closed], [null inputs produce null outputs and remain evaluable], + [validity push-down], + [representable], [the declared return dtype admits required nulls], + [advertising `is_strict`], + [total], [valid inputs never produce null], + [precomputing output validity], + [infallible], [no legal evaluation errors], + [speculative evaluation], + [dense-safe], [bytes behind nulls may be read safely], + [`NullHandling::Dense`], +) + +Representability is a type-level obligation: a strict `cast` with a pinned non-nullable return type +cannot represent the null its value semantics demand. Totality is different again. A strict `list_sum` +may return null for a valid empty list, so strictness only gives + +$ valid(f(a_1, ..., a_k)) subset.eq valid(a_1) " and " dots " and " valid(a_k). $ + +Equality, and hence a precomputed output-validity mask, additionally needs totality. + +`RowFn` supplies strictness structurally. Its `Filter` path evaluates only rows valid in every input and +scatters nulls back; its `Dense` path evaluates all rows then applies that combined validity. The latter +still needs `InputElement::DENSE_SAFE`, because an invalid string view may hold unsafe bytes. That is an +operational property of an element representation, not a consequence of strictness. From ea58061b5d2fe663642144a3f03ea1202e07baed Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 13:35:38 -0400 Subject: [PATCH 07/12] Fix RowFn documentation checks Signed-off-by: Connor Tsui --- SCALAR_FN_HANDOFF.md | 17 ++++++------- STRICT_SCALAR_FN_RESEARCH.md | 25 ++++++++++---------- vortex-array/src/scalar_fn/row/sink.rs | 2 +- vortex-array/src/scalar_fn/row/tests/sink.rs | 2 +- vortex-spatial/benches/null_strategies.rs | 6 ++--- 5 files changed, 27 insertions(+), 25 deletions(-) diff --git a/SCALAR_FN_HANDOFF.md b/SCALAR_FN_HANDOFF.md index f99f4cb0eaa..bff6cf82347 100644 --- a/SCALAR_FN_HANDOFF.md +++ b/SCALAR_FN_HANDOFF.md @@ -34,10 +34,10 @@ cargo bench -p vortex-tensor --bench l2_norm cargo bench -p vortex-tensor --bench inner_product cargo bench -p vortex-tensor --bench cosine_similarity cargo bench -p vortex-tensor --bench normalized -cargo bench -p vortex-geo --bench binary_predicates -cargo bench -p vortex-geo --bench distance -cargo bench -p vortex-geo --bench envelope -cargo bench -p vortex-geo --bench predicate_bbox +cargo bench -p vortex-spatial --bench binary_predicates +cargo bench -p vortex-spatial --bench distance +cargo bench -p vortex-spatial --bench envelope +cargo bench -p vortex-spatial --bench predicate_bbox ``` Run each revision at least twice in alternating order. If the host allows it, pin the process to one @@ -45,13 +45,13 @@ core. Record the timer and CPU configuration, and compare both fastest and media benchmark binaries and public names are now shared with `develop`, so the comparison no longer needs a frozen benchmark-local implementation as its primary control. -Also run the branch-only `vortex-geo` `null_strategies` diagnostic. It forces branch-and-skip and +Also run the branch-only `vortex-spatial` `null_strategies` diagnostic. It forces branch-and-skip and filter-and-scatter for the measured nullable geometry shapes. Confirm that automatic selection uses the faster mechanism for one costly decode at 50% survivors and for two costly decodes at about 81% survivors. This is the x86 runtime check that remains after the LLVM comparison. ```bash -cargo bench -p vortex-geo --bench null_strategies +cargo bench -p vortex-spatial --bench null_strategies ``` If a stable benchmark regresses, inspect optimized LLVM IR again. The previous cross-compile proves @@ -262,7 +262,8 @@ x86. ## Current implementation and checks -The implementation includes production users in `vortex-array`, `vortex-tensor`, and `vortex-geo`. +The implementation includes production users in `vortex-array`, `vortex-tensor`, and +`vortex-spatial`. `NumericBinary` is an unregistered `RowFn` used only for primitive arithmetic execution. Decimal arithmetic keeps its existing path. The stable public-path benchmark baseline landed as #9136. @@ -270,7 +271,7 @@ The checks recorded for the final API state are: - 67 focused RowFn tests; - 179 `vortex-tensor` tests; -- 230 `vortex-geo` tests; +- 230 `vortex-spatial` tests; - `cargo +nightly fmt --all`; and - full workspace clippy, with `PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1` because the host `/usr/bin/python3` is 3.9 while the workspace requires the Python 3.11 stable ABI. diff --git a/STRICT_SCALAR_FN_RESEARCH.md b/STRICT_SCALAR_FN_RESEARCH.md index 1c873108345..e0d5c8ad1d5 100644 --- a/STRICT_SCALAR_FN_RESEARCH.md +++ b/STRICT_SCALAR_FN_RESEARCH.md @@ -200,9 +200,9 @@ is one `apply` per row. Three whole classes of strict function are therefore ine `RowFn` at any cost: - **Output dtype outside the element set.** `ext_storage`'s output is an extension array's storage - dtype, so `vortex.geo.box` is a struct and `vortex.uuid` is a `FixedSizeList(u8,16)`. `vortex-geo`'s - zone-map pruning calls `ext_storage` on a `geo.box` statistic, and a row-function port breaks it at - plan time. + dtype, so `vortex.st.box` is a struct and `vortex.uuid` is a `FixedSizeList(u8,16)`. + Zone-map pruning in `vortex-spatial` calls `ext_storage` on an `st.box` statistic, and a + row-function port breaks it at plan time. - **Variadic arity.** `merge` and `select` take an unbounded number of children, while `RowFn` fixes `Arity::Exact(n <= 3)`. - **Sub-row-granular kernels.** `not` negates one 64-bit word at a time, so a row loop over `bool` is @@ -721,12 +721,12 @@ stopping them. | blocker | count | members | | --- | --- | --- | | **Not strict.** `RowFn` implies strict, so these cannot reach it at all. | 12 | `between`, `case_when`, `cast`, `dynamic`, `fill_null`, `is_null`, `is_not_null`, `list_contains`, `pack`, `stat`, `row_size`, `zip` | -| **The answer already exists in bulk.** Zero-copy child projection, a metadata field, or a vectorized slice kernel. A row loop would be strictly slower. | 12 | `not`, `list_length`, `binary`, `mask`, `ext_storage`, `get_item`, `select`, `merge`, `variant_get`, `geo.envelope`, `json_to_variant`, `row_encode` | +| **The answer already exists in bulk.** Zero-copy child projection, a metadata field, or a vectorized slice kernel. A row loop would be strictly slower. | 12 | `not`, `list_length`, `binary`, `mask`, `ext_storage`, `get_item`, `select`, `merge`, `variant_get`, `spatial.envelope`, `json_to_variant`, `row_encode` | | **No element rows to read.** Zero-arity, or a type-erasure adapter. | 5 | `literal`, `root`, `row_idx`, `row_count`, `ForeignScalarFnVTable` | -| **Output side.** Nullable output, or an output dtype that depends on runtime data. | 2 | `list_sum`, `geo.envelope` | +| **Output side.** Nullable output, or an output dtype that depends on runtime data. | 2 | `list_sum`, `spatial.envelope` | | **Value-dependent per-batch setup.** | 1 | `like` | -`geo.envelope` is the one function counted twice: its output is a struct-of-four extension type *and* +`spatial.envelope` is the one function counted twice: its output is a struct-of-four extension type *and* its fast paths hand back existing child arrays untouched. `binary` deserves a note, since on strictness alone it looks portable: only its Kleene `And`/`Or` are @@ -997,7 +997,7 @@ does generically for every function, and computed its output nullability by hand This is the justification to carry onto a clean branch. It also bounds the claim: a `vortex-tensor` local helper owning the same invariant would remove the same `unsafe`, so what earns the *generic* -placement in `vortex-array` is that `vortex-geo`'s three predicates and `byte_length` use it too, +placement in `vortex-array` is that `vortex-spatial`'s three predicates and `byte_length` use it too, over three different element types. Two downstream crates plus core is the second-caller test met, not anticipated. @@ -1241,7 +1241,7 @@ Method: a survey of every non-strict `ScalarFnVTable` impl in the workspace plus `is_strict` and `validity()`, and a working prototype (worktree branch `proto/null-strategies`, 2,034-line diff, not for merging) that implemented both a branch-and-skip execution strategy and a `Nullable` input element, benchmarked on 65,536-row batches at null densities from 0% to 90%. -All 435 vortex-array scalar_fn tests and 223 vortex-geo tests pass with the prototype strategy both +All 435 vortex-array scalar_fn tests and 223 vortex-spatial tests pass with the prototype strategy both off and on, including new hostile tests (out-of-bounds views and poison divisors behind null rows) proving the kernel never runs behind a null. @@ -1327,7 +1327,7 @@ selectable per batch from `Mask::true_count`, with Filter kept for the sparse ta implemented on this branch** (see "Adaptive null strategy, as shipped" below); the rest of this section records the prototype evidence that justified it. The prototype also validated the two supporting pieces: a null-tolerant `decode_branch` on `InputElement` -(defaulting to plain decode, correct for bulk canonicalizations) and `OutputElement::garbage()` +(defaulting to plain decode, correct for bulk canonicalization) and `OutputElement::garbage()` for pre-fill. Production caveats recorded in the prototype report: `reduce_encoded` is not consulted on the branch path, sinks fall back to Filter, the toggle must become per-execution and cost-based, and geo's null-tolerant decode covered Point and Polygon only, still paying a @@ -1410,7 +1410,7 @@ arity. Replace it with per-element/arity inputs or a small estimated-cost compar moves onto production branches. Batch size remains an unmeasured input to that model. Verified independently of the implementing agent: 3,441 tests pass across vortex-array and -vortex-geo (17 new: hostile out-of-bounds views behind nulls, a fallible kernel with poison +vortex-spatial (17 new: hostile out-of-bounds views behind nulls, a fallible kernel with poison divisors behind nulls in one and both operands, conjoined-mask honoring, constant operands, real errors still propagating, geo filter-versus-branch agreement, the unsupported-geometry fallback, and six selection-rule cases), vortex-tensor's 164 pass unchanged, clippy `--all-targets @@ -1767,12 +1767,13 @@ Fetch the latest `origin/develop`, record both exact revisions, and compare the Run `binary_ops` and `like` from `vortex-array`. Run `l2_norm`, `inner_product`, `cosine_similarity`, and `normalized` from `vortex-tensor`. Run `binary_predicates`, `distance`, -`envelope`, and `predicate_bbox` from `vortex-geo`. Use at least two alternating runs per revision. +`envelope`, and `predicate_bbox` from `vortex-spatial`. Use at least two alternating runs per +revision. If the host permits it, pin one core. Report both fastest and median values with the CPU, timer, and governor configuration. The stable production binaries are the cross-revision gate because they now exist on `develop`. -The branch-only `vortex-geo` `null_strategies` benchmark remains the forced-policy diagnostic. Run +The branch-only `vortex-spatial` `null_strategies` benchmark remains the forced-policy diagnostic. Run it on the same x86 host to verify both measured selector decisions: one costly decode at 50% survivors must select the faster mechanism, and two costly decodes at about 81% survivors must do the same. Inspect optimized LLVM IR again for any stable regression before changing the API or the diff --git a/vortex-array/src/scalar_fn/row/sink.rs b/vortex-array/src/scalar_fn/row/sink.rs index 50d07d1b7ff..886f9f5f9e1 100644 --- a/vortex-array/src/scalar_fn/row/sink.rs +++ b/vortex-array/src/scalar_fn/row/sink.rs @@ -23,7 +23,7 @@ use crate::scalar_fn::OutputElement; /// Relaxing the row closure to `FnMut` instead was measured at 8 to 11%, because a captured `&mut` /// inhibits vectorization of the loop. /// - **[`sink_dtype`](Self::sink_dtype) sees the input dtypes**, unlike -/// [`OutputElement::element_dtype`](crate::scalar_fn::OutputElement::element_dtype), which takes +/// [`OutputElement::element_dtype`], which takes /// none. That is the whole reason a runtime-shaped output fits here: the width comes out of the /// arguments. /// diff --git a/vortex-array/src/scalar_fn/row/tests/sink.rs b/vortex-array/src/scalar_fn/row/tests/sink.rs index 95b5e2df05f..45b91ed191d 100644 --- a/vortex-array/src/scalar_fn/row/tests/sink.rs +++ b/vortex-array/src/scalar_fn/row/tests/sink.rs @@ -355,7 +355,7 @@ fn a_failing_row_is_never_reached_behind_a_null() -> VortexResult<()> { } /// The sink names its output dtype from the input, so a wrong input dtype is rejected at plan -/// time rather than producing a mis-typed column. +/// time rather than producing a mistyped column. #[test] fn the_sink_dtype_validates_its_input() { let dtype = DType::Primitive(PType::F64, Nullability::NonNullable); diff --git a/vortex-spatial/benches/null_strategies.rs b/vortex-spatial/benches/null_strategies.rs index 1ef453d645b..89f43f97429 100644 --- a/vortex-spatial/benches/null_strategies.rs +++ b/vortex-spatial/benches/null_strategies.rs @@ -32,13 +32,13 @@ use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::NullStrategy; use vortex_array::scalar_fn::execute_row_fn_with_strategy; use vortex_array::validity::Validity; +use vortex_session::VortexSession; use vortex_spatial::scalar_fn::contains::SpatialContains; -use vortex_spatial::test_harness::geo_session; use vortex_spatial::test_harness::point_column; use vortex_spatial::test_harness::polygon_column; -use vortex_session::VortexSession; +use vortex_spatial::test_harness::spatial_session; -static SESSION: LazyLock = LazyLock::new(geo_session); +static SESSION: LazyLock = LazyLock::new(spatial_session); fn main() { LazyLock::force(&SESSION); From 66b874290ecd5884c88b6ced8438fd2ef7a8df65 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 14:49:20 -0400 Subject: [PATCH 08/12] Record RowFn regression and landing plan Signed-off-by: Connor Tsui --- AGENTS.md | 6 ++++ NUMERIC_ROWFN_PLAN.md | 28 ++++++++++++---- SCALAR_FN_HANDOFF.md | 63 ++++++++++++++++++++++-------------- STRICT_SCALAR_FN_RESEARCH.md | 39 ++++++++++++++++++++++ 4 files changed, 104 insertions(+), 32 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a0e4c6558cd..30d67e18cb5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,12 @@ Notes: Avoid hidden-cost per-element accessors in hot loops, follow the performance guidance in `STYLE.md`, and benchmark changes to hot paths. +Treat branchless indexing as a code-generation hypothesis, not as an optimization by itself. A +runtime expression such as `index & mask` can make a slice index non-affine, retain bounds checks, +and block vectorization. Inspect generated code before replacing a loop-invariant enum match because +LLVM can unswitch the match into specialized loops. For binary kernels, benchmark varying x varying, +varying x constant, constant x varying, and nullable constant shapes separately. + ## Tests - Strongly consider `rstest` cases when parameterizing repetitive test logic. diff --git a/NUMERIC_ROWFN_PLAN.md b/NUMERIC_ROWFN_PLAN.md index 2f69b708b01..5a304e88102 100644 --- a/NUMERIC_ROWFN_PLAN.md +++ b/NUMERIC_ROWFN_PLAN.md @@ -185,19 +185,33 @@ Re-measure the port against `develop` once that lands, because the comparison ab ### Measured dead ends -Recorded so they are not retried. All of these are in vortex-data/vortex#9130 as well. +Recorded so they are not retried. The entries that predate the broadcast-index-mask experiment are +also in vortex-data/vortex#9130. - Bounds-check elimination in the row loop is not available. Narrowing the varying view to the row count buys nothing, and `get_unchecked` is not uniformly a win: about 10% on `mul_u16` and `mul_u32`, and 22% slower on `mul_u8`. - A per-argument row source that keeps the `Varying` view when another argument is batch-constant is 4x slower than the `ArgColumn` branch it replaces, which already vectorizes. -- A batch-constant operand therefore still demotes its neighbours off the slice path. Closing that - needs the row loop monomorphized over which arguments are constant. Revisit when `Compare` moves - onto `RowFn`, since `col < literal` is exactly this shape. - -`mul_i32_constant` is the one regression that survives, and it is inside this host's drift. Let -CodSpeed settle whether it is real. +- Pairing each varying view with a runtime index mask also fails. Commit `ad24700088` used + `index & usize::MAX` for varying inputs and `index & 0` for constants. On x86 with + `RUSTFLAGS="-C target-feature=+avx2"`, the constant numeric cases became approximately 4x to 7x + slower in wall time while non-constant cases stayed at parity. CodSpeed reported smaller but + consistent regressions: `add_i64_constant` 31.34%, `sub_i64_constant` 32.38%, and + `mul_i32_constant` 46.24%. +- The disassembly explains the mask result. Each `index & mask` remained behind a slice bounds + check, so LLVM saw a non-affine index and emitted a scalar loop. The old enum match was + loop-invariant, and LLVM unswitched it into constant-pattern loops with affine varying + accesses. Removing a branch removed information that the vectorizer needed. + +Commit `ad24700088` was removed from `ct/row-fn` history. The clean head after the rewrite is +`ea58061b5d`. A compile-time varying x constant or constant x varying specialization remains a +possible design, but it is not work for the first PR. Implement it only after the clean branch has +a stable mixed-constant regression against the current merge base. + +The earlier `mul_i32_constant` result was within Apple host drift and predates the mask experiment. +It does not establish parity against current `develop`. Rerun the clean candidate and current merge +base on x86 before deleting the hand-written kernels in a mergeable PR. ### What this implies for #9129 and #9130 diff --git a/SCALAR_FN_HANDOFF.md b/SCALAR_FN_HANDOFF.md index bff6cf82347..8b5d6a6d053 100644 --- a/SCALAR_FN_HANDOFF.md +++ b/SCALAR_FN_HANDOFF.md @@ -14,18 +14,37 @@ The public design lives in these tracking issues, which now match the implementa - [#9129, Define the `RowFn` API](https://github.com/vortex-data/vortex/issues/9129) - [#9130, Execute `RowFn` over Vortex arrays](https://github.com/vortex-data/vortex/issues/9130) -The branch is `ct/row-fn`. It is publicly linked from #9128, so do -not rewrite or delete its history. Commit `4becc863ae` contains the final API simplification. Push -only when explicitly requested. +The branch is `ct/row-fn`, and draft PR #9255 remains the integration and research branch. Its +history was rewritten at `ea58061b5d` to remove the regressing broadcast-index-mask experiment. +Do not use the draft PR as the first mergeable change. Cut the first PR from the latest +`origin/develop`, and keep this branch as the source for later tensor and spatial ports. Push or +rewrite either branch only when explicitly requested. -## Next action: rerun the benchmarks on x86 +## Next action: cut the vortex-array PR -The next session will run on an x86 machine. Rerun the performance comparison there before treating -the implementation as complete. Do not reuse the Apple timings as the final runtime result. +The first mergeable PR must stay within `vortex-array` and contain: -The production benchmark baseline from #9136 is on `develop` at `9a482c0230`. Fetch the latest -`origin/develop`, record the exact baseline and candidate commits, and run the same public benchmark -binaries at both revisions: +1. the `RowFn` API, lifting, executor, and focused behavioral tests. +2. the primitive `NumericBinary` port as its production consumer. +3. only the executor and numeric benchmarks needed to support its performance claim. + +Do not include the tensor or spatial ports, these branch-only working notes, the unrelated `like` +benchmark additions, or the fixed-size-list test. `NumericBinary` is the only `RowFn` consumer in +`vortex-array` on this branch. It already exercises varying and constant inputs, all-constant +folding, null constants, nullable execution, deferred overflow evidence, and the valid-row retry. +Do not add another consumer only to make the PR appear broader. + +The numeric commit deletes the now-unused `vortex-compute::lane_kernels::map_into` helper. Leave +that helper in place for a strictly `vortex-array`-only PR, and remove it in a separate cleanup. + +The first PR must establish parity against the latest `origin/develop`, not the integration +branch's old merge base. Run the public `binary_ops` benchmark on x86 with identical build flags at +both revisions. Cover varying x varying, varying x constant, constant x varying, and nullable plus +constant inputs. Run each revision at least twice in alternating order. Record the exact commits, +CPU, timer, pinning, fastest values, and medians. Inspect optimized LLVM IR or assembly for every +stable regression before changing the row API. + +The production benchmark commands across the staged work are: ```bash cargo bench -p vortex-array --bench binary_ops @@ -40,23 +59,17 @@ cargo bench -p vortex-spatial --bench envelope cargo bench -p vortex-spatial --bench predicate_bbox ``` -Run each revision at least twice in alternating order. If the host allows it, pin the process to one -core. Record the timer and CPU configuration, and compare both fastest and median values. The -benchmark binaries and public names are now shared with `develop`, so the comparison no longer -needs a frozen benchmark-local implementation as its primary control. - -Also run the branch-only `vortex-spatial` `null_strategies` diagnostic. It forces branch-and-skip and -filter-and-scatter for the measured nullable geometry shapes. Confirm that automatic selection uses -the faster mechanism for one costly decode at 50% survivors and for two costly decodes at about 81% -survivors. This is the x86 runtime check that remains after the LLVM comparison. +For the spatial PR, also run the branch-only `vortex-spatial` `null_strategies` diagnostic. It +forces branch-and-skip and filter-and-scatter for the measured nullable geometry shapes. Confirm +that automatic selection uses the faster mechanism for one costly decode at 50% survivors and for +two costly decodes at about 81% survivors. ```bash cargo bench -p vortex-spatial --bench null_strategies ``` -If a stable benchmark regresses, inspect optimized LLVM IR again. The previous cross-compile proves -that the API cleanup preserved the x86_64-v3 loop shape. The x86 run must confirm runtime effects -from the revised null selector and the target CPU's vectorizer and branch predictor. +The public benchmark names are shared with `develop`, so cross-revision comparisons do not need a +frozen benchmark-local implementation as their primary control. ## The API in one screen @@ -343,10 +356,10 @@ Deliberately **not** done: and masking a full-length result looks like a simplification and is not one: `normalized_readthrough_survives_null_rows` pins that a filtered input is no longer `Normalized`, so which arrays reach `reduce_encoded` is load-bearing and differs per strategy. -- **No PR split.** Recommended landing order, each step individually revertible and separately - benchmarkable: (1) API + lifting with dense/filter only; (2) branch-and-skip + adaptive selection - + its benchmarks; (3) `NumericBinary`; (4) tensor; (5) geo. The seam already supports this split - and no API changes between steps. +- **No mixed-constant specialization without a failing benchmark.** The broadcast-index-mask + experiment regressed numeric constants by 4x to 7x on x86 and was removed. Keep the current + executor for the first PR. Add a specialized varying x constant or constant x varying loop only + after the clean branch has a stable regression against the current merge base. ### Three API changes proposed, and why none of them landed diff --git a/STRICT_SCALAR_FN_RESEARCH.md b/STRICT_SCALAR_FN_RESEARCH.md index e0d5c8ad1d5..b5125ffc953 100644 --- a/STRICT_SCALAR_FN_RESEARCH.md +++ b/STRICT_SCALAR_FN_RESEARCH.md @@ -73,6 +73,45 @@ fastest and median observations rather than only these compact ranges. The historical measurements below remain because they explain design decisions and experiments made while building the prototype; they are not the current before/after performance record. +### Later broadcast-index-mask experiment + +Commit `ad24700088` tried to preserve a varying neighbor's decoded slice when another input was +constant. Every argument exposed `(Varying, mask)`, where the mask was `usize::MAX` for a varying +column and `0` for a one-row constant, and the fallback loop indexed each input with +`index & mask`. The all-varying loop was unchanged. + +The design regressed the numeric mixed-constant cases. The comparison used baseline `fed7038` and +candidate `edb3953`, with `RUSTFLAGS="-C target-feature=+avx2"` at both revisions: + +| benchmark | `fed7038` | `edb3953` | result | +| --- | ---: | ---: | ---: | +| `add_i64_constant` | 9.5-9.7 us | 37-71 us | approximately 4x to 7x slower | +| `sub_i64_constant` | 9.4-9.7 us | 37-45 us | approximately 4x slower | +| `mul_i32_constant` | 10.7-11.1 us | 42-43 us | approximately 4x slower | +| `add_i64_nonnull` | 11.1 us | 11.2 us | parity | +| `mul_i32_nonnull` | 13.9 us | 13.4 us | parity | + +The x86 report did not record the CPU, timer, pinning, or fastest and median values separately, so +these wall-clock values diagnose the code-generation failure rather than satisfy the release gate. +CodSpeed reported the same direction at a smaller magnitude: `add_i64_constant` 31.34%, +`sub_i64_constant` 32.38%, and `mul_i32_constant` 46.24% slower. + +The generated assembly kept two bounds checks per row and performed scalar loads. The runtime mask +made each varying index non-affine, so LLVM could not prove it in bounds or vectorize the loop. The +previous `ArgColumnKind` match was loop-invariant, which allowed LLVM to unswitch the numeric loop +into constant-pattern variants. The experiment optimized the branch count and discarded the +information that enabled vectorization. + +The commit was removed from `ct/row-fn` history. The clean integration head is `ea58061b5d`. Two +unpinned Divan runs on an Apple M4 Max, with 41 ns timer precision, restored the constant medians to +8.71-8.73 us for `add_i64`, 8.79-9.00 us for `sub_i64`, and 5.71-5.75 us for `mul_i32`. These +values prove that the mask regression is gone. They are not an x86 comparison against current +`develop`. + +Do not reintroduce a runtime mask or another runtime-shaped per-argument source. A later +mixed-constant optimization must monomorphize the loop over the constant pattern and must first be +justified by a stable benchmark against the current merge base. + --- ## The design in one screen From 02beb2d4e55452338e448da68d05b7db4baa4059 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 19:34:47 -0400 Subject: [PATCH 09/12] Port owned RowFn numeric execution Signed-off-by: "Connor Tsui" --- .../scalar_fn/fns/binary/numeric/primitive.rs | 10 +- .../src/scalar_fn/fns/binary/numeric/row.rs | 218 +++--------------- vortex-array/src/scalar_fn/row/element/mod.rs | 8 +- .../src/scalar_fn/row/element/tuple.rs | 32 ++- vortex-array/src/scalar_fn/row/execute.rs | 82 +++++++ vortex-array/src/scalar_fn/row/lift.rs | 39 ++-- vortex-array/src/scalar_fn/row/mod.rs | 9 +- vortex-array/src/scalar_fn/row/row_fn.rs | 30 +++ .../scalar_fn/row/tests/null_strategies.rs | 2 +- vortex-array/src/scalar_fn/row/vtable.rs | 144 +++++++++--- vortex-compute/src/lane_kernels/map_into.rs | 64 +++++ 11 files changed, 394 insertions(+), 244 deletions(-) diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs index 6de544eaaa2..7a6c1de4641 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs @@ -8,9 +8,10 @@ //! Keeping them apart is what lets [`row`](super::row) write a value for every row and reduce the //! evidence without a branch, so the loop vectorizes. +use std::ops::BitOrAssign; + use crate::dtype::NativePType; use crate::dtype::half::f16; -use crate::scalar_fn::SinkResult; /// Checked addition, failing on integer overflow. pub(super) struct CheckedAdd; @@ -33,9 +34,9 @@ pub(super) struct CheckedDiv; /// never compares, so the multiply stays a widening vector multiply and the reduction stays a /// vector OR. **The width must not exceed the element's**, or the reduction becomes the loop's /// bottleneck instead of the arithmetic. -pub(super) trait Failure: SinkResult + Copy + Default {} +pub(super) trait Failure: Copy + Default + PartialEq + BitOrAssign {} -impl + Copy + Default> Failure for T {} +impl Failure for T {} /// One arithmetic operator at one width, as a value and its failure evidence. /// @@ -309,7 +310,6 @@ impl_checked_float!(f16, f32, f64); #[cfg(test)] mod tests { use super::CheckedArithmetic; - use crate::scalar_fn::SinkResult; /// Values whose pairwise products are worth probing: the saturating boundaries, the sign-change /// pivots, and a spread of magnitudes that straddles the 64-bit split. @@ -336,7 +336,7 @@ mod tests { /// hold each against `checked_mul`, whose `None` is the definition of overflow. #[track_caller] fn assert_agrees_with_checked_mul(lhs: T, rhs: T, reference: Option) { - let failed = ::occurred(lhs.mul_failure(rhs)); + let failed = lhs.mul_failure(rhs) != ::default(); assert_eq!(failed, reference.is_none(), "{lhs:?} * {rhs:?}"); } diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs index aab36bf196b..cbbfa9e22da 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs @@ -3,27 +3,15 @@ //! The primitive arithmetic operators as a [`RowFn`]. //! -//! [`Binary`] keeps its ID, its options serialization, and its strictness, fallibility and validity -//! contracts, and delegates only the _execution_ of `Add`, `Sub`, `Mul` and `Div` over primitive -//! columns to [`NumericBinary`]. Delegation rather than conversion is what makes the port possible -//! at all: `Binary` also covers Kleene `And`/`Or`, which are not strict, and the six comparisons, -//! which are infallible, so no single [`RowFn`] can stand in for the whole function. +//! [`Binary`] keeps its ID, options serialization, and semantic contracts. It delegates only the +//! execution of primitive `Add`, `Sub`, `Mul`, and `Div` to [`NumericBinary`]. The helper is not +//! registered and appears in no serialized expression. //! -//! [`NumericBinary`] is not registered and appears in no serialized expression. It is reached only -//! through the [`ScalarFnVTable::execute`] that the blanket [`RowFn`] implementation provides, so -//! it needs no rewrite rule, no ID in the registry, and no wire format of its own. -//! -//! Everything the previous hand-written implementation did around the arithmetic itself now comes -//! from the lifting: input decoding, the constant operand collapse, the all-constant fold, the -//! null-constant short circuit, output allocation, nullability widening, and masking. What is left -//! here is the per-type checked operation and the sink that carries its overflow bit. +//! Shared lifting owns decoding, constant handling, output allocation, nullability, validity, and +//! nullable retry. The declaration below contains only type dispatch and the per-row operation. //! //! [`Binary`]: crate::scalar_fn::fns::binary::Binary -use std::marker::PhantomData; -use std::mem::MaybeUninit; - -use vortex_buffer::BufferMut; use vortex_error::VortexResult; use vortex_error::vortex_err; use vortex_session::registry::CachedId; @@ -35,27 +23,18 @@ use super::primitive::CheckedPrimitiveOp; use super::primitive::CheckedSub; use crate::ArrayRef; use crate::ExecutionCtx; -use crate::IntoArray; -use crate::arrays::PrimitiveArray; use crate::dtype::DType; use crate::dtype::NativePType; -use crate::dtype::Nullability; use crate::dtype::PType; use crate::match_each_native_ptype; use crate::scalar::NumericOperator; -use crate::scalar_fn::DeferredError; -use crate::scalar_fn::OutputSink; use crate::scalar_fn::RowFn; use crate::scalar_fn::RowVisitor; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnVTable; use crate::scalar_fn::VecExecutionArgs; -use crate::validity::Validity; /// Execute a numeric operation between two primitive-typed arrays. -/// -/// The caller has already established that both operands are primitive, of the same type, and of -/// the same length. pub(super) fn execute_numeric_primitive( lhs: &ArrayRef, rhs: &ArrayRef, @@ -67,10 +46,7 @@ pub(super) fn execute_numeric_primitive( ScalarFnVTable::execute(&NumericBinary, &op, &args, ctx) } -/// The four arithmetic operators of [`Binary`] over primitive columns, as one row function per -/// operator and width. -/// -/// [`Binary`]: crate::scalar_fn::fns::binary::Binary +/// The primitive arithmetic operators as a row function. #[derive(Clone)] struct NumericBinary; @@ -79,9 +55,8 @@ impl RowFn for NumericBinary { const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; - /// Only the integer widths can overflow, and only integer division can divide by zero, but - /// fallibility is declared without input dtypes. The float widths are therefore covered by the - /// same `true`, which costs them nothing: a deferred error keeps the batch on the dense path. + // Fallibility is declared before dispatch knows the primitive width. The float widths inherit + // this conservative declaration at no execution cost. const FALLIBLE: bool = true; fn id(&self) -> ScalarFnId { @@ -89,29 +64,28 @@ impl RowFn for NumericBinary { *ID } - fn dispatch( + fn dispatch( &self, op: &Self::Options, args: &[DType], - visitor: V, - ) -> VortexResult { + visitor: Visitor, + ) -> VortexResult { let ptype = operand_ptype(args)?; - match_each_native_ptype!(ptype, |T| { + match_each_native_ptype!(ptype, |Primitive| { match op { - NumericOperator::Add => visit_checked::(visitor), - NumericOperator::Sub => visit_checked::(visitor), - NumericOperator::Mul => visit_checked::(visitor), - NumericOperator::Div => visit_checked::(visitor), + NumericOperator::Add => visit_checked::(visitor), + NumericOperator::Sub => visit_checked::(visitor), + NumericOperator::Mul => visit_checked::(visitor), + NumericOperator::Div => visit_checked::(visitor), } }) } } -/// The width both operands are read at. +/// Return the primitive width selected by the left operand. /// -/// Only the left operand is inspected. `(T, T)` validates each argument against the chosen width, -/// so a right operand of a different type is rejected by the visit rather than here. +/// The visited `(Primitive, Primitive)` tuple validates both operands against this width. fn operand_ptype(args: &[DType]) -> VortexResult { let lhs = args .first() @@ -120,150 +94,22 @@ fn operand_ptype(args: &[DType]) -> VortexResult { PType::try_from(lhs) } -/// Visit at two `T` columns, applying `Op` per row into the sink that defers its overflow bit. -/// -/// The const block enforces, at monomorphization time, the width rule stated on -/// [`Failure`](super::primitive::Failure): evidence wider than the element would make the -/// OR-reduction rather than the arithmetic decide how many rows fit in a vector. -fn visit_checked(visitor: V) -> VortexResult +/// Visit two primitive columns and defer one OR-reducible failure word per row. +fn visit_checked(visitor: Visitor) -> VortexResult where - T: NativePType, - Op: CheckedPrimitiveOp, - V: RowVisitor, + Primitive: NativePType, + Operator: CheckedPrimitiveOp, + Visitor: RowVisitor, { - const { - assert!( - size_of::() <= size_of::(), - "failure evidence must be no wider than the value, or it bounds the vector width" - ) - }; - - visitor.visit_prepared_into::<(T, T), CheckedSink, _, _>( + visitor.visit_prepared_deferred::<(Primitive, Primitive), Primitive, _, Operator::Failure>( |_| (), - |&(), (lhs, rhs), output| output.write(lhs, rhs), - ) -} - -/// The output column of one checked arithmetic batch, reporting failure once after the row loop. -/// -/// Deferring the failure is what keeps a fallible kernel on the dense path: every row writes a -/// value unconditionally and OR-reduces its failure evidence, so the loop holds no branch and no -/// `Result` discriminant. The lifting retries a nullable batch over only its valid rows if that -/// reduction is non-zero, which is what makes an overflow behind a null row invisible. -/// -/// The reduction lives in the sink rather than in the row closure's return type so that its width -/// is [`Op::Failure`](CheckedPrimitiveOp::Failure), the operator's choice, rather than one bit. That -/// is what lets unsigned multiplication report its discarded high half instead of a comparison, and -/// so stay vectorized. -/// -/// **The storage is deliberately uninitialized, not zeroed.** Substituting `BufferMut::zeroed` to -/// make the sink safe was measured at **1.65 to 1.71x** the cost of allocate-and-fill, stable across -/// two runs and every batch size from 8 KiB to 2 MiB, because `alloc_zeroed` does not avoid the -/// write: below glibc's mmap threshold `calloc` recycles a dirty chunk and memsets it, and above it -/// the first touch of each fresh page faults instead. The row loop overwrites every slot regardless, -/// so that pass is pure duplicate work on the hottest kernel in the system. This is the case the -/// repository's "avoid `unsafe` unless it is necessary" rule leaves room for: the safe spelling -/// exists, and it costs a second pass over the output. -/// -/// Rows are written into uninitialized storage, so this sink cannot finish a batch whose rows were -/// not all visited, and leaves [`OutputSink::SUPPORTS_SKIPPED_ROWS`] at `false`. Nothing is lost: -/// `SUPPORTS_SKIPPED_ROWS` is what makes branch-and-skip unavailable, which is the guard that keeps -/// the uninitialized slots sound. Note this is _not_ implied by the dispatch policy alone: a -/// deferred result still reaches the executor's valid-only policy whenever its arguments are not -/// dense-safe, so the `false` here is load-bearing rather than a restatement. -struct CheckedSink> { - /// The result values, initialized one row at a time up to `row_count`. - values: BufferMut, - - /// The batch length, which is the capacity `values` was allocated with. - row_count: usize, - - /// The operation applied to every row, which names the error reported by - /// [`finish`](OutputSink::finish). - op: PhantomData, -} - -/// The uninitialized output slots of a [`CheckedSink`], borrowed once for the row loop. -struct CheckedRows<'a, T: NativePType, Op: CheckedPrimitiveOp> { - values: &'a mut [MaybeUninit], - op: PhantomData, -} - -/// One output slot of a [`CheckedSink`]. -struct CheckedRow<'a, T: NativePType, Op: CheckedPrimitiveOp> { - value: &'a mut MaybeUninit, - op: PhantomData, -} - -impl> CheckedRow<'_, T, Op> { - /// Apply `Op` to one row, writing its value and handing back its failure evidence. - /// - /// The value is written whether or not the operation failed, since a failing row is either - /// masked away as null or turned into a batch error before it can be read. The evidence is - /// returned rather than reduced here so the executor can keep the reduction in a register, and - /// it is `Op`'s own width so the row never has to compare. - fn write(self, lhs: T, rhs: T) -> Op::Failure { - let (value, failure) = Op::apply(lhs, rhs); - self.value.write(value); - - failure - } -} - -impl> OutputSink for CheckedSink { - const ERRORS_ARE_DEFERRED: bool = true; - - type Rows<'a> - = CheckedRows<'a, T, Op> - where - Self: 'a; - type Row<'a> - = CheckedRow<'a, T, Op> - where - Self: 'a; - - fn sink_dtype(_args: &[DType]) -> VortexResult { - Ok(DType::Primitive(T::PTYPE, Nullability::NonNullable)) - } - - fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { - Ok(Self { - values: BufferMut::with_capacity(rows), - row_count: rows, - op: PhantomData, - }) - } - - fn rows(&mut self) -> Self::Rows<'_> { - let row_count = self.row_count; - CheckedRows { - values: &mut self.values.spare_capacity_mut()[..row_count], - op: PhantomData, - } - } - - fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { - rows.values.len() == row_count - } - - fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { - CheckedRow { - value: &mut rows.values[index], - op: PhantomData, - } - } - - fn finish(mut self, error: DeferredError) -> VortexResult { - if error.occurred() { - return Err(vortex_err!(InvalidArgument: "{}", Op::ERROR)); - } - - // SAFETY: the sink reports `SUPPORTS_SKIPPED_ROWS = false`, so every path that reaches - // `finish` without an error has written all `row_count` slots: dense execution visits - // `0..row_count`, and the valid-row retry runs densely over a sink allocated for exactly - // the filtered rows. - unsafe { self.values.set_len(self.row_count) }; + |&(), (lhs, rhs)| Operator::apply(lhs, rhs), + |failure| { + if failure != ::default() { + return Err(vortex_err!(InvalidArgument: "{}", Operator::ERROR)); + } - Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) - } + Ok(()) + }, + ) } diff --git a/vortex-array/src/scalar_fn/row/element/mod.rs b/vortex-array/src/scalar_fn/row/element/mod.rs index fc88690b6b0..384d8bc3aa7 100644 --- a/vortex-array/src/scalar_fn/row/element/mod.rs +++ b/vortex-array/src/scalar_fn/row/element/mod.rs @@ -8,9 +8,10 @@ //! `vortex-tensor`'s `TensorRow` drills through an extension wrapper into its storage. //! //! The two directions are deliberately asymmetric. [`InputElement::Elem`] is a GAT, so an input row -//! can borrow out of the decoded column, while an [`OutputElement`] is one owned value written into -//! an [`ElementSink`](crate::scalar_fn::ElementSink). Runtime-shaped output uses a custom -//! [`OutputSink`](crate::scalar_fn::OutputSink) instead. +//! can borrow out of the decoded column, while an [`OutputElement`] is one owned value returned by +//! an owned row computation or written through +//! [`ElementSink`](crate::scalar_fn::ElementSink); runtime-shaped output uses a custom +//! [`OutputSink`](crate::scalar_fn::OutputSink). use vortex_error::VortexResult; @@ -29,6 +30,7 @@ mod primitive; mod tuple; pub use tuple::ElementTuple; +pub use tuple::IndexedElementTuple; pub(super) use tuple::batch_constant; /// An element type that can be read row-wise out of an input column. diff --git a/vortex-array/src/scalar_fn/row/element/tuple.rs b/vortex-array/src/scalar_fn/row/element/tuple.rs index a3c31cfeb2e..fad02639e1a 100644 --- a/vortex-array/src/scalar_fn/row/element/tuple.rs +++ b/vortex-array/src/scalar_fn/row/element/tuple.rs @@ -3,6 +3,8 @@ //! Argument lists built from [`InputElement`]s, and the per-argument decode behind them. +use vortex_compute::lane_kernels::IndexedSource; +use vortex_compute::lane_kernels::LaneZip; use vortex_error::VortexResult; use vortex_error::vortex_ensure_eq; @@ -13,6 +15,7 @@ use crate::arrays::Masked; use crate::arrays::extension::ExtensionArrayExt; use crate::arrays::masked::MaskedArraySlotsExt; use crate::dtype::DType; +use crate::dtype::NativePType; use crate::scalar_fn::ExecutionArgs; use crate::scalar_fn::InputElement; @@ -162,9 +165,8 @@ pub trait ElementTuple: 'static + private::Sealed { /// /// `Some` marks an argument whose operand is constant for the batch and carries the element /// every row reads; `None` marks one that varies by row. This is what - /// [`visit_prepared_into`](crate::scalar_fn::RowVisitor::visit_prepared_into) hands to its prepare - /// closure, so a kernel can hoist work that depends only on a constant argument out of the - /// row loop. + /// [`RowVisitor`](crate::scalar_fn::RowVisitor) hands to a visit's prepare closure, so a kernel + /// can hoist work that depends only on a constant argument out of the row loop. type ConstElems<'a>; /// The number of arguments. @@ -223,6 +225,22 @@ pub trait ElementTuple: 'static + private::Sealed { fn constants(columns: &Self::Columns) -> Self::ConstElems<'_>; } +/// An argument tuple that can expose independent indexed reads after one length validation. +/// +/// This trait is sealed through [`ElementTuple`]. Tuples without a natural indexed source continue +/// to use ordinary row access and output sinks. +pub trait IndexedElementTuple: ElementTuple { + /// The source shared execution uses for a dense all-varying loop. + /// + /// Its length must be the common varying-column length. For every valid index it must preserve + /// row order, return the same value as [`ElementTuple::get_varying`], and uphold the unchecked + /// read contract of [`IndexedSource`]. + type Source<'a>: IndexedSource>; + + /// Borrow a source from columns already validated to vary within the batch. + fn indexed_source<'a>(columns: &Self::VaryingColumns<'a>) -> Self::Source<'a>; +} + impl private::Sealed for () {} impl ElementTuple for () { @@ -368,3 +386,11 @@ element_tuple!(9; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8); element_tuple!(10; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9); element_tuple!(11; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10); element_tuple!(12; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10, L:11); + +impl IndexedElementTuple for (A, B) { + type Source<'a> = LaneZip<&'a [A], &'a [B]>; + + fn indexed_source<'a>(columns: &Self::VaryingColumns<'a>) -> Self::Source<'a> { + LaneZip::new(columns.0, columns.1) + } +} diff --git a/vortex-array/src/scalar_fn/row/execute.rs b/vortex-array/src/scalar_fn/row/execute.rs index 345bed991fe..a92e3e43998 100644 --- a/vortex-array/src/scalar_fn/row/execute.rs +++ b/vortex-array/src/scalar_fn/row/execute.rs @@ -6,6 +6,10 @@ //! These back the blanket impls in [`row_fn`](super::row_fn) and are deliberately not public: //! [`RowFn`](crate::scalar_fn::RowFn) is the abstraction, these are its internals. +use std::mem::needs_drop; +use std::ops::BitOrAssign; + +use vortex_compute::lane_kernels::IndexedSourceExt; use vortex_error::VortexError; use vortex_error::VortexResult; use vortex_error::vortex_bail; @@ -19,6 +23,8 @@ use crate::dtype::DType; use crate::scalar_fn::DeferredError; use crate::scalar_fn::ElementTuple; use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::IndexedElementTuple; +use crate::scalar_fn::OutputElement; use crate::scalar_fn::OutputSink; use crate::scalar_fn::SinkResult; @@ -42,6 +48,19 @@ impl RowExecution { } } +/// Validate the input dtypes of an owned-output row function and return its output dtype. +pub(super) fn validate_row_output( + args: &[DType], +) -> VortexResult { + A::validate(args)?; + let dtype = O::element_dtype(); + vortex_ensure!( + !dtype.is_nullable(), + "row output elements must declare a non-nullable dtype, got {dtype}", + ); + Ok(dtype) +} + /// Validate the input dtypes of a sink-writing row function and return the dtype its sink builds. /// /// The output dtype may be a function of the inputs. A sink can also own a batch-wide builder, such @@ -58,6 +77,69 @@ pub(super) fn validate_row_sink( Ok(dtype) } +/// Decode every input column once, then store owned row outputs and reduce deferred failures. +pub(super) fn execute_row_output_prepared( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, F), + finish_failure: impl FnOnce(F) -> VortexResult<()>, +) -> VortexResult +where + A: IndexedElementTuple, + O: OutputElement, + F: Copy + Default + BitOrAssign, +{ + const { + assert!( + !needs_drop::(), + "owned deferred outputs must not require drop glue" + ) + }; + + let row_count = args.row_count(); + let mut values = Vec::::with_capacity(row_count); + let columns = A::decode(args, ctx)?; + let state = prepare(A::constants(&columns)); + let failed; + + { + let output = &mut values.spare_capacity_mut()[..row_count]; + + if let Some(varying) = A::varying(&columns) { + vortex_ensure!( + A::varying_len_matches(&varying, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + failed = + A::indexed_source(&varying).map_checked_into(output, |elems| apply(&state, elems)); + } else { + vortex_ensure!( + A::decoded_lens_match(&columns, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + let mut accumulated = F::default(); + for index in 0..row_count { + let (value, failure) = apply(&state, A::get(&columns, index)); + output[index].write(value); + accumulated |= failure; + } + failed = accumulated; + } + } + + // SAFETY: normal completion of either loop initializes every slot in `0..row_count` exactly + // once, and `values` was allocated with at least `row_count` capacity. + unsafe { values.set_len(row_count) }; + + match finish_failure(failed) { + Ok(()) => Ok(RowExecution::Output(O::build(values))), + Err(error) => Ok(RowExecution::DeferredError(error)), + } +} + /// Decode every input column once, allocate the sink once, then write one row at a time. /// /// The sink lives here rather than in the closure, so `apply` stays [`Fn`] and the loop keeps the diff --git a/vortex-array/src/scalar_fn/row/lift.rs b/vortex-array/src/scalar_fn/row/lift.rs index a842ef5deef..0a44b066826 100644 --- a/vortex-array/src/scalar_fn/row/lift.rs +++ b/vortex-array/src/scalar_fn/row/lift.rs @@ -76,9 +76,9 @@ impl ExecutionArgs for BorrowedExecutionArgs<'_> { /// The arguments handed to one kernel invocation. /// -/// `arrays` may be filtered or sliced, while `dtypes` and `sink_dtype` always describe the original -/// planned batch. Keeping them together prevents an execution path from accidentally pairing an -/// input view with unrelated planning metadata. +/// `arrays` may be filtered or sliced, while `dtypes` and `output_dtype` always describe the +/// original planned batch. Keeping them together prevents an execution path from accidentally +/// pairing an input view with unrelated planning metadata. #[derive(Clone, Copy)] pub(super) struct KernelArgs<'a> { /// The executor-facing view, including the row count for this invocation. @@ -90,14 +90,14 @@ pub(super) struct KernelArgs<'a> { /// The original input dtypes used to select the row implementation. pub(super) dtypes: &'a [DType], - /// The non-nullable dtype allocated by the selected output sink. - pub(super) sink_dtype: &'a DType, + /// The non-nullable dtype built by the selected output capability. + pub(super) output_dtype: &'a DType, } /// The execution policy and output dtype selected by a planning visit. pub(super) struct BatchPlan { - /// The non-nullable dtype built by the selected sink. - pub(super) sink_dtype: DType, + /// The non-nullable dtype built by the selected output capability. + pub(super) output_dtype: DType, /// How this concrete dispatch executes nullable rows. pub(super) policy: RowPolicy, @@ -118,6 +118,17 @@ pub(super) enum RowPolicy { } impl RowPolicy { + /// The policy for an owned output carrying batch-deferred failure evidence. + pub(super) const fn for_deferred_output() -> Self { + if A::DENSE_SAFE && !A::DECODE_FALLIBLE { + Self::DenseWithRetry + } else { + Self::ValidOnly { + filtered_decode_cost: A::FILTERED_DECODE_COST, + } + } + } + /// The policy one concrete dispatch executes nullable rows under. /// /// Note what is deliberately **not** read here: [`OutputSink::SUPPORTS_SKIPPED_ROWS`]. Hoisting @@ -130,7 +141,7 @@ impl RowPolicy { /// answer differently from its row loop, that is a wrong answer rather than a slow one. /// /// [`OutputSink::SUPPORTS_SKIPPED_ROWS`]: crate::scalar_fn::OutputSink::SUPPORTS_SKIPPED_ROWS - pub(super) const fn for_dispatch() -> Self { + pub(super) const fn for_sink() -> Self { if A::DENSE_SAFE && !A::DECODE_FALLIBLE && !R::FALLIBLE { if R::DEFERRED { Self::DenseWithRetry @@ -178,8 +189,8 @@ pub(super) struct Batch<'a> { /// against. Already widened to nullable if any input is nullable. result_dtype: DType, - /// The non-nullable dtype the dispatched sink builds, computed once while planning. - sink_dtype: DType, + /// The non-nullable dtype the dispatched output capability builds, computed while planning. + output_dtype: DType, /// How the concrete dispatch executes nullable rows. policy: RowPolicy, @@ -203,9 +214,9 @@ impl<'a> Batch<'a> { let arg_dtypes: SmallVec<[DType; 4]> = inputs.iter().map(|input| input.dtype().clone()).collect(); let plan = plan(&arg_dtypes)?; - let nullability = plan.sink_dtype.nullability() + let nullability = plan.output_dtype.nullability() | Nullability::from(arg_dtypes.iter().any(DType::is_nullable)); - let result_dtype = plan.sink_dtype.with_nullability(nullability); + let result_dtype = plan.output_dtype.with_nullability(nullability); let mut validity = Validity::NonNullable; for input in &inputs { @@ -219,7 +230,7 @@ impl<'a> Batch<'a> { arg_dtypes, validity, result_dtype, - sink_dtype: plan.sink_dtype, + output_dtype: plan.output_dtype, policy: plan.policy, }) } @@ -507,7 +518,7 @@ impl<'a> Batch<'a> { execution, arrays, dtypes: &self.arg_dtypes, - sink_dtype: &self.sink_dtype, + output_dtype: &self.output_dtype, } } diff --git a/vortex-array/src/scalar_fn/row/mod.rs b/vortex-array/src/scalar_fn/row/mod.rs index f62d8bb8d49..d9799fff678 100644 --- a/vortex-array/src/scalar_fn/row/mod.rs +++ b/vortex-array/src/scalar_fn/row/mod.rs @@ -19,9 +19,11 @@ //! [`RowFn`] does not say how a row is _stored_, which is the element's job: `vortex-tensor` adds a //! `TensorRow` [`InputElement`] and writes ordinary kernels over it. //! -//! Output always goes through [`RowVisitor::visit_prepared_into`]. [`ElementSink`] covers one owned -//! [`OutputElement`] per row; custom [`OutputSink`] implementations cover runtime-shaped rows. The -//! prepare closure sees every batch-constant input and returns shared state for the row loop. Pass +//! Output has two capabilities. [`RowVisitor::visit_prepared_deferred`] returns one independent +//! [`OutputElement`] and failure word per row, letting shared execution own the stores and choose an +//! indexed dense source. [`RowVisitor::visit_prepared_into`] writes through an [`OutputSink`] for +//! runtime-shaped rows, shared builders, skip-capable output, and values requiring drop glue. Both +//! prepare closures see every batch-constant input and return shared state for the row loop. Pass //! `|_| ()` when there is nothing to prepare. //! //! A kernel that can safely write a provisional value uses [`DeferredError`] instead of returning @@ -43,6 +45,7 @@ mod element; pub use element::ElementTuple; +pub use element::IndexedElementTuple; pub use element::InputElement; pub use element::OutputElement; #[cfg(any(test, feature = "_test-harness"))] diff --git a/vortex-array/src/scalar_fn/row/row_fn.rs b/vortex-array/src/scalar_fn/row/row_fn.rs index f2a63636b1d..f790d21d8a0 100644 --- a/vortex-array/src/scalar_fn/row/row_fn.rs +++ b/vortex-array/src/scalar_fn/row/row_fn.rs @@ -6,6 +6,7 @@ use std::fmt::Debug; use std::fmt::Display; use std::hash::Hash; +use std::ops::BitOrAssign; use vortex_error::VortexResult; use vortex_error::vortex_bail; @@ -15,6 +16,8 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::dtype::DType; use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::IndexedElementTuple; +use crate::scalar_fn::OutputElement; use crate::scalar_fn::OutputSink; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::SinkResult; @@ -116,6 +119,33 @@ pub trait RowVisitor: private::Sealed { /// What this visit produces. type Out; + /// Visit at indexed argument tuple `A`, returning one independently owned output and one + /// deferred failure word per row. + /// + /// The executor allocates and writes the output column. It reads through + /// [`IndexedElementTuple`] when every argument varies; batches containing a constant use + /// ordinary row access selected once outside the loop. `F::default()` **must** mean success, + /// including for empty execution, and `|=` must combine the evidence from independent rows. + /// `finish_failure` runs once after the loop: it must return `Ok(())` for successful evidence + /// and may report only the operation's row error for failed evidence. That error is deferred, + /// so nullable lifting may retry over only valid rows. + /// + /// `A` **must** have the arity declared by [`RowFn::ARG_NAMES`]. This method requires + /// [`RowFn::FALLIBLE`] to be `true`, and `O` must be no narrower than `F` so failure reduction + /// does not constrain vector width. `O` must not require drop glue. Use + /// [`visit_prepared_into`](Self::visit_prepared_into) for non-indexed tuples, runtime-shaped + /// output, shared builders, and output that requires drop. + fn visit_prepared_deferred( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, F), + finish_failure: impl FnOnce(F) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + F: 'static + Copy + Default + BitOrAssign; + /// Visit at argument tuple `A`, preparing shared state once and writing every output row into /// sink `S`. /// diff --git a/vortex-array/src/scalar_fn/row/tests/null_strategies.rs b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs index d8a7fe18816..487cb4786f6 100644 --- a/vortex-array/src/scalar_fn/row/tests/null_strategies.rs +++ b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs @@ -493,7 +493,7 @@ mod selection { #[test] fn planning_adds_decode_cost_across_arguments() { assert_eq!( - RowPolicy::for_dispatch::<(TrackedI64<1>, TrackedI64<1>), ()>(), + RowPolicy::for_sink::<(TrackedI64<1>, TrackedI64<1>), ()>(), RowPolicy::ValidOnly { filtered_decode_cost: 2 } diff --git a/vortex-array/src/scalar_fn/row/vtable.rs b/vortex-array/src/scalar_fn/row/vtable.rs index cff4831aa8f..26c83e9c11b 100644 --- a/vortex-array/src/scalar_fn/row/vtable.rs +++ b/vortex-array/src/scalar_fn/row/vtable.rs @@ -4,6 +4,8 @@ //! Blanket scalar-function implementation and execution visitors for row functions. use std::marker::PhantomData; +use std::mem::needs_drop; +use std::ops::BitOrAssign; use vortex_error::VortexResult; #[cfg(any(test, feature = "_test-harness"))] @@ -24,8 +26,10 @@ use crate::scalar_fn::Arity; use crate::scalar_fn::ChildName; use crate::scalar_fn::ElementTuple; use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::IndexedElementTuple; #[cfg(any(test, feature = "_test-harness"))] use crate::scalar_fn::NullStrategy; +use crate::scalar_fn::OutputElement; use crate::scalar_fn::OutputSink; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnVTable; @@ -33,8 +37,10 @@ use crate::scalar_fn::SinkResult; #[cfg(any(test, feature = "_test-harness"))] use crate::scalar_fn::VecExecutionArgs; use crate::scalar_fn::row::execute::RowExecution; +use crate::scalar_fn::row::execute::execute_row_output_prepared; use crate::scalar_fn::row::execute::execute_row_sink_branch; use crate::scalar_fn::row::execute::execute_row_sink_prepared; +use crate::scalar_fn::row::execute::validate_row_output; use crate::scalar_fn::row::execute::validate_row_sink; use crate::scalar_fn::row::lift::Batch; use crate::scalar_fn::row::lift::BatchPlan; @@ -42,11 +48,8 @@ use crate::scalar_fn::row::lift::KernelArgs; use crate::scalar_fn::row::lift::RowPolicy; use crate::scalar_fn::row::lift::reconcile_return; -/// Compile-time check that a dispatched `(A, S, R)` agrees with `F`'s public metadata. Evaluated by -/// monomorphizing -/// [`visit_prepared_into`](RowVisitor::visit_prepared_into), so even a dispatch arm that never runs -/// is checked. -const fn assert_dispatch_agrees() { +/// Compile-time checks shared by both output capabilities. +const fn assert_input_dispatch_agrees() { assert!( A::ARITY == F::ARG_NAMES.len(), "dispatch visited a tuple whose arity differs from RowFn::ARG_NAMES", @@ -57,6 +60,11 @@ const fn assert_dispatch_agrees() { + assert_input_dispatch_agrees::(); assert!( !R::FALLIBLE || F::FALLIBLE, "dispatch returned an error without declaring RowFn::FALLIBLE", @@ -71,7 +79,30 @@ const fn assert_dispatch_agrees() +where + F: RowFn, + A: IndexedElementTuple, + O: OutputElement, + Failure: Copy + Default + BitOrAssign, +{ + assert_input_dispatch_agrees::(); + assert!( + F::FALLIBLE, + "dispatch deferred an error without declaring RowFn::FALLIBLE", + ); + assert!( + size_of::() <= size_of::(), + "failure evidence must be no wider than the value, or it bounds the vector width", + ); + assert!( + !needs_drop::(), + "owned deferred outputs must not require drop glue", + ); +} + +/// The plan-time visit: validate the dtypes and derive execution from the output capability and row /// closure selected by dispatch. struct PlanRows<'a, F> { args: &'a [DType], @@ -85,16 +116,35 @@ impl private::Sealed for PlanRows<'_, F> {} impl RowVisitor for PlanRows<'_, F> { type Out = BatchPlan; + fn visit_prepared_deferred( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + _finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + + Ok(BatchPlan { + output_dtype: validate_row_output::(self.args)?, + policy: RowPolicy::for_deferred_output::(), + }) + } + fn visit_prepared_into( self, _prepare: impl FnOnce(A::ConstElems<'_>) -> P, _apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, ) -> VortexResult { - const { assert_dispatch_agrees::() }; + const { assert_sink_dispatch_agrees::() }; Ok(BatchPlan { - sink_dtype: validate_row_sink::(self.args)?, - policy: RowPolicy::for_dispatch::(), + output_dtype: validate_row_sink::(self.args)?, + policy: RowPolicy::for_sink::(), }) } } @@ -103,8 +153,8 @@ impl RowVisitor for PlanRows<'_, F> { struct ExecuteRows<'a, 'b, F> { args: &'a dyn ExecutionArgs, - /// The sink dtype computed by the planning visit. - sink_dtype: &'a DType, + /// The output dtype computed by the planning visit. + output_dtype: &'a DType, ctx: &'b mut ExecutionCtx, @@ -117,15 +167,36 @@ impl private::Sealed for ExecuteRows<'_, '_, F> {} impl RowVisitor for ExecuteRows<'_, '_, F> { type Out = RowExecution; + fn visit_prepared_deferred( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + execute_row_output_prepared::( + self.args, + self.ctx, + prepare, + apply, + finish_failure, + ) + } + fn visit_prepared_into( self, prepare: impl FnOnce(A::ConstElems<'_>) -> P, apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, ) -> VortexResult { - const { assert_dispatch_agrees::() }; + const { assert_sink_dispatch_agrees::() }; execute_row_sink_prepared::( self.args, - self.sink_dtype, + self.output_dtype, self.ctx, prepare, apply, @@ -136,13 +207,13 @@ impl RowVisitor for ExecuteRows<'_, '_, F> { /// The run-time visit for the branch-and-skip null strategy: compute only the conjoined-valid /// rows over unfiltered columns. /// -/// `Ok(None)` means the visit cannot take that strategy because the sink cannot skip rows or an -/// argument has no null-tolerant decode, and the lifting falls back to the filter strategy. +/// `Ok(None)` means the visit requires filtering, a sink cannot skip rows, or an argument has no +/// null-tolerant decode. The lifting then falls back to the filter strategy. struct ExecuteRowsBranch<'a, 'b, F> { args: &'a dyn ExecutionArgs, - /// The sink dtype computed by the planning visit. - sink_dtype: &'a DType, + /// The output dtype computed by the planning visit. + output_dtype: &'a DType, /// The conjoined validity, materialized by the lifting and guaranteed mixed. valid: &'a Mask, @@ -158,15 +229,30 @@ impl private::Sealed for ExecuteRowsBranch<'_, '_, F> {} impl RowVisitor for ExecuteRowsBranch<'_, '_, F> { type Out = Option; + fn visit_prepared_deferred( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + _finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult> + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + Ok(None) + } + fn visit_prepared_into( self, prepare: impl FnOnce(A::ConstElems<'_>) -> P, apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, ) -> VortexResult> { - const { assert_dispatch_agrees::() }; + const { assert_sink_dispatch_agrees::() }; execute_row_sink_branch::( self.args, - self.sink_dtype, + self.output_dtype, self.valid, self.ctx, prepare, @@ -192,7 +278,7 @@ fn execute_rows( args.dtypes, ExecuteRows:: { args: args.execution, - sink_dtype: args.sink_dtype, + output_dtype: args.output_dtype, ctx, row_fn: PhantomData, }, @@ -221,7 +307,7 @@ fn execute_rows_branch( args.dtypes, ExecuteRowsBranch:: { args: args.execution, - sink_dtype: args.sink_dtype, + output_dtype: args.output_dtype, valid, ctx, row_fn: PhantomData, @@ -229,7 +315,7 @@ fn execute_rows_branch( ) } -/// The batch facts for `row_fn` over `args`, derived from its dispatched elements and sink. +/// The batch facts for `row_fn` over `args`, derived from its selected output capability. fn lift_batch<'a, F: RowFn>( row_fn: &F, options: &F::Options, @@ -307,9 +393,9 @@ impl ScalarFnVTable for F { }, )?; - let nullability = - plan.sink_dtype.nullability() | Nullability::from(args.iter().any(DType::is_nullable)); - Ok(plan.sink_dtype.with_nullability(nullability)) + let nullability = plan.output_dtype.nullability() + | Nullability::from(args.iter().any(DType::is_nullable)); + Ok(plan.output_dtype.with_nullability(nullability)) } fn execute( @@ -328,7 +414,7 @@ impl ScalarFnVTable for F { execution: args, arrays: &[], dtypes: &[], - sink_dtype: &result_dtype, + output_dtype: &result_dtype, }, ctx, )? @@ -343,9 +429,9 @@ impl ScalarFnVTable for F { ) } - /// Output sinks build an all-valid column, so a row kernel cannot turn a wholly non-null row into - /// a null and the output validity is exactly the conjunction of the inputs'. Letting a sink - /// produce nulls would invalidate this. + /// Row output capabilities build an all-valid column, so a kernel cannot turn a wholly non-null + /// row into a null and the output validity is exactly the conjunction of the inputs'. Letting an + /// output capability produce nulls would invalidate this. fn validity( &self, _options: &Self::Options, diff --git a/vortex-compute/src/lane_kernels/map_into.rs b/vortex-compute/src/lane_kernels/map_into.rs index 258913e9fc7..9ede2df69e5 100644 --- a/vortex-compute/src/lane_kernels/map_into.rs +++ b/vortex-compute/src/lane_kernels/map_into.rs @@ -5,6 +5,7 @@ //! caller-provided `&mut [MaybeUninit]`. use std::mem::MaybeUninit; +use std::ops::BitOrAssign; use vortex_buffer::BitBuffer; @@ -156,6 +157,50 @@ pub trait IndexedSourceExt: IndexedSource + Sized { } } + /// Write each mapped value and OR-reduce independent failure evidence across the batch. + /// + /// The failure stays local to this method so the optimizer can keep it in a register. The + /// caller receives only whether the batch failed and can attribute errors on a cold retry. + /// **`Failure` must be no wider than `Output`**, or its reduction can limit vector width. + /// + /// # Panics + /// + /// Panics if `out.len() != self.len()`. + #[inline] + fn map_checked_into( + self, + out: &mut [MaybeUninit], + mut apply: Apply, + ) -> Failure + where + Failure: Copy + Default + BitOrAssign, + Apply: FnMut(Self::Item) -> (Output, Failure), + { + const { + assert!( + size_of::() <= size_of::(), + "failure evidence must be no wider than the value, or it bounds the vector width" + ) + }; + + let values = self; + let len = values.len(); + assert_eq!(out.len(), len, "out must have the same length as values"); + + let mut failed = Failure::default(); + for index in 0..len { + // SAFETY: `index < len` by the loop bound. + let value = unsafe { values.get_unchecked(index) }; + let (output, failure) = apply(value); + failed |= failure; + + // SAFETY: `index < len == out.len()`. + unsafe { out.get_unchecked_mut(index).write(output) }; + } + + failed + } + /// Apply the predicate `f(value)` lane-by-lane and bit-pack the results into /// `words`, LSB-first, 64 lanes per `u64`. /// @@ -546,6 +591,25 @@ mod tests { assert!(res.is_ok(), "null lane should bypass the range check"); } + #[test] + fn map_checked_into_writes_all_lanes_and_reduces_failure() { + let mut values: Vec = (0..130).collect(); + let mut output = vec![MaybeUninit::::uninit(); 130]; + let failed = values + .as_slice() + .map_checked_into(&mut output, |value| (value as u32, value > u32::MAX as u64)); + assert!(!failed); + assert_eq!(write_t(output), (0..130u32).collect::>()); + + values[77] = (u32::MAX as u64) + 1; + let mut output = vec![MaybeUninit::::uninit(); 130]; + let failed = values + .as_slice() + .map_checked_into(&mut output, |value| (value as u32, value > u32::MAX as u64)); + assert!(failed); + assert_eq!(write_t(output)[76], 76); + } + #[test] fn map_bits_into_packs_full_and_remainder_words() { let values: Vec = (0..130).collect(); From ecb3826cb43d7f5f3bf010180f1161a8a091fb9d Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 19:38:35 -0400 Subject: [PATCH 10/12] Record RowFn x86 vectorization research Signed-off-by: "Connor Tsui" --- NUMERIC_ROWFN_PLAN.md | 5 + SCALAR_FN_HANDOFF.md | 2 + research/rowfn-x86-2026-08-07/README.md | 239 +++++++++++++ .../benchmarks/land-base-1.md | 39 +++ .../benchmarks/land-base-2.md | 39 +++ .../benchmarks/land-candidate-1.md | 39 +++ .../benchmarks/land-candidate-2.md | 39 +++ .../benchmarks/land-final-1.md | 39 +++ .../benchmarks/land-final-2.md | 39 +++ .../benchmarks/stage0-base-1.md | 38 +++ .../benchmarks/stage0-base-2.md | 38 +++ .../benchmarks/stage0-candidate-1.md | 38 +++ .../benchmarks/stage0-candidate-2.md | 38 +++ .../benchmarks/stage1-base-1.md | 38 +++ .../benchmarks/stage1-base-2.md | 38 +++ .../benchmarks/stage1-candidate-1.md | 38 +++ .../benchmarks/stage1-candidate-2.md | 38 +++ .../benchmarks/stage1-owned-1.md | 38 +++ .../benchmarks/stage1-owned-2.md | 38 +++ .../benchmarks/stage2-base-1.md | 39 +++ .../benchmarks/stage2-base-2.md | 39 +++ .../benchmarks/stage2-candidate-1.md | 39 +++ .../benchmarks/stage2-candidate-2.md | 39 +++ .../benchmarks/stage2-indexed-1.md | 39 +++ .../benchmarks/stage2-indexed-2.md | 39 +++ .../codegen/base-codegen-summary.md | 139 ++++++++ .../codegen/candidate-i64-mul-dense-ll.md | 84 +++++ .../codegen/candidate-i64-mul-dense-s.md | 69 ++++ .../codegen/candidate-u64-mul-dense-ll.md | 137 ++++++++ .../codegen/candidate-u64-mul-dense-s.md | 70 ++++ .../codegen/final-i32-mul-constant-ll.md | 86 +++++ .../codegen/final-i32-mul-constant-s.md | 49 +++ .../codegen/final-i64-mul-dense-ll.md | 96 ++++++ .../codegen/final-i64-mul-dense-s.md | 34 ++ .../codegen/final-u64-mul-dense-ll.md | 322 ++++++++++++++++++ .../codegen/final-u64-mul-dense-s.md | 71 ++++ .../codegen/indexed-i64-mul-dense-ll.md | 96 ++++++ .../codegen/indexed-i64-mul-dense-s.md | 35 ++ .../codegen/indexed-u64-mul-dense-ll.md | 322 ++++++++++++++++++ .../codegen/indexed-u64-mul-dense-s.md | 71 ++++ .../codegen/owned-i64-mul-dense-ll.md | 84 +++++ .../codegen/owned-i64-mul-dense-s.md | 46 +++ .../codegen/owned-u64-mul-dense-ll.md | 121 +++++++ .../codegen/owned-u64-mul-dense-s.md | 72 ++++ 44 files changed, 3098 insertions(+) create mode 100644 research/rowfn-x86-2026-08-07/README.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md create mode 100644 research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md create mode 100644 research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md diff --git a/NUMERIC_ROWFN_PLAN.md b/NUMERIC_ROWFN_PLAN.md index 5a304e88102..6fb337afc04 100644 --- a/NUMERIC_ROWFN_PLAN.md +++ b/NUMERIC_ROWFN_PLAN.md @@ -3,6 +3,11 @@ # Plan: fit the numeric binary operators onto `RowFn` +> The later x86 follow-up found that the sink-only API regressed varying `i64`/`u64` multiply and +> added separate owned-output and stateful-sink capabilities. Its complete evidence is in +> [`research/rowfn-x86-2026-08-07/README.md`](research/rowfn-x86-2026-08-07/README.md). Treat that +> record as authoritative where it supersedes the pre-x86 conclusions below. + Working note, branch-only, like `SCALAR_FN_HANDOFF.md`. Written so this survives a conversation compaction: everything needed to start is here, and nothing below depends on chat history. diff --git a/SCALAR_FN_HANDOFF.md b/SCALAR_FN_HANDOFF.md index 8b5d6a6d053..10224704829 100644 --- a/SCALAR_FN_HANDOFF.md +++ b/SCALAR_FN_HANDOFF.md @@ -6,6 +6,8 @@ This is the concise source of truth for the branch. `STRICT_SCALAR_FN_RESEARCH.md` keeps the full design history, rejected alternatives, measurements, and generated-code evidence. `NUMERIC_ROWFN_PLAN.md` records the numeric-binary migration and its narrower performance boundary. +`research/rowfn-x86-2026-08-07/README.md` records the later x86 regression reproduction, the +owned-output and indexed-source experiments, raw benchmark logs, and exact production IR/assembly. All three are branch-only working notes for agents. They are not intended to land with the API. The public design lives in these tracking issues, which now match the implementation: diff --git a/research/rowfn-x86-2026-08-07/README.md b/research/rowfn-x86-2026-08-07/README.md new file mode 100644 index 00000000000..e1df97740ba --- /dev/null +++ b/research/rowfn-x86-2026-08-07/README.md @@ -0,0 +1,239 @@ + + + +# RowFn owned-output and x86 numeric research + +This is the durable record for the investigation that produced the owned-output RowFn path. The +result is not that RowFn is inherently difficult to optimize. The declaration must distinguish an +independent returned value from a stateful output sink, and dense primitive inputs must cross a +validated indexed-source boundary that shared execution can lower directly. + +The selected implementation restores `i64` and `u64` varying multiplication to within about 1% of +the actual merge-base throughput. It does so without a numeric array downcast, `reduce_encoded` +override, numeric-owned allocation, or numeric-specific null and constant policy. + +## Revisions and environment + +- Merge-base baseline: `19f771f2a426103aa7d1bf7153a258bb1bab1e19`. +- Untouched sink-only RowFn: `35098c72118f1b555a24bd2f9b58b0400fa46dc5`. +- Selected implementation: `1a0a055c752b54448c8e1d54af032fe43acf8517`. +- Selected diff fingerprint: + `928e7a0baa2895609d102c98d110c21fb7a12e079b04195b85903277c71537a2`. + +The research branch has older tensor and spatial RowFn users. The result was ported rather than +rebased so that history remains intact. Its port also backports `map_checked_into`, which already +exists at the mergeable branch's base. + +```text +AMD Ryzen 9 7950X +1 socket, 16 physical cores, 32 threads +benchmark logical CPU: 8; SMT sibling: 24 +Linux CTCachyDesktop 7.1.6-1-cachyos, x86_64 +rustc 1.91.0, LLVM 21.1.2 +cargo 1.91.0 +``` + +The CPU reports AVX2 and AVX-512F/DQ/BW/VL. Builds used the default repository target and bench +profile without LTO, `target-cpu=native`, profile changes, or forced inlining. The scaling governor +was `performance`. Timed executions were pinned to CPU 8 and never overlapped compilation. + +```bash +taskset -c 8 "$BENCH" --bench --sample-count 100 --max-time 0.5 --color never \ + mul_i8_nonnull mul_u8_nonnull mul_i16_nonnull mul_u16_nonnull \ + mul_i32_nonnull mul_u32_nonnull mul_i64_nonnull mul_u64_nonnull \ + add_i64_nonnull add_i64_constant sub_i64_constant \ + mul_i32_constant mul_i32_nullable div_i64_nonnull +``` + +Every file in [`benchmarks`](benchmarks) is unedited Divan output wrapped in Markdown. It includes +fastest, slowest, median, mean, samples, and iterations rather than only the selected medians. + +## Stage 0: reproduction + +Order: baseline, candidate, baseline, candidate. Values are median microseconds. + +| Benchmark | Baseline 1 / 2 | Candidate 1 / 2 | Candidate/baseline | +| --- | ---: | ---: | ---: | +| `add_i64_constant` | 8.449 / 8.399 | 9.269 / 9.290 | 1.097 / 1.106 | +| `add_i64_nonnull` | 9.205 / 9.149 | 9.455 / 9.490 | 1.027 / 1.037 | +| `div_i64_nonnull` | 44.850 / 44.800 | 45.090 / 45.160 | 1.005 / 1.008 | +| `mul_i8_nonnull` | 6.184 / 6.199 | 4.694 / 4.699 | 0.759 / 0.758 | +| `mul_i16_nonnull` | 4.099 / 4.119 | 4.269 / 4.269 | 1.041 / 1.036 | +| `mul_i32_constant` | 26.420 / 26.430 | 18.880 / 18.870 | 0.715 / 0.714 | +| `mul_i32_nonnull` | 26.410 / 26.420 | 28.390 / 28.350 | 1.075 / 1.073 | +| `mul_i32_nullable` | 27.350 / 27.400 | 29.180 / 29.150 | 1.067 / 1.064 | +| `mul_i64_nonnull` | 23.220 / 23.200 | 30.020 / 30.080 | **1.293 / 1.297** | +| `mul_u8_nonnull` | 3.319 / 3.329 | 3.539 / 3.529 | 1.066 / 1.060 | +| `mul_u16_nonnull` | 2.599 / 2.599 | 2.429 / 2.429 | 0.935 / 0.935 | +| `mul_u32_nonnull` | 6.939 / 6.949 | 7.069 / 7.059 | 1.019 / 1.016 | +| `mul_u64_nonnull` | 19.210 / 19.190 | 30.430 / 30.490 | **1.584 / 1.589** | +| `sub_i64_constant` | 8.255 / 8.239 | 9.099 / 9.099 | 1.102 / 1.104 | + +The x86 regression reproduced. Raw runs are the four `stage0-*` files. + +## Stage 1: owned output without indexed input + +The closure returned `(output, failure)`, shared execution owned the store, and failure remained a +loop-local OR. This removed the numeric checked sink and materially improved 64-bit cases, but did +not solve the general problem. + +| Benchmark | Baseline 1 / 2 | Owned 1 / 2 | Owned/baseline | +| --- | ---: | ---: | ---: | +| `mul_i64_nonnull` | 23.20 / 23.26 | 25.65 / 25.59 | 1.106 / 1.100 | +| `mul_u64_nonnull` | 19.18 / 19.21 | 19.41 / 19.41 | 1.012 / 1.010 | +| `mul_i32_constant` | 26.43 / 26.44 | 32.36 / 32.38 | 1.224 / 1.225 | +| `mul_i32_nonnull` | 26.42 / 26.41 | 31.24 / 31.23 | 1.182 / 1.183 | +| `mul_i32_nullable` | 27.36 / 27.35 | 32.04 / 32.04 | 1.171 / 1.171 | + +The six `stage1-*` files contain the full matrix. This falsifies output ownership as a complete +explanation: it matters, but does not give LLVM the specialized kernel's input representation. + +## Stage 2: indexed dense input + +`IndexedElementTuple` lets a primitive pair expose `LaneZip<&[Left], &[Right]>` after shared +execution validates both varying lengths once. The generic owned executor calls +`map_checked_into`; numeric code still declares only row types, operation, failure, and error. + +| Benchmark | Baseline 1 / 2 | Indexed 1 / 2 | Candidate 1 / 2 | +| --- | ---: | ---: | ---: | +| `mul_i32_nonnull` | 26.39 / 26.41 | 26.58 / 26.60 | 28.34 / 28.36 | +| `mul_i32_nullable` | 27.37 / 27.38 | 27.41 / 27.43 | 29.20 / 29.17 | +| `mul_i64_nonnull` | 23.22 / 23.24 | 23.43 / 23.44 | 30.02 / 30.10 | +| `mul_u64_nonnull` | 19.22 / 19.21 | 19.41 / 19.42 | 30.41 / 30.43 | +| `div_i64_nonnull` | 44.84 / 44.87 | 45.07 / 45.03 | 45.07 / 45.12 | +| `mul_i32_constant` | 26.42 / 26.43 | 32.38 / 32.39 | 18.88 / 18.88 | + +The indexed source closed the varying and nullable gap. It did not affect mixed constants, which +exposed the next compiler-sensitive detail. + +## Store placement and the `Copy` ablation + +Moving the output store before the failure OR changed `mul_i32_constant` from about 32.38 to +18.68 microseconds. Final assembly records overflow `setb`, output store, then loop-carried OR. This +is compiler scheduling sensitivity, not a semantic difference. + +Adding the descriptive `Output: Copy` bound regressed that case to 29.88/29.86 microseconds. +Replacing it with compile-time `!needs_drop::()` returned it to 18.65/18.67. A generic store +helper did not repair the `Copy` case. The executor needs only the no-drop property for safe panic +cleanup; it never copies an output. The API therefore enforces the actual requirement without the +measured optimizer-visible bound. Re-test this workaround whenever LLVM changes. + +## Final results + +Order: baseline, final, candidate, repeated twice. Values are median microseconds. + +| Benchmark | Baseline 1 / 2 | Candidate 1 / 2 | Final 1 / 2 | Final/baseline | +| --- | ---: | ---: | ---: | ---: | +| `add_i64_constant` | 8.399 / 8.449 | 9.310 / 9.289 | 9.269 / 9.279 | 1.104 / 1.098 | +| `add_i64_nonnull` | 9.159 / 9.239 | 9.374 / 9.449 | 9.379 / 9.389 | 1.024 / 1.016 | +| `div_i64_nonnull` | 44.820 / 44.860 | 45.040 / 45.080 | 45.020 / 45.060 | 1.004 / 1.004 | +| `mul_i8_nonnull` | 6.209 / 6.199 | 4.719 / 4.699 | 6.389 / 6.409 | 1.029 / 1.034 | +| `mul_i16_nonnull` | 4.099 / 4.109 | 4.269 / 4.269 | 4.265 / 4.299 | 1.040 / 1.046 | +| `mul_i32_constant` | 26.440 / 26.440 | 18.890 / 18.840 | 18.690 / 18.700 | **0.707 / 0.707** | +| `mul_i32_nonnull` | 26.410 / 26.420 | 28.350 / 28.390 | 26.590 / 26.640 | 1.007 / 1.008 | +| `mul_i32_nullable` | 27.380 / 27.360 | 29.170 / 29.170 | 27.400 / 27.440 | 1.001 / 1.003 | +| `mul_i64_nonnull` | 23.200 / 23.350 | 30.010 / 30.050 | 23.460 / 23.430 | **1.011 / 1.003** | +| `mul_u8_nonnull` | 3.319 / 3.319 | 3.545 / 3.519 | 3.514 / 3.549 | 1.059 / 1.069 | +| `mul_u16_nonnull` | 2.609 / 2.609 | 2.429 / 2.429 | 2.789 / 2.810 | 1.069 / 1.077 | +| `mul_u32_nonnull` | 6.949 / 6.959 | 7.060 / 7.059 | 7.129 / 7.149 | 1.026 / 1.027 | +| `mul_u64_nonnull` | 19.180 / 19.210 | 30.370 / 30.400 | 19.370 / 19.380 | **1.010 / 1.009** | +| `sub_i64_constant` | 8.239 / 8.259 | 9.114 / 9.079 | 9.149 / 9.159 | 1.110 / 1.109 | + +The six `land-*` logs preserve every final run. Narrow widths avoid the rejected zipped-iterator +experiment's 3x to 9x losses. Constant add/sub retain the untouched candidate's roughly 10% gap; +constant multiplication is faster than merge base. Division stays at parity. + +## Generated code: confirmed evidence + +```bash +CARGO_TARGET_DIR="$TARGET" cargo rustc -p vortex-array --lib --profile bench -- \ + --emit=llvm-ir,asm -C codegen-units=1 -C remark=loop-vectorize +``` + +Full output was about 1.85 GiB IR plus 1.02 GiB assembly and was deleted after extracting exact +production monomorphs into [`codegen`](codegen). These are not fixture or benchmark control loops. + +Baseline, candidate, owned, and final signed `i64` use a scalar one-lane loop: one high/low `imulq`, +one store, `sarq`/`xorq` overflow evidence, register OR, and one backedge. Unsigned `u64` uses two +independent scalar `mulq` groups per backedge plus an odd remainder. Neither final loop has a hot +call, panic edge, bounds check, runtime alias check, or vector body. The second input length check is +an `llvm.assume`; loads and stores carry disjoint alias metadata; failure is a register `phi`. + +Therefore host SIMD did not hide a deficient loop. The default build did not enable optional native +AVX features, and LLVM selected the same essential scalar high-half strategy as merge base. See +[`base summary`](codegen/base-codegen-summary.md), +[`final i64 assembly`](codegen/final-i64-mul-dense-s.md), and +[`final u64 assembly`](codegen/final-u64-mul-dense-s.md). + +`-C remark=loop-vectorize` emitted no remark attributable to the exact dense production loop. The +constant fallback source line had successes for other monomorphs and duplicated cost-model misses, +but diagnostics lacked function identity. Exact IR proves the measured specialization is scalar; +it cannot assign those remarks to it. The merge-base focused remark rebuild was cancelled, so no +merge-base missed-vectorization reason is claimed. + +## Findings + +Confirmed: + +- Bounds checks are not the all-varying blocker; candidate dense multiply had no hot bounds edge. +- `SinkResult` already reduced to a register OR and did not impose a per-row `Result`. +- Output ownership materially helped but was insufficient alone. +- A typed indexed source restored stable parity for varying primitive tuples. +- Store-before-OR and omission of a `Copy` bound materially affect LLVM 21.1.2 constant codegen. +- The default x86 target prefers scalar high-half 64-bit multiply; SIMD is not the recovered speed. + +Still inference: + +- No single alias defect explains the original gap. Baseline and candidate had useful metadata too. +- The nearly identical dense inner loops do not explain all end-to-end timing. Surrounding control + flow, placement, and instruction-cache effects remain candidates. +- Unattributed source-line remarks do not prove a missed-vectorization reason for one monomorph. + +Rejected controls: checked unchecked access only partially helped and regressed some `u8` runs; +direct failure accumulation matched existing IR; safe zipped iterators caused 3x to 9x narrow +losses; a numeric `reduce_encoded` fast path recovered speed by duplicating shared policy; and a +primitive-binary visitor seam moved that specialization into generic execution. Earlier Apple work, +including the non-affine `index & mask` failure, remains in +[`NUMERIC_ROWFN_PLAN.md`](../../NUMERIC_ROWFN_PLAN.md). + +## Why both visitor methods exist + +`visit_prepared_deferred` represents an independent owned value and OR-reducible failure per row. +The executor allocates contiguous output, owns the store, and can use a typed indexed source. It is +intentionally limited to indexed inputs, fixed no-drop output, and a batch-deferred row error. + +`visit_prepared_into` represents stateful construction: shared buffers, runtime-shaped layouts, +multiple coordinated builders, skip-capable output, drop-requiring values, non-indexed tuples, and +ordinary immediate or deferred `SinkResult` forms. Encoding those through the owned method would +either hide a mutable builder reference inside a supposed value, allocate a temporary per row, +forbid legitimate output, or duplicate lifting. Encoding numeric output only through the sink loses +the fact that each value and store are independent. These are distinct capabilities. + +## Indexed source, specialization, and safety + +`InputElement` is open and many elements are not contiguous. Sealed `ElementTuple` is the safe +composition point for unchecked reads after one length validation. Stable Rust cannot overlap a +blanket fallback for every tuple with a more specific associated dense source without +specialization. Runtime erasure would obscure the source type LLVM needs. The indexed capability is +therefore explicit and opt-in; only the proven primitive pair implements it today. + +The executor reserves `row_count` slots and exposes exactly that many `MaybeUninit` values. It +validates varying lengths before `LaneZip`; `map_checked_into` validates output length. Either loop +writes every slot exactly once before `set_len`. On panic the vector length remains zero, and the +compile-time no-drop assertion makes abandoning initialized slots safe. Deferred errors are examined +only after initialization. Nullable lifting retries a deferred error over valid rows, so a failure +shaped value behind null cannot surface. + +## Open improvements + +- Investigate infallible owned output only with a measured caller; avoid a speculative result tree. +- Revisit constant add/sub only with exact production IR and a stable regression. +- Add indexed tuple/element families only for real consumers with a safe source. +- Re-run the store-order and `Copy` ablations after LLVM upgrades. +- Produce an upstream LLVM reproducer for those compiler sensitivities. +- Preserve assembly checks because throughput can hide compensating target-specific instructions. + +The selected branch passed focused checks, 87 numeric tests, 3,385 nextest tests with one skipped, +73 doctests with 13 ignored, nightly formatting, all-target/all-feature clippy, and `diff --check`. +One intermediate 1.85 GiB IR copy hit `ENOSPC`; exact-final codegen later completed. The requested +`ROWFN_FIRST_PR_PROMPT.md` was absent from the repository, fetched refs, home tree, and worktrees. diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md new file mode 100644 index 00000000000..2759e0fa9e5 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md @@ -0,0 +1,39 @@ + + + +# `land-base-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.059 µs │ 781.5 µs │ 8.399 µs │ 16.16 µs │ 100 │ 100 +│ 4.065 Gitem/s │ 41.92 Mitem/s │ 3.901 Gitem/s │ 2.027 Gitem/s │ │ +├─ add_i64_nonnull 9.079 µs │ 29.55 µs │ 9.159 µs │ 9.466 µs │ 100 │ 100 +│ 3.608 Gitem/s │ 1.108 Gitem/s │ 3.577 Gitem/s │ 3.461 Gitem/s │ │ +├─ div_i64_nonnull 44.73 µs │ 78.44 µs │ 44.82 µs │ 45.35 µs │ 100 │ 100 +│ 732.4 Mitem/s │ 417.6 Mitem/s │ 731 Mitem/s │ 722.5 Mitem/s │ │ +├─ mul_i8_nonnull 5.819 µs │ 72.73 µs │ 6.209 µs │ 6.939 µs │ 100 │ 100 +│ 5.63 Gitem/s │ 450.5 Mitem/s │ 5.276 Gitem/s │ 4.721 Gitem/s │ │ +├─ mul_i16_nonnull 4.039 µs │ 66.44 µs │ 4.099 µs │ 4.731 µs │ 100 │ 100 +│ 8.111 Gitem/s │ 493.1 Mitem/s │ 7.992 Gitem/s │ 6.926 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 56.27 µs │ 26.44 µs │ 26.97 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 582.2 Mitem/s │ 1.238 Gitem/s │ 1.214 Gitem/s │ │ +├─ mul_i32_nonnull 26.35 µs │ 38.26 µs │ 26.41 µs │ 26.64 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 856.4 Mitem/s │ 1.24 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.28 µs │ 340.1 µs │ 27.38 µs │ 30.62 µs │ 100 │ 100 +│ 1.2 Gitem/s │ 96.34 Mitem/s │ 1.196 Gitem/s │ 1.069 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 45.96 µs │ 23.2 µs │ 23.55 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 712.9 Mitem/s │ 1.411 Gitem/s │ 1.391 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 52.07 µs │ 3.319 µs │ 3.838 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 629.1 Mitem/s │ 9.87 Gitem/s │ 8.535 Gitem/s │ │ +├─ mul_u16_nonnull 2.539 µs │ 30.81 µs │ 2.609 µs │ 2.888 µs │ 100 │ 100 +│ 12.9 Gitem/s │ 1.063 Gitem/s │ 12.55 Gitem/s │ 11.34 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 26.44 µs │ 6.949 µs │ 7.183 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 1.238 Gitem/s │ 4.714 Gitem/s │ 4.561 Gitem/s │ │ +├─ mul_u64_nonnull 19.11 µs │ 41.4 µs │ 19.18 µs │ 19.53 µs │ 100 │ 100 +│ 1.713 Gitem/s │ 791.4 Mitem/s │ 1.707 Gitem/s │ 1.677 Gitem/s │ │ +╰─ sub_i64_constant 8.109 µs │ 40.38 µs │ 8.239 µs │ 8.604 µs │ 100 │ 100 + 4.04 Gitem/s │ 811.2 Mitem/s │ 3.976 Gitem/s │ 3.808 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md new file mode 100644 index 00000000000..ec40d57a6e2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md @@ -0,0 +1,39 @@ + + + +# `land-base-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.319 µs │ 39.44 µs │ 8.449 µs │ 8.813 µs │ 100 │ 100 +│ 3.938 Gitem/s │ 830.8 Mitem/s │ 3.877 Gitem/s │ 3.718 Gitem/s │ │ +├─ add_i64_nonnull 9.169 µs │ 12.56 µs │ 9.239 µs │ 9.304 µs │ 100 │ 100 +│ 3.573 Gitem/s │ 2.608 Gitem/s │ 3.546 Gitem/s │ 3.521 Gitem/s │ │ +├─ div_i64_nonnull 44.77 µs │ 50.37 µs │ 44.86 µs │ 45.15 µs │ 100 │ 100 +│ 731.7 Mitem/s │ 650.4 Mitem/s │ 730.2 Mitem/s │ 725.7 Mitem/s │ │ +├─ mul_i8_nonnull 5.829 µs │ 8.179 µs │ 6.199 µs │ 6.265 µs │ 100 │ 100 +│ 5.62 Gitem/s │ 4.005 Gitem/s │ 5.285 Gitem/s │ 5.23 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 8.029 µs │ 4.109 µs │ 4.15 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 4.08 Gitem/s │ 7.973 Gitem/s │ 7.895 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 35.43 µs │ 26.44 µs │ 26.77 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 924.6 Mitem/s │ 1.239 Gitem/s │ 1.223 Gitem/s │ │ +├─ mul_i32_nonnull 26.37 µs │ 35.11 µs │ 26.42 µs │ 26.84 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 933 Mitem/s │ 1.239 Gitem/s │ 1.22 Gitem/s │ │ +├─ mul_i32_nullable 27.26 µs │ 40.16 µs │ 27.36 µs │ 27.78 µs │ 100 │ 100 +│ 1.201 Gitem/s │ 815.9 Mitem/s │ 1.197 Gitem/s │ 1.179 Gitem/s │ │ +├─ mul_i64_nonnull 23.24 µs │ 32.01 µs │ 23.35 µs │ 23.63 µs │ 100 │ 100 +│ 1.409 Gitem/s │ 1.023 Gitem/s │ 1.402 Gitem/s │ 1.386 Gitem/s │ │ +├─ mul_u8_nonnull 3.26 µs │ 4.759 µs │ 3.319 µs │ 3.349 µs │ 100 │ 100 +│ 10.04 Gitem/s │ 6.884 Gitem/s │ 9.87 Gitem/s │ 9.783 Gitem/s │ │ +├─ mul_u16_nonnull 2.53 µs │ 7.289 µs │ 2.609 µs │ 2.664 µs │ 100 │ 100 +│ 12.94 Gitem/s │ 4.495 Gitem/s │ 12.55 Gitem/s │ 12.29 Gitem/s │ │ +├─ mul_u32_nonnull 6.889 µs │ 12.89 µs │ 6.959 µs │ 7.027 µs │ 100 │ 100 +│ 4.756 Gitem/s │ 2.54 Gitem/s │ 4.708 Gitem/s │ 4.663 Gitem/s │ │ +├─ mul_u64_nonnull 19.13 µs │ 22.83 µs │ 19.21 µs │ 19.29 µs │ 100 │ 100 +│ 1.712 Gitem/s │ 1.435 Gitem/s │ 1.704 Gitem/s │ 1.698 Gitem/s │ │ +╰─ sub_i64_constant 8.139 µs │ 11.06 µs │ 8.259 µs │ 8.297 µs │ 100 │ 100 + 4.025 Gitem/s │ 2.96 Gitem/s │ 3.967 Gitem/s │ 3.949 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md new file mode 100644 index 00000000000..8ec200b7df1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md @@ -0,0 +1,39 @@ + + + +# `land-candidate-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.989 µs │ 1.016 ms │ 9.31 µs │ 19.44 µs │ 100 │ 100 +│ 3.645 Gitem/s │ 32.25 Mitem/s │ 3.519 Gitem/s │ 1.684 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 13.06 µs │ 9.374 µs │ 9.443 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 2.508 Gitem/s │ 3.495 Gitem/s │ 3.469 Gitem/s │ │ +├─ div_i64_nonnull 44.97 µs │ 63.03 µs │ 45.04 µs │ 45.41 µs │ 100 │ 100 +│ 728.5 Mitem/s │ 519.7 Mitem/s │ 727.3 Mitem/s │ 721.5 Mitem/s │ │ +├─ mul_i8_nonnull 4.649 µs │ 60.73 µs │ 4.719 µs │ 5.455 µs │ 100 │ 100 +│ 7.047 Gitem/s │ 539.4 Mitem/s │ 6.942 Gitem/s │ 6.006 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 71.24 µs │ 4.269 µs │ 4.959 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 459.9 Mitem/s │ 7.674 Gitem/s │ 6.607 Gitem/s │ │ +├─ mul_i32_constant 18.79 µs │ 72.37 µs │ 18.89 µs │ 19.53 µs │ 100 │ 100 +│ 1.742 Gitem/s │ 452.7 Mitem/s │ 1.734 Gitem/s │ 1.677 Gitem/s │ │ +├─ mul_i32_nonnull 28.24 µs │ 33.43 µs │ 28.35 µs │ 28.48 µs │ 100 │ 100 +│ 1.159 Gitem/s │ 980.1 Mitem/s │ 1.155 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.01 µs │ 234.9 µs │ 29.17 µs │ 31.37 µs │ 100 │ 100 +│ 1.129 Gitem/s │ 139.4 Mitem/s │ 1.122 Gitem/s │ 1.044 Gitem/s │ │ +├─ mul_i64_nonnull 29.63 µs │ 52.67 µs │ 30.01 µs │ 30.5 µs │ 100 │ 100 +│ 1.105 Gitem/s │ 622 Mitem/s │ 1.091 Gitem/s │ 1.074 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 14.69 µs │ 3.545 µs │ 3.659 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 2.229 Gitem/s │ 9.242 Gitem/s │ 8.953 Gitem/s │ │ +├─ mul_u16_nonnull 2.369 µs │ 13.01 µs │ 2.429 µs │ 2.615 µs │ 100 │ 100 +│ 13.82 Gitem/s │ 2.516 Gitem/s │ 13.48 Gitem/s │ 12.52 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 18.45 µs │ 7.06 µs │ 7.181 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 1.775 Gitem/s │ 4.641 Gitem/s │ 4.562 Gitem/s │ │ +├─ mul_u64_nonnull 30.27 µs │ 43.99 µs │ 30.37 µs │ 30.65 µs │ 100 │ 100 +│ 1.082 Gitem/s │ 744.8 Mitem/s │ 1.078 Gitem/s │ 1.068 Gitem/s │ │ +╰─ sub_i64_constant 8.959 µs │ 31.53 µs │ 9.114 µs │ 9.385 µs │ 100 │ 100 + 3.657 Gitem/s │ 1.038 Gitem/s │ 3.595 Gitem/s │ 3.491 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md new file mode 100644 index 00000000000..84f969b8646 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md @@ -0,0 +1,39 @@ + + + +# `land-candidate-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.129 µs │ 48.92 µs │ 9.289 µs │ 9.744 µs │ 100 │ 100 +│ 3.589 Gitem/s │ 669.8 Mitem/s │ 3.527 Gitem/s │ 3.362 Gitem/s │ │ +├─ add_i64_nonnull 9.349 µs │ 10.43 µs │ 9.449 µs │ 9.46 µs │ 100 │ 100 +│ 3.504 Gitem/s │ 3.141 Gitem/s │ 3.467 Gitem/s │ 3.463 Gitem/s │ │ +├─ div_i64_nonnull 44.99 µs │ 50.41 µs │ 45.08 µs │ 45.29 µs │ 100 │ 100 +│ 728.1 Mitem/s │ 650 Mitem/s │ 726.8 Mitem/s │ 723.5 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 7.969 µs │ 4.699 µs │ 4.761 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 4.111 Gitem/s │ 6.972 Gitem/s │ 6.881 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 7.669 µs │ 4.269 µs │ 4.308 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 4.272 Gitem/s │ 7.674 Gitem/s │ 7.605 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.77 µs │ 18.84 µs │ 18.95 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.439 Gitem/s │ 1.738 Gitem/s │ 1.728 Gitem/s │ │ +├─ mul_i32_nonnull 28.27 µs │ 45.57 µs │ 28.39 µs │ 28.65 µs │ 100 │ 100 +│ 1.158 Gitem/s │ 718.9 Mitem/s │ 1.154 Gitem/s │ 1.143 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 43.06 µs │ 29.17 µs │ 29.41 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 760.8 Mitem/s │ 1.122 Gitem/s │ 1.113 Gitem/s │ │ +├─ mul_i64_nonnull 29.7 µs │ 35.65 µs │ 30.05 µs │ 30.18 µs │ 100 │ 100 +│ 1.103 Gitem/s │ 918.9 Mitem/s │ 1.09 Gitem/s │ 1.085 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 4.579 µs │ 3.519 µs │ 3.532 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 7.154 Gitem/s │ 9.309 Gitem/s │ 9.275 Gitem/s │ │ +├─ mul_u16_nonnull 2.359 µs │ 3.269 µs │ 2.429 µs │ 2.441 µs │ 100 │ 100 +│ 13.88 Gitem/s │ 10.02 Gitem/s │ 13.48 Gitem/s │ 13.42 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 11.21 µs │ 7.059 µs │ 7.105 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 2.92 Gitem/s │ 4.641 Gitem/s │ 4.611 Gitem/s │ │ +├─ mul_u64_nonnull 30.33 µs │ 34.65 µs │ 30.4 µs │ 30.53 µs │ 100 │ 100 +│ 1.08 Gitem/s │ 945.4 Mitem/s │ 1.077 Gitem/s │ 1.073 Gitem/s │ │ +╰─ sub_i64_constant 8.949 µs │ 12.59 µs │ 9.079 µs │ 9.155 µs │ 100 │ 100 + 3.661 Gitem/s │ 2.6 Gitem/s │ 3.608 Gitem/s │ 3.578 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md new file mode 100644 index 00000000000..fc941735f9c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md @@ -0,0 +1,39 @@ + + + +# `land-final-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.979 µs │ 88.65 µs │ 9.269 µs │ 10.12 µs │ 100 │ 100 +│ 3.649 Gitem/s │ 369.6 Mitem/s │ 3.534 Gitem/s │ 3.237 Gitem/s │ │ +├─ add_i64_nonnull 9.299 µs │ 13.4 µs │ 9.379 µs │ 9.444 µs │ 100 │ 100 +│ 3.523 Gitem/s │ 2.443 Gitem/s │ 3.493 Gitem/s │ 3.469 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 55 µs │ 45.02 µs │ 45.48 µs │ 100 │ 100 +│ 728.8 Mitem/s │ 595.6 Mitem/s │ 727.6 Mitem/s │ 720.4 Mitem/s │ │ +├─ mul_i8_nonnull 5.959 µs │ 44.56 µs │ 6.389 µs │ 6.855 µs │ 100 │ 100 +│ 5.498 Gitem/s │ 735.3 Mitem/s │ 5.128 Gitem/s │ 4.78 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 7.599 µs │ 4.265 µs │ 4.321 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 4.311 Gitem/s │ 7.682 Gitem/s │ 7.581 Gitem/s │ │ +├─ mul_i32_constant 18.6 µs │ 22.22 µs │ 18.69 µs │ 18.81 µs │ 100 │ 100 +│ 1.76 Gitem/s │ 1.474 Gitem/s │ 1.753 Gitem/s │ 1.741 Gitem/s │ │ +├─ mul_i32_nonnull 26.52 µs │ 34.76 µs │ 26.59 µs │ 26.77 µs │ 100 │ 100 +│ 1.235 Gitem/s │ 942.6 Mitem/s │ 1.231 Gitem/s │ 1.223 Gitem/s │ │ +├─ mul_i32_nullable 27.3 µs │ 51.11 µs │ 27.4 µs │ 27.77 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 641 Mitem/s │ 1.195 Gitem/s │ 1.179 Gitem/s │ │ +├─ mul_i64_nonnull 23.37 µs │ 27.17 µs │ 23.46 µs │ 23.56 µs │ 100 │ 100 +│ 1.401 Gitem/s │ 1.206 Gitem/s │ 1.396 Gitem/s │ 1.39 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 59.52 µs │ 3.514 µs │ 4.079 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 550.5 Mitem/s │ 9.322 Gitem/s │ 8.033 Gitem/s │ │ +├─ mul_u16_nonnull 2.709 µs │ 3.799 µs │ 2.789 µs │ 2.798 µs │ 100 │ 100 +│ 12.09 Gitem/s │ 8.623 Gitem/s │ 11.74 Gitem/s │ 11.71 Gitem/s │ │ +├─ mul_u32_nonnull 7.049 µs │ 10.9 µs │ 7.129 µs │ 7.19 µs │ 100 │ 100 +│ 4.648 Gitem/s │ 3.003 Gitem/s │ 4.595 Gitem/s │ 4.556 Gitem/s │ │ +├─ mul_u64_nonnull 19.26 µs │ 22.46 µs │ 19.37 µs │ 19.45 µs │ 100 │ 100 +│ 1.7 Gitem/s │ 1.458 Gitem/s │ 1.691 Gitem/s │ 1.683 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 13.94 µs │ 9.149 µs │ 9.215 µs │ 100 │ 100 + 3.636 Gitem/s │ 2.348 Gitem/s │ 3.581 Gitem/s │ 3.555 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md new file mode 100644 index 00000000000..6588d08f1e1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md @@ -0,0 +1,39 @@ + + + +# `land-final-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 73.57 µs │ 9.279 µs │ 9.987 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 445.3 Mitem/s │ 3.531 Gitem/s │ 3.28 Gitem/s │ │ +├─ add_i64_nonnull 9.319 µs │ 14.28 µs │ 9.389 µs │ 9.475 µs │ 100 │ 100 +│ 3.515 Gitem/s │ 2.293 Gitem/s │ 3.489 Gitem/s │ 3.458 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 62.15 µs │ 45.06 µs │ 45.46 µs │ 100 │ 100 +│ 728.6 Mitem/s │ 527.2 Mitem/s │ 727 Mitem/s │ 720.7 Mitem/s │ │ +├─ mul_i8_nonnull 5.979 µs │ 41.01 µs │ 6.409 µs │ 6.853 µs │ 100 │ 100 +│ 5.479 Gitem/s │ 798.8 Mitem/s │ 5.112 Gitem/s │ 4.78 Gitem/s │ │ +├─ mul_i16_nonnull 4.229 µs │ 5.739 µs │ 4.299 µs │ 4.314 µs │ 100 │ 100 +│ 7.746 Gitem/s │ 5.708 Gitem/s │ 7.62 Gitem/s │ 7.595 Gitem/s │ │ +├─ mul_i32_constant 18.6 µs │ 23.91 µs │ 18.7 µs │ 18.79 µs │ 100 │ 100 +│ 1.76 Gitem/s │ 1.369 Gitem/s │ 1.751 Gitem/s │ 1.743 Gitem/s │ │ +├─ mul_i32_nonnull 26.56 µs │ 30.13 µs │ 26.64 µs │ 26.74 µs │ 100 │ 100 +│ 1.233 Gitem/s │ 1.087 Gitem/s │ 1.229 Gitem/s │ 1.225 Gitem/s │ │ +├─ mul_i32_nullable 27.35 µs │ 42.74 µs │ 27.44 µs │ 27.69 µs │ 100 │ 100 +│ 1.197 Gitem/s │ 766.5 Mitem/s │ 1.193 Gitem/s │ 1.183 Gitem/s │ │ +├─ mul_i64_nonnull 23.31 µs │ 27.31 µs │ 23.43 µs │ 23.56 µs │ 100 │ 100 +│ 1.405 Gitem/s │ 1.199 Gitem/s │ 1.398 Gitem/s │ 1.39 Gitem/s │ │ +├─ mul_u8_nonnull 3.479 µs │ 52.76 µs │ 3.549 µs │ 4.103 µs │ 100 │ 100 +│ 9.416 Gitem/s │ 620.9 Mitem/s │ 9.231 Gitem/s │ 7.984 Gitem/s │ │ +├─ mul_u16_nonnull 2.739 µs │ 3.799 µs │ 2.81 µs │ 2.824 µs │ 100 │ 100 +│ 11.96 Gitem/s │ 8.623 Gitem/s │ 11.66 Gitem/s │ 11.6 Gitem/s │ │ +├─ mul_u32_nonnull 7.089 µs │ 10.28 µs │ 7.149 µs │ 7.207 µs │ 100 │ 100 +│ 4.621 Gitem/s │ 3.184 Gitem/s │ 4.583 Gitem/s │ 4.546 Gitem/s │ │ +├─ mul_u64_nonnull 19.32 µs │ 23.55 µs │ 19.38 µs │ 19.47 µs │ 100 │ 100 +│ 1.695 Gitem/s │ 1.391 Gitem/s │ 1.689 Gitem/s │ 1.682 Gitem/s │ │ +╰─ sub_i64_constant 8.939 µs │ 30.07 µs │ 9.159 µs │ 9.386 µs │ 100 │ 100 + 3.665 Gitem/s │ 1.089 Gitem/s │ 3.577 Gitem/s │ 3.49 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md new file mode 100644 index 00000000000..b12390b54c2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md @@ -0,0 +1,38 @@ + + + +# `stage0-base-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.319 µs │ 62.83 µs │ 8.449 µs │ 9.036 µs │ 100 │ 100 +│ 3.938 Gitem/s │ 521.5 Mitem/s │ 3.877 Gitem/s │ 3.626 Gitem/s │ │ +├─ add_i64_nonnull 9.139 µs │ 13.11 µs │ 9.205 µs │ 9.316 µs │ 100 │ 100 +│ 3.585 Gitem/s │ 2.497 Gitem/s │ 3.559 Gitem/s │ 3.517 Gitem/s │ │ +├─ div_i64_nonnull 44.78 µs │ 66.09 µs │ 44.85 µs │ 45.23 µs │ 100 │ 100 +│ 731.5 Mitem/s │ 495.8 Mitem/s │ 730.4 Mitem/s │ 724.4 Mitem/s │ │ +├─ mul_i8_nonnull 5.799 µs │ 17.72 µs │ 6.184 µs │ 6.39 µs │ 100 │ 100 +│ 5.649 Gitem/s │ 1.848 Gitem/s │ 5.298 Gitem/s │ 5.127 Gitem/s │ │ +├─ mul_i16_nonnull 4.009 µs │ 10.13 µs │ 4.099 µs │ 4.214 µs │ 100 │ 100 +│ 8.172 Gitem/s │ 3.234 Gitem/s │ 7.992 Gitem/s │ 7.774 Gitem/s │ │ +├─ mul_i32_constant 26.35 µs │ 30.02 µs │ 26.42 µs │ 26.53 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 1.091 Gitem/s │ 1.239 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 30.26 µs │ 26.41 µs │ 26.51 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.082 Gitem/s │ 1.24 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nullable 27.26 µs │ 48.92 µs │ 27.35 µs │ 27.7 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 669.8 Mitem/s │ 1.197 Gitem/s │ 1.182 Gitem/s │ │ +├─ mul_i64_nonnull 23.13 µs │ 28.03 µs │ 23.22 µs │ 23.37 µs │ 100 │ 100 +│ 1.416 Gitem/s │ 1.168 Gitem/s │ 1.41 Gitem/s │ 1.402 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 6.989 µs │ 3.319 µs │ 3.365 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 4.687 Gitem/s │ 9.87 Gitem/s │ 9.735 Gitem/s │ │ +├─ mul_u16_nonnull 2.539 µs │ 3.489 µs │ 2.599 µs │ 2.613 µs │ 100 │ 100 +│ 12.9 Gitem/s │ 9.389 Gitem/s │ 12.6 Gitem/s │ 12.53 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 12.13 µs │ 6.939 µs │ 7.009 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 2.699 Gitem/s │ 4.721 Gitem/s │ 4.674 Gitem/s │ │ +├─ mul_u64_nonnull 19.14 µs │ 23.82 µs │ 19.21 µs │ 19.32 µs │ 100 │ 100 +│ 1.711 Gitem/s │ 1.375 Gitem/s │ 1.704 Gitem/s │ 1.695 Gitem/s │ │ +╰─ sub_i64_constant 8.129 µs │ 12.18 µs │ 8.255 µs │ 8.34 µs │ 100 │ 100 + 4.03 Gitem/s │ 2.688 Gitem/s │ 3.969 Gitem/s │ 3.928 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md new file mode 100644 index 00000000000..c2254e0238f --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md @@ -0,0 +1,38 @@ + + + +# `stage0-base-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.089 µs │ 63.25 µs │ 8.399 µs │ 8.966 µs │ 100 │ 100 +│ 4.05 Gitem/s │ 518 Mitem/s │ 3.901 Gitem/s │ 3.654 Gitem/s │ │ +├─ add_i64_nonnull 9.069 µs │ 13.16 µs │ 9.149 µs │ 9.232 µs │ 100 │ 100 +│ 3.612 Gitem/s │ 2.488 Gitem/s │ 3.581 Gitem/s │ 3.549 Gitem/s │ │ +├─ div_i64_nonnull 44.73 µs │ 51.02 µs │ 44.8 µs │ 45.03 µs │ 100 │ 100 +│ 732.4 Mitem/s │ 642.1 Mitem/s │ 731.2 Mitem/s │ 727.6 Mitem/s │ │ +├─ mul_i8_nonnull 5.979 µs │ 11.05 µs │ 6.199 µs │ 6.323 µs │ 100 │ 100 +│ 5.479 Gitem/s │ 2.962 Gitem/s │ 5.285 Gitem/s │ 5.182 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 8.709 µs │ 4.119 µs │ 4.205 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 3.762 Gitem/s │ 7.953 Gitem/s │ 7.791 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 29.65 µs │ 26.43 µs │ 26.51 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.104 Gitem/s │ 1.239 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nonnull 26.37 µs │ 36.95 µs │ 26.42 µs │ 26.61 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 886.8 Mitem/s │ 1.239 Gitem/s │ 1.231 Gitem/s │ │ +├─ mul_i32_nullable 27.3 µs │ 49.11 µs │ 27.4 µs │ 27.76 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 667.1 Mitem/s │ 1.195 Gitem/s │ 1.18 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 27.98 µs │ 23.2 µs │ 23.32 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 1.171 Gitem/s │ 1.411 Gitem/s │ 1.404 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 4.809 µs │ 3.329 µs │ 3.345 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 6.812 Gitem/s │ 9.84 Gitem/s │ 9.794 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.649 µs │ 2.599 µs │ 2.611 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 8.978 Gitem/s │ 12.6 Gitem/s │ 12.54 Gitem/s │ │ +├─ mul_u32_nonnull 6.889 µs │ 10.15 µs │ 6.949 µs │ 6.999 µs │ 100 │ 100 +│ 4.756 Gitem/s │ 3.228 Gitem/s │ 4.714 Gitem/s │ 4.681 Gitem/s │ │ +├─ mul_u64_nonnull 19.12 µs │ 24.11 µs │ 19.19 µs │ 19.3 µs │ 100 │ 100 +│ 1.712 Gitem/s │ 1.358 Gitem/s │ 1.706 Gitem/s │ 1.697 Gitem/s │ │ +╰─ sub_i64_constant 8.119 µs │ 12.58 µs │ 8.239 µs │ 8.323 µs │ 100 │ 100 + 4.035 Gitem/s │ 2.602 Gitem/s │ 3.976 Gitem/s │ 3.936 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md new file mode 100644 index 00000000000..c8539b0b1c6 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md @@ -0,0 +1,38 @@ + + + +# `stage0-candidate-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.13 µs │ 93.77 µs │ 9.269 µs │ 10.17 µs │ 100 │ 100 +│ 3.588 Gitem/s │ 349.4 Mitem/s │ 3.534 Gitem/s │ 3.22 Gitem/s │ │ +├─ add_i64_nonnull 9.369 µs │ 12.45 µs │ 9.455 µs │ 9.51 µs │ 100 │ 100 +│ 3.497 Gitem/s │ 2.629 Gitem/s │ 3.465 Gitem/s │ 3.445 Gitem/s │ │ +├─ div_i64_nonnull 45.01 µs │ 54.4 µs │ 45.09 µs │ 45.42 µs │ 100 │ 100 +│ 727.8 Mitem/s │ 602.2 Mitem/s │ 726.5 Mitem/s │ 721.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 12.25 µs │ 4.694 µs │ 4.777 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 2.672 Gitem/s │ 6.979 Gitem/s │ 6.858 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 6.999 µs │ 4.269 µs │ 4.33 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 4.681 Gitem/s │ 7.674 Gitem/s │ 7.567 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 21.99 µs │ 18.88 µs │ 19 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 1.489 Gitem/s │ 1.734 Gitem/s │ 1.724 Gitem/s │ │ +├─ mul_i32_nonnull 28.23 µs │ 31.75 µs │ 28.39 µs │ 28.48 µs │ 100 │ 100 +│ 1.16 Gitem/s │ 1.031 Gitem/s │ 1.153 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 44.74 µs │ 29.18 µs │ 29.42 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 732.2 Mitem/s │ 1.122 Gitem/s │ 1.113 Gitem/s │ │ +├─ mul_i64_nonnull 29.7 µs │ 34.12 µs │ 30.02 µs │ 30.14 µs │ 100 │ 100 +│ 1.102 Gitem/s │ 960.1 Mitem/s │ 1.091 Gitem/s │ 1.087 Gitem/s │ │ +├─ mul_u8_nonnull 3.489 µs │ 7.519 µs │ 3.539 µs │ 3.602 µs │ 100 │ 100 +│ 9.389 Gitem/s │ 4.357 Gitem/s │ 9.257 Gitem/s │ 9.095 Gitem/s │ │ +├─ mul_u16_nonnull 2.349 µs │ 3.849 µs │ 2.429 µs │ 2.446 µs │ 100 │ 100 +│ 13.94 Gitem/s │ 8.511 Gitem/s │ 13.48 Gitem/s │ 13.39 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 9.889 µs │ 7.069 µs │ 7.11 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 3.313 Gitem/s │ 4.634 Gitem/s │ 4.608 Gitem/s │ │ +├─ mul_u64_nonnull 30.36 µs │ 33.89 µs │ 30.43 µs │ 30.55 µs │ 100 │ 100 +│ 1.078 Gitem/s │ 966.6 Mitem/s │ 1.076 Gitem/s │ 1.072 Gitem/s │ │ +╰─ sub_i64_constant 8.979 µs │ 12.15 µs │ 9.099 µs │ 9.159 µs │ 100 │ 100 + 3.649 Gitem/s │ 2.696 Gitem/s │ 3.6 Gitem/s │ 3.577 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md new file mode 100644 index 00000000000..401b1ae9bc8 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md @@ -0,0 +1,38 @@ + + + +# `stage0-candidate-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.159 µs │ 88.36 µs │ 9.29 µs │ 10.41 µs │ 100 │ 100 +│ 3.577 Gitem/s │ 370.8 Mitem/s │ 3.527 Gitem/s │ 3.147 Gitem/s │ │ +├─ add_i64_nonnull 9.389 µs │ 12.37 µs │ 9.49 µs │ 9.622 µs │ 100 │ 100 +│ 3.489 Gitem/s │ 2.646 Gitem/s │ 3.452 Gitem/s │ 3.405 Gitem/s │ │ +├─ div_i64_nonnull 45.1 µs │ 48.73 µs │ 45.16 µs │ 45.34 µs │ 100 │ 100 +│ 726.4 Mitem/s │ 672.3 Mitem/s │ 725.5 Mitem/s │ 722.6 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 7.529 µs │ 4.699 µs │ 4.751 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 4.351 Gitem/s │ 6.972 Gitem/s │ 6.897 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 6.379 µs │ 4.269 µs │ 4.295 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 5.136 Gitem/s │ 7.674 Gitem/s │ 7.629 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 24.05 µs │ 18.87 µs │ 19 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 1.361 Gitem/s │ 1.736 Gitem/s │ 1.724 Gitem/s │ │ +├─ mul_i32_nonnull 28.19 µs │ 31.61 µs │ 28.35 µs │ 28.45 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 1.036 Gitem/s │ 1.155 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29 µs │ 50.06 µs │ 29.15 µs │ 29.47 µs │ 100 │ 100 +│ 1.129 Gitem/s │ 654.5 Mitem/s │ 1.123 Gitem/s │ 1.111 Gitem/s │ │ +├─ mul_i64_nonnull 29.82 µs │ 33.7 µs │ 30.08 µs │ 30.21 µs │ 100 │ 100 +│ 1.098 Gitem/s │ 972 Mitem/s │ 1.089 Gitem/s │ 1.084 Gitem/s │ │ +├─ mul_u8_nonnull 3.469 µs │ 9.249 µs │ 3.529 µs │ 3.592 µs │ 100 │ 100 +│ 9.443 Gitem/s │ 3.542 Gitem/s │ 9.283 Gitem/s │ 9.119 Gitem/s │ │ +├─ mul_u16_nonnull 2.369 µs │ 3.699 µs │ 2.429 µs │ 2.448 µs │ 100 │ 100 +│ 13.82 Gitem/s │ 8.856 Gitem/s │ 13.48 Gitem/s │ 13.38 Gitem/s │ │ +├─ mul_u32_nonnull 6.989 µs │ 9.659 µs │ 7.059 µs │ 7.111 µs │ 100 │ 100 +│ 4.687 Gitem/s │ 3.392 Gitem/s │ 4.641 Gitem/s │ 4.607 Gitem/s │ │ +├─ mul_u64_nonnull 30.41 µs │ 33.95 µs │ 30.49 µs │ 30.63 µs │ 100 │ 100 +│ 1.077 Gitem/s │ 964.9 Mitem/s │ 1.074 Gitem/s │ 1.069 Gitem/s │ │ +╰─ sub_i64_constant 8.989 µs │ 11.76 µs │ 9.099 µs │ 9.169 µs │ 100 │ 100 + 3.645 Gitem/s │ 2.784 Gitem/s │ 3.6 Gitem/s │ 3.573 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md new file mode 100644 index 00000000000..4b6e17ff63e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-base-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.089 µs │ 783.5 µs │ 8.419 µs │ 16.21 µs │ 100 │ 100 +│ 4.05 Gitem/s │ 41.81 Mitem/s │ 3.891 Gitem/s │ 2.02 Gitem/s │ │ +├─ add_i64_nonnull 9.089 µs │ 32.99 µs │ 9.189 µs │ 9.53 µs │ 100 │ 100 +│ 3.604 Gitem/s │ 992.9 Mitem/s │ 3.565 Gitem/s │ 3.438 Gitem/s │ │ +├─ div_i64_nonnull 44.76 µs │ 76.23 µs │ 44.84 µs │ 45.51 µs │ 100 │ 100 +│ 731.9 Mitem/s │ 429.8 Mitem/s │ 730.6 Mitem/s │ 719.8 Mitem/s │ │ +├─ mul_i8_nonnull 5.929 µs │ 72.34 µs │ 6.239 µs │ 7.053 µs │ 100 │ 100 +│ 5.526 Gitem/s │ 452.9 Mitem/s │ 5.251 Gitem/s │ 4.645 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 61.69 µs │ 4.114 µs │ 4.692 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 531.1 Mitem/s │ 7.963 Gitem/s │ 6.983 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 55.98 µs │ 26.43 µs │ 26.85 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 585.2 Mitem/s │ 1.239 Gitem/s │ 1.219 Gitem/s │ │ +├─ mul_i32_nonnull 26.38 µs │ 37.56 µs │ 26.42 µs │ 26.65 µs │ 100 │ 100 +│ 1.241 Gitem/s │ 872.3 Mitem/s │ 1.239 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.23 µs │ 340.3 µs │ 27.36 µs │ 30.62 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 96.26 Mitem/s │ 1.197 Gitem/s │ 1.07 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 44.46 µs │ 23.2 µs │ 23.5 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 736.8 Mitem/s │ 1.411 Gitem/s │ 1.394 Gitem/s │ │ +├─ mul_u8_nonnull 3.269 µs │ 51.96 µs │ 3.329 µs │ 3.829 µs │ 100 │ 100 +│ 10.02 Gitem/s │ 630.6 Mitem/s │ 9.84 Gitem/s │ 8.556 Gitem/s │ │ +├─ mul_u16_nonnull 2.559 µs │ 30.97 µs │ 2.609 µs │ 2.898 µs │ 100 │ 100 +│ 12.8 Gitem/s │ 1.057 Gitem/s │ 12.55 Gitem/s │ 11.3 Gitem/s │ │ +├─ mul_u32_nonnull 6.88 µs │ 26.3 µs │ 6.959 µs │ 7.202 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 1.245 Gitem/s │ 4.708 Gitem/s │ 4.549 Gitem/s │ │ +├─ mul_u64_nonnull 19.11 µs │ 40.79 µs │ 19.18 µs │ 19.48 µs │ 100 │ 100 +│ 1.713 Gitem/s │ 803.3 Mitem/s │ 1.707 Gitem/s │ 1.681 Gitem/s │ │ +╰─ sub_i64_constant 8.109 µs │ 41.21 µs │ 8.219 µs │ 8.589 µs │ 100 │ 100 + 4.04 Gitem/s │ 794.9 Mitem/s │ 3.986 Gitem/s │ 3.814 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md new file mode 100644 index 00000000000..e540fafc12f --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-base-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.309 µs │ 51.73 µs │ 8.459 µs │ 8.93 µs │ 100 │ 100 +│ 3.943 Gitem/s │ 633.3 Mitem/s │ 3.873 Gitem/s │ 3.669 Gitem/s │ │ +├─ add_i64_nonnull 9.119 µs │ 19.9 µs │ 9.199 µs │ 9.345 µs │ 100 │ 100 +│ 3.593 Gitem/s │ 1.646 Gitem/s │ 3.561 Gitem/s │ 3.506 Gitem/s │ │ +├─ div_i64_nonnull 44.77 µs │ 49.58 µs │ 44.85 µs │ 45.06 µs │ 100 │ 100 +│ 731.7 Mitem/s │ 660.7 Mitem/s │ 730.5 Mitem/s │ 727.1 Mitem/s │ │ +├─ mul_i8_nonnull 5.779 µs │ 10.19 µs │ 6.15 µs │ 6.267 µs │ 100 │ 100 +│ 5.669 Gitem/s │ 3.212 Gitem/s │ 5.327 Gitem/s │ 5.228 Gitem/s │ │ +├─ mul_i16_nonnull 4.059 µs │ 8.189 µs │ 4.109 µs │ 4.198 µs │ 100 │ 100 +│ 8.071 Gitem/s │ 4.001 Gitem/s │ 7.973 Gitem/s │ 7.804 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 30.04 µs │ 26.44 µs │ 26.54 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.09 Gitem/s │ 1.238 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 29.99 µs │ 26.41 µs │ 26.54 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.092 Gitem/s │ 1.24 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nullable 27.23 µs │ 42.3 µs │ 27.35 µs │ 27.61 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 774.4 Mitem/s │ 1.197 Gitem/s │ 1.186 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 27.63 µs │ 23.26 µs │ 23.41 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 1.185 Gitem/s │ 1.408 Gitem/s │ 1.399 Gitem/s │ │ +├─ mul_u8_nonnull 3.269 µs │ 4.809 µs │ 3.319 µs │ 3.339 µs │ 100 │ 100 +│ 10.02 Gitem/s │ 6.812 Gitem/s │ 9.87 Gitem/s │ 9.812 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.519 µs │ 2.609 µs │ 2.618 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 9.309 Gitem/s │ 12.55 Gitem/s │ 12.51 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 11.37 µs │ 6.95 µs │ 7.026 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 2.879 Gitem/s │ 4.714 Gitem/s │ 4.663 Gitem/s │ │ +├─ mul_u64_nonnull 19.16 µs │ 22.33 µs │ 19.21 µs │ 19.3 µs │ 100 │ 100 +│ 1.709 Gitem/s │ 1.466 Gitem/s │ 1.705 Gitem/s │ 1.697 Gitem/s │ │ +╰─ sub_i64_constant 8.139 µs │ 11.6 µs │ 8.259 µs │ 8.319 µs │ 100 │ 100 + 4.025 Gitem/s │ 2.822 Gitem/s │ 3.967 Gitem/s │ 3.938 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md new file mode 100644 index 00000000000..89efbeab66c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-candidate-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 1.038 ms │ 9.279 µs │ 19.64 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 31.54 Mitem/s │ 3.531 Gitem/s │ 1.667 Gitem/s │ │ +├─ add_i64_nonnull 9.399 µs │ 23.57 µs │ 9.459 µs │ 9.66 µs │ 100 │ 100 +│ 3.486 Gitem/s │ 1.389 Gitem/s │ 3.463 Gitem/s │ 3.391 Gitem/s │ │ +├─ div_i64_nonnull 45.03 µs │ 63.07 µs │ 45.1 µs │ 45.48 µs │ 100 │ 100 +│ 727.5 Mitem/s │ 519.4 Mitem/s │ 726.4 Mitem/s │ 720.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.629 µs │ 65.15 µs │ 4.689 µs │ 5.344 µs │ 100 │ 100 +│ 7.077 Gitem/s │ 502.8 Mitem/s │ 6.987 Gitem/s │ 6.131 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 71.32 µs │ 4.259 µs │ 4.934 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 459.3 Mitem/s │ 7.692 Gitem/s │ 6.64 Gitem/s │ │ +├─ mul_i32_constant 18.71 µs │ 73.84 µs │ 18.82 µs │ 19.43 µs │ 100 │ 100 +│ 1.75 Gitem/s │ 443.7 Mitem/s │ 1.74 Gitem/s │ 1.685 Gitem/s │ │ +├─ mul_i32_nonnull 28.22 µs │ 32.07 µs │ 28.34 µs │ 28.45 µs │ 100 │ 100 +│ 1.16 Gitem/s │ 1.021 Gitem/s │ 1.155 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 237.4 µs │ 29.16 µs │ 31.4 µs │ 100 │ 100 +│ 1.127 Gitem/s │ 138 Mitem/s │ 1.123 Gitem/s │ 1.043 Gitem/s │ │ +├─ mul_i64_nonnull 29.72 µs │ 54.86 µs │ 30.07 µs │ 30.53 µs │ 100 │ 100 +│ 1.102 Gitem/s │ 597.1 Mitem/s │ 1.089 Gitem/s │ 1.073 Gitem/s │ │ +├─ mul_u8_nonnull 3.469 µs │ 15.4 µs │ 3.529 µs │ 3.658 µs │ 100 │ 100 +│ 9.443 Gitem/s │ 2.126 Gitem/s │ 9.283 Gitem/s │ 8.956 Gitem/s │ │ +├─ mul_u16_nonnull 2.339 µs │ 13.45 µs │ 2.419 µs │ 2.574 µs │ 100 │ 100 +│ 14 Gitem/s │ 2.434 Gitem/s │ 13.54 Gitem/s │ 12.72 Gitem/s │ │ +├─ mul_u32_nonnull 6.969 µs │ 19.59 µs │ 7.049 µs │ 7.223 µs │ 100 │ 100 +│ 4.701 Gitem/s │ 1.671 Gitem/s │ 4.648 Gitem/s │ 4.536 Gitem/s │ │ +├─ mul_u64_nonnull 30.36 µs │ 42.47 µs │ 30.45 µs │ 30.69 µs │ 100 │ 100 +│ 1.079 Gitem/s │ 771.3 Mitem/s │ 1.075 Gitem/s │ 1.067 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 31.68 µs │ 9.119 µs │ 9.431 µs │ 100 │ 100 + 3.636 Gitem/s │ 1.034 Gitem/s │ 3.593 Gitem/s │ 3.474 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md new file mode 100644 index 00000000000..4bb5ec2261e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-candidate-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 65.44 µs │ 9.309 µs │ 9.916 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 500.6 Mitem/s │ 3.519 Gitem/s │ 3.304 Gitem/s │ │ +├─ add_i64_nonnull 9.429 µs │ 18.82 µs │ 9.529 µs │ 9.64 µs │ 100 │ 100 +│ 3.474 Gitem/s │ 1.74 Gitem/s │ 3.438 Gitem/s │ 3.399 Gitem/s │ │ +├─ div_i64_nonnull 45.09 µs │ 51.42 µs │ 45.16 µs │ 45.37 µs │ 100 │ 100 +│ 726.5 Mitem/s │ 637.2 Mitem/s │ 725.4 Mitem/s │ 722.1 Mitem/s │ │ +├─ mul_i8_nonnull 4.669 µs │ 7.159 µs │ 4.729 µs │ 4.781 µs │ 100 │ 100 +│ 7.017 Gitem/s │ 4.576 Gitem/s │ 6.928 Gitem/s │ 6.853 Gitem/s │ │ +├─ mul_i16_nonnull 4.249 µs │ 8.269 µs │ 4.319 µs │ 4.391 µs │ 100 │ 100 +│ 7.71 Gitem/s │ 3.962 Gitem/s │ 7.585 Gitem/s │ 7.461 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.35 µs │ 18.85 µs │ 18.93 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.465 Gitem/s │ 1.737 Gitem/s │ 1.73 Gitem/s │ │ +├─ mul_i32_nonnull 28.25 µs │ 32.31 µs │ 28.39 µs │ 28.46 µs │ 100 │ 100 +│ 1.159 Gitem/s │ 1.013 Gitem/s │ 1.153 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 38.59 µs │ 29.18 µs │ 29.37 µs │ 100 │ 100 +│ 1.127 Gitem/s │ 848.9 Mitem/s │ 1.122 Gitem/s │ 1.115 Gitem/s │ │ +├─ mul_i64_nonnull 29.84 µs │ 33.8 µs │ 30.16 µs │ 30.24 µs │ 100 │ 100 +│ 1.097 Gitem/s │ 969.1 Mitem/s │ 1.086 Gitem/s │ 1.083 Gitem/s │ │ +├─ mul_u8_nonnull 3.509 µs │ 6.339 µs │ 3.579 µs │ 3.604 µs │ 100 │ 100 +│ 9.336 Gitem/s │ 5.168 Gitem/s │ 9.153 Gitem/s │ 9.091 Gitem/s │ │ +├─ mul_u16_nonnull 2.389 µs │ 38.12 µs │ 2.474 µs │ 2.857 µs │ 100 │ 100 +│ 13.71 Gitem/s │ 859.3 Mitem/s │ 13.24 Gitem/s │ 11.46 Gitem/s │ │ +├─ mul_u32_nonnull 7.019 µs │ 8.19 µs │ 7.109 µs │ 7.121 µs │ 100 │ 100 +│ 4.667 Gitem/s │ 4 Gitem/s │ 4.608 Gitem/s │ 4.601 Gitem/s │ │ +├─ mul_u64_nonnull 30.4 µs │ 33.46 µs │ 30.49 µs │ 30.63 µs │ 100 │ 100 +│ 1.077 Gitem/s │ 979 Mitem/s │ 1.074 Gitem/s │ 1.069 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 10.35 µs │ 9.119 µs │ 9.142 µs │ 100 │ 100 + 3.636 Gitem/s │ 3.163 Gitem/s │ 3.593 Gitem/s │ 3.584 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md new file mode 100644 index 00000000000..06e474ab7f0 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-owned-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.869 µs │ 97.56 µs │ 9.224 µs │ 10.16 µs │ 100 │ 100 +│ 3.694 Gitem/s │ 335.8 Mitem/s │ 3.552 Gitem/s │ 3.225 Gitem/s │ │ +├─ add_i64_nonnull 9.299 µs │ 13.77 µs │ 9.389 µs │ 9.495 µs │ 100 │ 100 +│ 3.523 Gitem/s │ 2.377 Gitem/s │ 3.489 Gitem/s │ 3.45 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 54.06 µs │ 45.04 µs │ 45.34 µs │ 100 │ 100 +│ 728.8 Mitem/s │ 606.1 Mitem/s │ 727.5 Mitem/s │ 722.7 Mitem/s │ │ +├─ mul_i8_nonnull 4.559 µs │ 58.61 µs │ 4.619 µs │ 5.202 µs │ 100 │ 100 +│ 7.186 Gitem/s │ 558.9 Mitem/s │ 7.092 Gitem/s │ 6.298 Gitem/s │ │ +├─ mul_i16_nonnull 4.159 µs │ 5.809 µs │ 4.229 µs │ 4.244 µs │ 100 │ 100 +│ 7.877 Gitem/s │ 5.64 Gitem/s │ 7.746 Gitem/s │ 7.72 Gitem/s │ │ +├─ mul_i32_constant 32.23 µs │ 36.15 µs │ 32.36 µs │ 32.49 µs │ 100 │ 100 +│ 1.016 Gitem/s │ 906.2 Mitem/s │ 1.012 Gitem/s │ 1.008 Gitem/s │ │ +├─ mul_i32_nonnull 27.81 µs │ 43.9 µs │ 31.24 µs │ 31.22 µs │ 100 │ 100 +│ 1.177 Gitem/s │ 746.2 Mitem/s │ 1.048 Gitem/s │ 1.049 Gitem/s │ │ +├─ mul_i32_nullable 28.55 µs │ 50.24 µs │ 32.04 µs │ 31.62 µs │ 100 │ 100 +│ 1.147 Gitem/s │ 652.1 Mitem/s │ 1.022 Gitem/s │ 1.036 Gitem/s │ │ +├─ mul_i64_nonnull 25.31 µs │ 29.26 µs │ 25.65 µs │ 25.77 µs │ 100 │ 100 +│ 1.294 Gitem/s │ 1.119 Gitem/s │ 1.277 Gitem/s │ 1.271 Gitem/s │ │ +├─ mul_u8_nonnull 3.399 µs │ 55.89 µs │ 3.469 µs │ 3.999 µs │ 100 │ 100 +│ 9.638 Gitem/s │ 586.1 Mitem/s │ 9.443 Gitem/s │ 8.193 Gitem/s │ │ +├─ mul_u16_nonnull 2.289 µs │ 6.769 µs │ 2.369 µs │ 2.415 µs │ 100 │ 100 +│ 14.31 Gitem/s │ 4.84 Gitem/s │ 13.82 Gitem/s │ 13.56 Gitem/s │ │ +├─ mul_u32_nonnull 6.919 µs │ 9.879 µs │ 7.009 µs │ 7.059 µs │ 100 │ 100 +│ 4.735 Gitem/s │ 3.316 Gitem/s │ 4.674 Gitem/s │ 4.641 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 22.38 µs │ 19.41 µs │ 19.5 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.463 Gitem/s │ 1.687 Gitem/s │ 1.68 Gitem/s │ │ +╰─ sub_i64_constant 9.419 µs │ 12.37 µs │ 9.554 µs │ 9.607 µs │ 100 │ 100 + 3.478 Gitem/s │ 2.646 Gitem/s │ 3.429 Gitem/s │ 3.41 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md new file mode 100644 index 00000000000..a653d31d205 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-owned-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.919 µs │ 99.93 µs │ 9.229 µs │ 10.19 µs │ 100 │ 100 +│ 3.673 Gitem/s │ 327.8 Mitem/s │ 3.55 Gitem/s │ 3.214 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 12.53 µs │ 9.339 µs │ 9.417 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 2.613 Gitem/s │ 3.508 Gitem/s │ 3.479 Gitem/s │ │ +├─ div_i64_nonnull 44.91 µs │ 54.36 µs │ 45.01 µs │ 45.32 µs │ 100 │ 100 +│ 729.4 Mitem/s │ 602.6 Mitem/s │ 727.8 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 4.579 µs │ 61.3 µs │ 4.649 µs │ 5.229 µs │ 100 │ 100 +│ 7.154 Gitem/s │ 534.5 Mitem/s │ 7.047 Gitem/s │ 6.265 Gitem/s │ │ +├─ mul_i16_nonnull 4.169 µs │ 7.419 µs │ 4.229 µs │ 4.276 µs │ 100 │ 100 +│ 7.858 Gitem/s │ 4.416 Gitem/s │ 7.746 Gitem/s │ 7.661 Gitem/s │ │ +├─ mul_i32_constant 32.27 µs │ 35.87 µs │ 32.38 µs │ 32.49 µs │ 100 │ 100 +│ 1.015 Gitem/s │ 913.5 Mitem/s │ 1.011 Gitem/s │ 1.008 Gitem/s │ │ +├─ mul_i32_nonnull 27.77 µs │ 32.39 µs │ 31.23 µs │ 30.31 µs │ 100 │ 100 +│ 1.179 Gitem/s │ 1.011 Gitem/s │ 1.048 Gitem/s │ 1.08 Gitem/s │ │ +├─ mul_i32_nullable 28.53 µs │ 49.46 µs │ 32.04 µs │ 31.34 µs │ 100 │ 100 +│ 1.148 Gitem/s │ 662.3 Mitem/s │ 1.022 Gitem/s │ 1.045 Gitem/s │ │ +├─ mul_i64_nonnull 25.25 µs │ 29.06 µs │ 25.59 µs │ 25.68 µs │ 100 │ 100 +│ 1.297 Gitem/s │ 1.127 Gitem/s │ 1.28 Gitem/s │ 1.275 Gitem/s │ │ +├─ mul_u8_nonnull 3.429 µs │ 60.44 µs │ 3.479 µs │ 4.062 µs │ 100 │ 100 +│ 9.553 Gitem/s │ 542 Mitem/s │ 9.416 Gitem/s │ 8.065 Gitem/s │ │ +├─ mul_u16_nonnull 2.319 µs │ 7.879 µs │ 2.389 µs │ 2.446 µs │ 100 │ 100 +│ 14.12 Gitem/s │ 4.158 Gitem/s │ 13.71 Gitem/s │ 13.39 Gitem/s │ │ +├─ mul_u32_nonnull 6.949 µs │ 10.35 µs │ 7.019 µs │ 7.069 µs │ 100 │ 100 +│ 4.714 Gitem/s │ 3.163 Gitem/s │ 4.667 Gitem/s │ 4.635 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 22.75 µs │ 19.41 µs │ 19.49 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.44 Gitem/s │ 1.687 Gitem/s │ 1.68 Gitem/s │ │ +╰─ sub_i64_constant 9.439 µs │ 12.1 µs │ 9.569 µs │ 9.636 µs │ 100 │ 100 + 3.471 Gitem/s │ 2.705 Gitem/s │ 3.424 Gitem/s │ 3.4 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md new file mode 100644 index 00000000000..3053244a92a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-base-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.099 µs │ 766.1 µs │ 8.419 µs │ 16.03 µs │ 100 │ 100 +│ 4.045 Gitem/s │ 42.76 Mitem/s │ 3.891 Gitem/s │ 2.043 Gitem/s │ │ +├─ add_i64_nonnull 9.099 µs │ 30.45 µs │ 9.179 µs │ 9.474 µs │ 100 │ 100 +│ 3.6 Gitem/s │ 1.075 Gitem/s │ 3.569 Gitem/s │ 3.458 Gitem/s │ │ +├─ div_i64_nonnull 44.76 µs │ 75.04 µs │ 44.84 µs │ 45.32 µs │ 100 │ 100 +│ 731.9 Mitem/s │ 436.6 Mitem/s │ 730.6 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 5.809 µs │ 69.77 µs │ 6.209 µs │ 6.952 µs │ 100 │ 100 +│ 5.64 Gitem/s │ 469.5 Mitem/s │ 5.276 Gitem/s │ 4.713 Gitem/s │ │ +├─ mul_i16_nonnull 4.029 µs │ 66.19 µs │ 4.099 µs │ 4.725 µs │ 100 │ 100 +│ 8.131 Gitem/s │ 494.9 Mitem/s │ 7.992 Gitem/s │ 6.934 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 55.21 µs │ 26.42 µs │ 26.88 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 593.4 Mitem/s │ 1.239 Gitem/s │ 1.218 Gitem/s │ │ +├─ mul_i32_nonnull 26.34 µs │ 39.19 µs │ 26.39 µs │ 26.64 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 835.9 Mitem/s │ 1.241 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.21 µs │ 333.7 µs │ 27.37 µs │ 30.56 µs │ 100 │ 100 +│ 1.203 Gitem/s │ 98.17 Mitem/s │ 1.196 Gitem/s │ 1.072 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 43.91 µs │ 23.22 µs │ 23.6 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 746 Mitem/s │ 1.41 Gitem/s │ 1.388 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 52.38 µs │ 3.329 µs │ 3.817 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 625.4 Mitem/s │ 9.84 Gitem/s │ 8.582 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 30.18 µs │ 2.609 µs │ 2.929 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 1.085 Gitem/s │ 12.55 Gitem/s │ 11.18 Gitem/s │ │ +├─ mul_u32_nonnull 6.869 µs │ 27.32 µs │ 6.939 µs │ 7.147 µs │ 100 │ 100 +│ 4.769 Gitem/s │ 1.198 Gitem/s │ 4.721 Gitem/s │ 4.584 Gitem/s │ │ +├─ mul_u64_nonnull 19.14 µs │ 41.82 µs │ 19.22 µs │ 19.57 µs │ 100 │ 100 +│ 1.711 Gitem/s │ 783.3 Mitem/s │ 1.704 Gitem/s │ 1.674 Gitem/s │ │ +╰─ sub_i64_constant 8.159 µs │ 40.98 µs │ 8.249 µs │ 8.63 µs │ 100 │ 100 + 4.015 Gitem/s │ 799.4 Mitem/s │ 3.971 Gitem/s │ 3.796 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md new file mode 100644 index 00000000000..793ece2b6f1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-base-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.299 µs │ 47.08 µs │ 8.429 µs │ 8.916 µs │ 100 │ 100 +│ 3.948 Gitem/s │ 695.8 Mitem/s │ 3.887 Gitem/s │ 3.675 Gitem/s │ │ +├─ add_i64_nonnull 9.139 µs │ 12.35 µs │ 9.209 µs │ 9.286 µs │ 100 │ 100 +│ 3.585 Gitem/s │ 2.651 Gitem/s │ 3.557 Gitem/s │ 3.528 Gitem/s │ │ +├─ div_i64_nonnull 44.8 µs │ 49.64 µs │ 44.87 µs │ 45.09 µs │ 100 │ 100 +│ 731.2 Mitem/s │ 659.9 Mitem/s │ 730.1 Mitem/s │ 726.6 Mitem/s │ │ +├─ mul_i8_nonnull 5.869 µs │ 9.279 µs │ 6.174 µs │ 6.295 µs │ 100 │ 100 +│ 5.582 Gitem/s │ 3.531 Gitem/s │ 5.306 Gitem/s │ 5.204 Gitem/s │ │ +├─ mul_i16_nonnull 4.039 µs │ 7.909 µs │ 4.104 µs │ 4.153 µs │ 100 │ 100 +│ 8.111 Gitem/s │ 4.142 Gitem/s │ 7.982 Gitem/s │ 7.888 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 29.56 µs │ 26.43 µs │ 26.52 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.108 Gitem/s │ 1.239 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 30.91 µs │ 26.41 µs │ 26.53 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.059 Gitem/s │ 1.24 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nullable 27.27 µs │ 42.76 µs │ 27.38 µs │ 27.63 µs │ 100 │ 100 +│ 1.201 Gitem/s │ 766.1 Mitem/s │ 1.196 Gitem/s │ 1.185 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 26.52 µs │ 23.24 µs │ 23.33 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 1.235 Gitem/s │ 1.409 Gitem/s │ 1.404 Gitem/s │ │ +├─ mul_u8_nonnull 3.279 µs │ 6.329 µs │ 3.329 µs │ 3.378 µs │ 100 │ 100 +│ 9.99 Gitem/s │ 5.176 Gitem/s │ 9.84 Gitem/s │ 9.698 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.489 µs │ 2.599 µs │ 2.615 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 9.389 Gitem/s │ 12.6 Gitem/s │ 12.52 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 9.609 µs │ 6.939 µs │ 7.003 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 3.409 Gitem/s │ 4.721 Gitem/s │ 4.678 Gitem/s │ │ +├─ mul_u64_nonnull 19.15 µs │ 22.82 µs │ 19.21 µs │ 19.33 µs │ 100 │ 100 +│ 1.71 Gitem/s │ 1.435 Gitem/s │ 1.704 Gitem/s │ 1.694 Gitem/s │ │ +╰─ sub_i64_constant 8.119 µs │ 11.08 µs │ 8.249 µs │ 8.293 µs │ 100 │ 100 + 4.035 Gitem/s │ 2.954 Gitem/s │ 3.971 Gitem/s │ 3.951 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md new file mode 100644 index 00000000000..6364ccde44c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-candidate-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.099 µs │ 1.015 ms │ 9.269 µs │ 19.41 µs │ 100 │ 100 +│ 3.6 Gitem/s │ 32.26 Mitem/s │ 3.534 Gitem/s │ 1.687 Gitem/s │ │ +├─ add_i64_nonnull 9.319 µs │ 12.12 µs │ 9.429 µs │ 9.48 µs │ 100 │ 100 +│ 3.515 Gitem/s │ 2.701 Gitem/s │ 3.474 Gitem/s │ 3.456 Gitem/s │ │ +├─ div_i64_nonnull 44.99 µs │ 61.85 µs │ 45.07 µs │ 45.6 µs │ 100 │ 100 +│ 728.1 Mitem/s │ 529.7 Mitem/s │ 726.8 Mitem/s │ 718.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 61.66 µs │ 4.689 µs │ 5.268 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 531.3 Mitem/s │ 6.987 Gitem/s │ 6.219 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 66.06 µs │ 4.279 µs │ 4.931 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 495.9 Mitem/s │ 7.656 Gitem/s │ 6.644 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 71.76 µs │ 18.88 µs │ 19.64 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 456.5 Mitem/s │ 1.734 Gitem/s │ 1.667 Gitem/s │ │ +├─ mul_i32_nonnull 28.21 µs │ 33.28 µs │ 28.34 µs │ 28.48 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 984.3 Mitem/s │ 1.155 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.02 µs │ 233.9 µs │ 29.2 µs │ 31.37 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 140 Mitem/s │ 1.121 Gitem/s │ 1.044 Gitem/s │ │ +├─ mul_i64_nonnull 29.73 µs │ 52.85 µs │ 30.02 µs │ 30.37 µs │ 100 │ 100 +│ 1.101 Gitem/s │ 619.9 Mitem/s │ 1.091 Gitem/s │ 1.078 Gitem/s │ │ +├─ mul_u8_nonnull 3.449 µs │ 14.5 µs │ 3.529 µs │ 3.679 µs │ 100 │ 100 +│ 9.498 Gitem/s │ 2.258 Gitem/s │ 9.283 Gitem/s │ 8.906 Gitem/s │ │ +├─ mul_u16_nonnull 2.349 µs │ 13.44 µs │ 2.419 µs │ 2.529 µs │ 100 │ 100 +│ 13.94 Gitem/s │ 2.436 Gitem/s │ 13.54 Gitem/s │ 12.95 Gitem/s │ │ +├─ mul_u32_nonnull 6.979 µs │ 18.54 µs │ 7.059 µs │ 7.228 µs │ 100 │ 100 +│ 4.694 Gitem/s │ 1.766 Gitem/s │ 4.641 Gitem/s │ 4.532 Gitem/s │ │ +├─ mul_u64_nonnull 30.31 µs │ 42.57 µs │ 30.41 µs │ 30.68 µs │ 100 │ 100 +│ 1.08 Gitem/s │ 769.5 Mitem/s │ 1.077 Gitem/s │ 1.067 Gitem/s │ │ +╰─ sub_i64_constant 8.979 µs │ 32.68 µs │ 9.064 µs │ 9.358 µs │ 100 │ 100 + 3.649 Gitem/s │ 1.002 Gitem/s │ 3.614 Gitem/s │ 3.501 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md new file mode 100644 index 00000000000..6093fe215ee --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-candidate-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.159 µs │ 57.71 µs │ 9.269 µs │ 9.826 µs │ 100 │ 100 +│ 3.577 Gitem/s │ 567.7 Mitem/s │ 3.534 Gitem/s │ 3.334 Gitem/s │ │ +├─ add_i64_nonnull 9.359 µs │ 18.58 µs │ 9.454 µs │ 9.765 µs │ 100 │ 100 +│ 3.5 Gitem/s │ 1.762 Gitem/s │ 3.465 Gitem/s │ 3.355 Gitem/s │ │ +├─ div_i64_nonnull 45.03 µs │ 49.11 µs │ 45.12 µs │ 45.32 µs │ 100 │ 100 +│ 727.5 Mitem/s │ 667.1 Mitem/s │ 726 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 4.649 µs │ 8.699 µs │ 4.729 µs │ 4.798 µs │ 100 │ 100 +│ 7.047 Gitem/s │ 3.766 Gitem/s │ 6.928 Gitem/s │ 6.828 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 7.469 µs │ 4.309 µs │ 4.374 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 4.386 Gitem/s │ 7.603 Gitem/s │ 7.491 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.67 µs │ 18.88 µs │ 18.95 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.444 Gitem/s │ 1.734 Gitem/s │ 1.728 Gitem/s │ │ +├─ mul_i32_nonnull 28.2 µs │ 40.32 µs │ 28.36 µs │ 28.69 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 812.5 Mitem/s │ 1.155 Gitem/s │ 1.141 Gitem/s │ │ +├─ mul_i32_nullable 29.02 µs │ 40.92 µs │ 29.17 µs │ 29.4 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 800.5 Mitem/s │ 1.123 Gitem/s │ 1.114 Gitem/s │ │ +├─ mul_i64_nonnull 29.63 µs │ 33.89 µs │ 30.1 µs │ 30.22 µs │ 100 │ 100 +│ 1.105 Gitem/s │ 966.6 Mitem/s │ 1.088 Gitem/s │ 1.084 Gitem/s │ │ +├─ mul_u8_nonnull 3.489 µs │ 4.839 µs │ 3.559 µs │ 3.576 µs │ 100 │ 100 +│ 9.389 Gitem/s │ 6.77 Gitem/s │ 9.205 Gitem/s │ 9.163 Gitem/s │ │ +├─ mul_u16_nonnull 2.379 µs │ 5.529 µs │ 2.439 µs │ 2.489 µs │ 100 │ 100 +│ 13.76 Gitem/s │ 5.925 Gitem/s │ 13.43 Gitem/s │ 13.16 Gitem/s │ │ +├─ mul_u32_nonnull 6.989 µs │ 8.019 µs │ 7.079 µs │ 7.089 µs │ 100 │ 100 +│ 4.687 Gitem/s │ 4.085 Gitem/s │ 4.628 Gitem/s │ 4.621 Gitem/s │ │ +├─ mul_u64_nonnull 30.35 µs │ 34.77 µs │ 30.43 µs │ 30.57 µs │ 100 │ 100 +│ 1.079 Gitem/s │ 942.1 Mitem/s │ 1.076 Gitem/s │ 1.071 Gitem/s │ │ +╰─ sub_i64_constant 8.949 µs │ 10.5 µs │ 9.089 µs │ 9.109 µs │ 100 │ 100 + 3.661 Gitem/s │ 3.117 Gitem/s │ 3.604 Gitem/s │ 3.597 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md new file mode 100644 index 00000000000..440304b08fe --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-indexed-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.889 µs │ 89.9 µs │ 9.244 µs │ 10.11 µs │ 100 │ 100 +│ 3.686 Gitem/s │ 364.4 Mitem/s │ 3.544 Gitem/s │ 3.24 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 18.28 µs │ 9.399 µs │ 9.605 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 1.791 Gitem/s │ 3.486 Gitem/s │ 3.411 Gitem/s │ │ +├─ div_i64_nonnull 44.92 µs │ 52.66 µs │ 45.07 µs │ 45.39 µs │ 100 │ 100 +│ 729.3 Mitem/s │ 622.1 Mitem/s │ 726.8 Mitem/s │ 721.9 Mitem/s │ │ +├─ mul_i8_nonnull 6.069 µs │ 69.59 µs │ 6.339 µs │ 7.085 µs │ 100 │ 100 +│ 5.398 Gitem/s │ 470.8 Mitem/s │ 5.168 Gitem/s │ 4.624 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 5.439 µs │ 4.259 µs │ 4.274 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 6.023 Gitem/s │ 7.692 Gitem/s │ 7.665 Gitem/s │ │ +├─ mul_i32_constant 32.23 µs │ 36.63 µs │ 32.38 µs │ 32.54 µs │ 100 │ 100 +│ 1.016 Gitem/s │ 894.3 Mitem/s │ 1.011 Gitem/s │ 1.006 Gitem/s │ │ +├─ mul_i32_nonnull 26.52 µs │ 31.34 µs │ 26.58 µs │ 26.69 µs │ 100 │ 100 +│ 1.235 Gitem/s │ 1.045 Gitem/s │ 1.232 Gitem/s │ 1.227 Gitem/s │ │ +├─ mul_i32_nullable 27.31 µs │ 47.02 µs │ 27.41 µs │ 27.83 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 696.7 Mitem/s │ 1.195 Gitem/s │ 1.177 Gitem/s │ │ +├─ mul_i64_nonnull 23.33 µs │ 32.13 µs │ 23.43 µs │ 23.74 µs │ 100 │ 100 +│ 1.403 Gitem/s │ 1.019 Gitem/s │ 1.397 Gitem/s │ 1.379 Gitem/s │ │ +├─ mul_u8_nonnull 3.439 µs │ 61.51 µs │ 3.509 µs │ 4.121 µs │ 100 │ 100 +│ 9.526 Gitem/s │ 532.6 Mitem/s │ 9.336 Gitem/s │ 7.951 Gitem/s │ │ +├─ mul_u16_nonnull 2.699 µs │ 6.979 µs │ 2.769 µs │ 2.813 µs │ 100 │ 100 +│ 12.13 Gitem/s │ 4.694 Gitem/s │ 11.83 Gitem/s │ 11.64 Gitem/s │ │ +├─ mul_u32_nonnull 7.029 µs │ 9.939 µs │ 7.109 µs │ 7.16 µs │ 100 │ 100 +│ 4.661 Gitem/s │ 3.296 Gitem/s │ 4.608 Gitem/s │ 4.576 Gitem/s │ │ +├─ mul_u64_nonnull 19.35 µs │ 22.92 µs │ 19.41 µs │ 19.51 µs │ 100 │ 100 +│ 1.692 Gitem/s │ 1.429 Gitem/s │ 1.687 Gitem/s │ 1.679 Gitem/s │ │ +╰─ sub_i64_constant 9.499 µs │ 12.48 µs │ 9.609 µs │ 9.668 µs │ 100 │ 100 + 3.449 Gitem/s │ 2.623 Gitem/s │ 3.409 Gitem/s │ 3.389 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md new file mode 100644 index 00000000000..aa7ed2c846a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-indexed-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.009 µs │ 93.1 µs │ 9.259 µs │ 10.14 µs │ 100 │ 100 +│ 3.636 Gitem/s │ 351.9 Mitem/s │ 3.538 Gitem/s │ 3.23 Gitem/s │ │ +├─ add_i64_nonnull 9.309 µs │ 12.56 µs │ 9.379 µs │ 9.426 µs │ 100 │ 100 +│ 3.519 Gitem/s │ 2.606 Gitem/s │ 3.493 Gitem/s │ 3.476 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 48.24 µs │ 45.03 µs │ 45.22 µs │ 100 │ 100 +│ 728.6 Mitem/s │ 679.1 Mitem/s │ 727.5 Mitem/s │ 724.5 Mitem/s │ │ +├─ mul_i8_nonnull 5.969 µs │ 50.06 µs │ 6.359 µs │ 6.902 µs │ 100 │ 100 +│ 5.488 Gitem/s │ 654.4 Mitem/s │ 5.152 Gitem/s │ 4.747 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 15.43 µs │ 4.284 µs │ 4.418 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 2.122 Gitem/s │ 7.647 Gitem/s │ 7.415 Gitem/s │ │ +├─ mul_i32_constant 32.25 µs │ 35.51 µs │ 32.39 µs │ 32.51 µs │ 100 │ 100 +│ 1.015 Gitem/s │ 922.5 Mitem/s │ 1.011 Gitem/s │ 1.007 Gitem/s │ │ +├─ mul_i32_nonnull 26.54 µs │ 29.82 µs │ 26.6 µs │ 26.7 µs │ 100 │ 100 +│ 1.234 Gitem/s │ 1.098 Gitem/s │ 1.231 Gitem/s │ 1.226 Gitem/s │ │ +├─ mul_i32_nullable 27.32 µs │ 40.06 µs │ 27.43 µs │ 27.68 µs │ 100 │ 100 +│ 1.198 Gitem/s │ 817.7 Mitem/s │ 1.194 Gitem/s │ 1.183 Gitem/s │ │ +├─ mul_i64_nonnull 23.33 µs │ 26.7 µs │ 23.44 µs │ 23.53 µs │ 100 │ 100 +│ 1.403 Gitem/s │ 1.226 Gitem/s │ 1.397 Gitem/s │ 1.392 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 61.29 µs │ 3.519 µs │ 4.131 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 534.5 Mitem/s │ 9.309 Gitem/s │ 7.931 Gitem/s │ │ +├─ mul_u16_nonnull 2.709 µs │ 6.939 µs │ 2.789 µs │ 2.828 µs │ 100 │ 100 +│ 12.09 Gitem/s │ 4.721 Gitem/s │ 11.74 Gitem/s │ 11.58 Gitem/s │ │ +├─ mul_u32_nonnull 7.039 µs │ 10.56 µs │ 7.119 µs │ 7.18 µs │ 100 │ 100 +│ 4.654 Gitem/s │ 3.1 Gitem/s │ 4.602 Gitem/s │ 4.563 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 23.2 µs │ 19.42 µs │ 19.48 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.411 Gitem/s │ 1.686 Gitem/s │ 1.681 Gitem/s │ │ +╰─ sub_i64_constant 9.509 µs │ 12.69 µs │ 9.619 µs │ 9.668 µs │ 100 │ 100 + 3.445 Gitem/s │ 2.58 Gitem/s │ 3.406 Gitem/s │ 3.389 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md b/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md new file mode 100644 index 00000000000..1296a6b3bb1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md @@ -0,0 +1,139 @@ + + + +# Merge-base production numeric multiply code generation + +Revision: `19f771f2a426103aa7d1bf7153a258bb1bab1e19` + +Command: + +```text +CARGO_TARGET_DIR=/tmp/rowfn-x86.ccCdz5/target-base-codegen \ + cargo rustc -p vortex-array --lib --profile bench -- \ + --emit=llvm-ir,asm -C codegen-units=1 +``` + +Artifacts: + +```text +/tmp/rowfn-x86.ccCdz5/target-base-codegen/release/deps/vortex_array-4e5fe3dd7af89793.ll +/tmp/rowfn-x86.ccCdz5/target-base-codegen/release/deps/vortex_array-4e5fe3dd7af89793.s +``` + +## Production symbols + +```text +i64 execute_checked_typed: 648ec4b22808a2d4 +i64 checked_op_lanes (varying x varying): df33f84e66e75a91 +u64 execute_checked_typed: 8da1eac40a9b0934 +u64 checked_op_lanes (varying x varying): 84edf83ddcd3fe05 +``` + +## i64 hot loop + +Assembly source begins at line 6,698,561 in the `.s` artifact. The loop is +`.LBB6227_8`: + +```asm +movq (%rdi,%rsi,8), %rax +imulq (%r15,%rsi,8) +movq %rax, (%r13,%rsi,8) +incq %rsi +sarq $63, %rax +xorq %rdx, %rax +orq %rax, %rcx +cmpq %rsi, %rbx +jne .LBB6227_8 +``` + +This is one lane per backedge. The one-operand `imulq` produces the signed +128-bit product in `RDX:RAX`; the low half is stored and the high half is +compared with the low-half sign extension through `sarq`/`xorq`. Failure stays +in register `%rcx`. There is no `vector.body`, unroll, or separate remainder. + +## u64 hot loop + +Assembly source begins at line 6,646,461 in the `.s` artifact. The loop is +`.LBB6175_10`: + +```asm +movq (%rbx,%rdi,8), %rax +mulq (%r11,%rdi,8) +movq %rdx, %rsi +movq %rax, -8(%r9,%rdi,8) +movq 8(%rbx,%rdi,8), %rax +mulq 8(%r11,%rdi,8) +orq %rcx, %rsi +movq %rax, (%r9,%rdi,8) +addq $2, %rdi +movq %rdx, %rcx +orq %rsi, %rcx +cmpq %r10, %rdi +jne .LBB6175_10 +``` + +This is scalar unsigned high-half multiplication unrolled by two, followed by +a one-lane remainder when the row count is odd. The two loads, multiplies, and +stores are independent except for the register OR reduction. There is no +`vector.body` in this fast value loop. + +## IR facts + +The all-varying functions are internal and take the source structure through a +`noalias readonly` pointer and return storage through a `noalias writeonly` +pointer. The allocated output stores carry a distinct `!alias.scope` and +`!noalias`; both input loads carry input-side `!noalias`. The second input +length check has become `llvm.assume`, so no panic branch remains in either hot +loop. A slice/assert failure edge exists before the loop at the output-length +validation boundary. + +The u64 IR loop is unrolled by two and reduces two i128 high halves through +scalar `or i64`; it has a one-lane epilogue. The i64 IR loop is scalar and uses +an i128 signed multiply, truncation, arithmetic sign extraction, XOR, and a +loop-carried register OR. Neither fast loop contains a call. + +Both monomorphs have this parameter-level ownership shape (metadata IDs differ +between them): + +```llvm +define internal fastcc void @checked_op_lanes( + ptr noalias writable writeonly %output, + ptr noalias readonly %source, + i64 %valid_rows_tag, + ptr readonly %valid_rows_data) +``` + +The relevant u64 body is structurally: + +```llvm +%failed = phi i64 [ 0, %preheader ], [ %failed_2, %loop ] +%lhs_0 = load i64, ptr %lhs_ptr_0, !noalias !input_scope +%rhs_0 = load i64, ptr %rhs_ptr_0, !noalias !input_scope +%low_0 = mul i64 %rhs_0, %lhs_0 +%wide_0 = mul nuw i128 (zext i64 %rhs_0), (zext i64 %lhs_0) +%high_0 = trunc i128 (lshr i128 %wide_0, 64) to i64 +%failed_1 = or i64 %failed, %high_0 +store i64 %low_0, ptr %output_0, !alias.scope !output_scope, !noalias !output_noalias + +%lhs_1 = load i64, ptr %lhs_ptr_1, !noalias !input_scope +%rhs_1 = load i64, ptr %rhs_ptr_1, !noalias !input_scope +%low_1 = mul i64 %rhs_1, %lhs_1 +%wide_1 = mul nuw i128 (zext i64 %rhs_1), (zext i64 %lhs_1) +%high_1 = trunc i128 (lshr i128 %wide_1, 64) to i64 +%failed_2 = or i64 %failed_1, %high_1 +store i64 %low_1, ptr %output_1, !alias.scope !output_scope, !noalias !output_noalias +``` + +The relevant i64 body is structurally: + +```llvm +%failed = phi i64 [ 0, %preheader ], [ %failed_next, %loop ] +%lhs = load i64, ptr %lhs_ptr, !noalias !input_scope +%rhs = load i64, ptr %rhs_ptr, !noalias !input_scope +%wide = mul nsw i128 (sext i64 %rhs), (sext i64 %lhs) +%low = trunc i128 %wide to i64 +%high = trunc i128 (lshr i128 %wide, 64) to i64 +%discarded_mismatch = xor i64 (ashr i64 %low, 63), %high +%failed_next = or i64 %discarded_mismatch, %failed +store i64 %low, ptr %output, !alias.scope !output_scope, !noalias !output_noalias +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md new file mode 100644 index 00000000000..b93c15fd04e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md @@ -0,0 +1,84 @@ + + + +# `candidate-i64-mul-dense.ll` + +```ll + %_16456.i = phi i64 [ 1, %bb28.lr.ph.i ], [ %_164.i, %bb28.i ] + %iter.sroa.0.055.i = phi i64 [ 0, %bb28.lr.ph.i ], [ %_16456.i, %bb28.i ] + %accumulated.sroa.0.054.i = phi i64 [ 0, %bb28.lr.ph.i ], [ %77, %bb28.i ] + #dbg_value(i64 %iter.sroa.0.055.i, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + #dbg_value(i64 %accumulated.sroa.0.054.i, !561355, !DIExpression(), !561631) + #dbg_value(i64 %iter.sroa.0.055.i, !561378, !DIExpression(), !562028) + #dbg_value(ptr undef, !558643, !DIExpression(), !561458) + #dbg_value(i64 %iter.sroa.0.055.i, !558649, !DIExpression(), !561458) + #dbg_value(ptr poison, !559346, !DIExpression(), !562029) + #dbg_value(i64 %iter.sroa.0.055.i, !559351, !DIExpression(), !562029) + #dbg_value(ptr poison, !559346, !DIExpression(), !562031) + #dbg_value(i64 %iter.sroa.0.055.i, !559351, !DIExpression(), !562031) + #dbg_value(ptr poison, !561841, !DIExpression(), !562033) + #dbg_value(i64 %iter.sroa.0.055.i, !561851, !DIExpression(), !562033) + %75 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.055.i, !dbg !562035 + %_0.i5.i.i = load i64, ptr %75, align 8, !dbg !562035, !noalias !562036, !noundef !23 + %76 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.055.i, !dbg !562039 + %_0.i.i123.i = load i64, ptr %76, align 8, !dbg !562039, !noalias !562036, !noundef !23 + %_3.i126.i = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %iter.sroa.0.055.i, !dbg !562040 + #dbg_value(ptr poison, !561857, !DIExpression(), !562041) + #dbg_value(ptr poison, !561867, !DIExpression(), !562041) + #dbg_value(i64 %_0.i.i123.i, !561868, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !562041) + #dbg_value(i64 %_0.i5.i.i, !561868, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !562041) + #dbg_value(ptr %_3.i126.i, !561863, !DIExpression(), !562041) + #dbg_value(i64 %_0.i.i123.i, !561864, !DIExpression(), !562043) + #dbg_value(i64 %_0.i.i123.i, !561873, !DIExpression(), !562044) + #dbg_value(i64 %_0.i5.i.i, !561866, !DIExpression(), !562043) + #dbg_value(i64 %_0.i5.i.i, !561880, !DIExpression(), !562044) + #dbg_value(ptr %_3.i126.i, !561879, !DIExpression(), !562044) + #dbg_value(ptr %_3.i126.i, !561886, !DIExpression(), !562046) + #dbg_value(i64 %_0.i.i123.i, !561892, !DIExpression(), !562048) + #dbg_value(i64 %_0.i5.i.i, !561901, !DIExpression(), !562048) + #dbg_value(i64 %_0.i.i123.i, !561904, !DIExpression(), !562050) + #dbg_value(i64 %_0.i.i123.i, !561910, !DIExpression(), !562052) + #dbg_value(i64 %_0.i5.i.i, !561907, !DIExpression(), !562050) + #dbg_value(i64 %_0.i5.i.i, !561913, !DIExpression(), !562052) + %_0.i.i128.i = mul i64 %_0.i.i123.i, %_0.i5.i.i, !dbg !562054 + #dbg_value(i64 %_0.i.i123.i, !561917, !DIExpression(), !562055) + #dbg_value(i64 %_0.i.i123.i, !561923, !DIExpression(), !562057) + #dbg_value(i64 %_0.i5.i.i, !561922, !DIExpression(), !562055) + #dbg_value(i64 %_0.i5.i.i, !561925, !DIExpression(), !562057) + %_4.i1.i.i = sext i64 %_0.i.i123.i to i128, !dbg !562058 + %_5.i.i.i = sext i64 %_0.i5.i.i to i128, !dbg !562059 + %wide.i.i.i = mul nsw i128 %_4.i1.i.i, %_5.i.i.i, !dbg !562058 + #dbg_value(i128 %wide.i.i.i, !561926, !DIExpression(), !562060) + %kept.i.i.i = trunc i128 %wide.i.i.i to i64, !dbg !562061 + #dbg_value(i64 %kept.i.i.i, !561928, !DIExpression(), !562062) + %_8.i.i.i = lshr i128 %wide.i.i.i, 64, !dbg !562063 + %discarded.i.i.i = trunc nuw i128 %_8.i.i.i to i64, !dbg !562064 + #dbg_value(i64 %discarded.i.i.i, !561930, !DIExpression(), !562065) + %_10.i.i.i = ashr i64 %kept.i.i.i, 63, !dbg !562066 + %_9.i.i.i = xor i64 %_10.i.i.i, %discarded.i.i.i, !dbg !562067 + #dbg_value(i64 %_0.i.i128.i, !561881, !DIExpression(), !562068) + #dbg_value(i64 %_0.i.i128.i, !561889, !DIExpression(), !562046) + #dbg_value(i64 %_9.i.i.i, !561883, !DIExpression(), !562068) + store i64 %_0.i.i128.i, ptr %_3.i126.i, align 8, !dbg !562069, !alias.scope !562070, !noalias !561484 + #dbg_value(i64 %_9.i.i.i, !561469, !DIExpression(), !561472) + #dbg_value(ptr undef, !561463, !DIExpression(), !561472) + %77 = or i64 %_9.i.i.i, %accumulated.sroa.0.054.i, !dbg !562073 + #dbg_value(i64 %77, !561355, !DIExpression(), !561631) + #dbg_value(i64 %_16456.i, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + #dbg_value(ptr undef, !561426, !DIExpression(), !561451) + #dbg_value(ptr undef, !561414, !DIExpression(), !561447) + #dbg_value(ptr undef, !561430, !DIExpression(), !561452) + #dbg_value(ptr poison, !561433, !DIExpression(), !561452) + %_164.i = add i64 %_16456.i, 1, !dbg !562074 + #dbg_value(i64 poison, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + %exitcond.not.i = icmp eq i64 %_16456.i, %4, !dbg !561962 + br i1 %exitcond.not.i, label %bb54.i, label %bb28.i, !dbg !561963 + +bb59.i: ; preds = %bb24.i + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(48) %71, ptr noundef nonnull align 8 dereferenceable(48) %_59.i, i64 48, i1 false), !dbg !562075, !noalias !561484 + call void @llvm.lifetime.end.p0(i64 48, ptr nonnull %_59.i), !dbg !561556, !noalias !561565 + %_49.sroa.4.0..sroa_idx.i = getelementptr inbounds nuw i8, ptr %_0, i64 16, !dbg !561964 + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(24) %_49.sroa.4.0..sroa_idx.i, ptr noundef nonnull align 8 dereferenceable(24) %_57.i, i64 24, i1 false), !dbg !561556, !noalias !561605 + call void @llvm.lifetime.end.p0(i64 24, ptr nonnull %_57.i), !dbg !561556, !noalias !561565 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md new file mode 100644 index 00000000000..9d2c024bac0 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md @@ -0,0 +1,69 @@ + + + +# `candidate-i64-mul-dense.s` + +```s + movq -376(%rbp), %r15 +.Ltmp108913: + .loc 524 86 32 + testq %r15, %r15 + .loc 524 86 16 is_stmt 0 + je .LBB1673_26 +.Ltmp108914: + .loc 563 318 19 is_stmt 1 + xorq %r14, %rdi +.Ltmp108915: + .loc 563 0 19 is_stmt 0 + xorq %r14, %r10 +.Ltmp108916: + .loc 524 88 17 is_stmt 1 + orq %rdi, %r10 +.Ltmp108917: + jne .LBB1673_42 +.Ltmp108918: + .loc 182 1904 50 + testq %r14, %r14 +.Ltmp108919: + .loc 524 92 26 + je .LBB1673_41 +.Ltmp108920: + .loc 524 0 26 is_stmt 0 + movq -320(%rbp), %rsi +.Ltmp108921: + xorl %edi, %edi + xorl %ecx, %ecx +.Ltmp108922: + .p2align 4 +.LBB1673_25: + .loc 564 62 9 is_stmt 1 + movq (%r15,%rdi,8), %rax +.Ltmp108923: + .loc 565 193 24 + imulq (%rsi,%rdi,8) +.Ltmp108924: + .loc 207 475 9 + movq %rax, (%r9,%rdi,8) +.Ltmp108925: + .loc 565 197 26 + sarq $63, %rax +.Ltmp108926: + .loc 565 197 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp108927: + .loc 566 109 21 is_stmt 1 + orq %rax, %rcx +.Ltmp108928: + .loc 182 1904 50 + incq %rdi +.Ltmp108929: + cmpq %rdi, %r14 + jne .LBB1673_25 + jmp .LBB1673_62 +.Ltmp108930: +.LBB1673_26: + .loc 563 90 47 + cmpq %r14, %rdi + sete %cl + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md new file mode 100644 index 00000000000..05db6cccb49 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md @@ -0,0 +1,137 @@ + + + +# `candidate-u64-mul-dense.ll` + +```ll +terminate.i81.i: ; preds = %cleanup.i80.i + %114 = landingpad { ptr, i32 } + filter [0 x ptr] zeroinitializer +; call core::panicking::panic_in_cleanup + call void @_ZN4core9panicking16panic_in_cleanup17h8f68387bb6cbbf54E() #88, !dbg !588172, !noalias !587626 + unreachable, !dbg !588172 + +bb28.i: ; preds = %bb28.i, %bb28.lr.ph.i.new + %_16456.i = phi i64 [ 1, %bb28.lr.ph.i.new ], [ %_164.i.1, %bb28.i ] + %iter.sroa.0.055.i = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %_164.i, %bb28.i ] + %accumulated.sroa.0.054.i = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %120, %bb28.i ] + %niter = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %niter.next.1, %bb28.i ] + #dbg_value(i64 %iter.sroa.0.055.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %accumulated.sroa.0.054.i, !587504, !DIExpression(), !587773) + #dbg_value(i64 %iter.sroa.0.055.i, !587527, !DIExpression(), !588173) + #dbg_value(ptr undef, !579662, !DIExpression(), !587607) + #dbg_value(i64 %iter.sroa.0.055.i, !579668, !DIExpression(), !587607) + #dbg_value(ptr poison, !580362, !DIExpression(), !588174) + #dbg_value(i64 %iter.sroa.0.055.i, !580367, !DIExpression(), !588174) + #dbg_value(ptr poison, !580362, !DIExpression(), !588176) + #dbg_value(i64 %iter.sroa.0.055.i, !580367, !DIExpression(), !588176) + #dbg_value(ptr poison, !587985, !DIExpression(), !588178) + #dbg_value(i64 %iter.sroa.0.055.i, !587986, !DIExpression(), !588178) + %115 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.055.i, !dbg !588180 + %_0.i5.i.i = load i64, ptr %115, align 8, !dbg !588180, !noalias !588181, !noundef !23 + %116 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.055.i, !dbg !588184 + %_0.i.i123.i = load i64, ptr %116, align 8, !dbg !588184, !noalias !588181, !noundef !23 + %_3.i126.i = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %iter.sroa.0.055.i, !dbg !588185 + #dbg_value(ptr poison, !588025, !DIExpression(), !588186) + #dbg_value(ptr poison, !588026, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i, !588027, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588186) + #dbg_value(i64 %_0.i5.i.i, !588027, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !588186) + #dbg_value(ptr %_3.i126.i, !588022, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i, !588023, !DIExpression(), !588188) + #dbg_value(i64 %_0.i.i123.i, !588010, !DIExpression(), !588189) + #dbg_value(i64 %_0.i5.i.i, !588024, !DIExpression(), !588188) + #dbg_value(i64 %_0.i5.i.i, !588011, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i, !588009, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i, !588036, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i123.i, !588001, !DIExpression(), !588193) + #dbg_value(i64 %_0.i5.i.i, !588002, !DIExpression(), !588193) + #dbg_value(i64 %_0.i.i123.i, !588067, !DIExpression(), !588195) + #dbg_value(i64 %_0.i.i123.i, !588073, !DIExpression(), !588197) + #dbg_value(i64 %_0.i5.i.i, !588070, !DIExpression(), !588195) + #dbg_value(i64 %_0.i5.i.i, !588076, !DIExpression(), !588197) + %_0.i3.i.i = mul i64 %_0.i.i123.i, %_0.i5.i.i, !dbg !588199 + #dbg_value(i64 %_0.i.i123.i, !587992, !DIExpression(), !588200) + #dbg_value(i64 %_0.i.i123.i, !587994, !DIExpression(), !588202) + #dbg_value(i64 %_0.i5.i.i, !587993, !DIExpression(), !588200) + #dbg_value(i64 %_0.i5.i.i, !587995, !DIExpression(), !588202) + %_5.i.i.i = zext i64 %_0.i.i123.i to i128, !dbg !588203 + %_6.i.i.i = zext i64 %_0.i5.i.i to i128, !dbg !588204 + %_4.i1.i.i = mul nuw i128 %_5.i.i.i, %_6.i.i.i, !dbg !588205 + %_3.i2.i.i = lshr i128 %_4.i1.i.i, 64, !dbg !588206 + %_0.i.i128.i = trunc nuw i128 %_3.i2.i.i to i64, !dbg !588207 + #dbg_value(i64 %_0.i3.i.i, !588012, !DIExpression(), !588208) + #dbg_value(i64 %_0.i3.i.i, !588037, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i128.i, !588014, !DIExpression(), !588208) + store i64 %_0.i3.i.i, ptr %_3.i126.i, align 8, !dbg !588209, !alias.scope !588210, !noalias !587626 + #dbg_value(i64 %_0.i.i128.i, !561469, !DIExpression(), !587614) + #dbg_value(ptr undef, !561463, !DIExpression(), !587614) + %117 = or i64 %accumulated.sroa.0.054.i, %_0.i.i128.i, !dbg !588213 + #dbg_value(i64 %117, !587504, !DIExpression(), !587773) + #dbg_value(i64 %_16456.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(ptr undef, !587575, !DIExpression(), !587600) + #dbg_value(ptr undef, !587563, !DIExpression(), !587596) + #dbg_value(ptr undef, !587579, !DIExpression(), !587601) + #dbg_value(ptr poison, !587582, !DIExpression(), !587601) + %_164.i = add i64 %_16456.i, 1, !dbg !588214 + #dbg_value(i64 poison, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %_16456.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %_16456.i, !587527, !DIExpression(), !588173) + #dbg_value(i64 %_16456.i, !579668, !DIExpression(), !587607) + #dbg_value(i64 %_16456.i, !580367, !DIExpression(), !588174) + #dbg_value(i64 %_16456.i, !580367, !DIExpression(), !588176) + #dbg_value(i64 %_16456.i, !587986, !DIExpression(), !588178) + %118 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_16456.i, !dbg !588180 + %_0.i5.i.i.1 = load i64, ptr %118, align 8, !dbg !588180, !noalias !588181, !noundef !23 + %119 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_16456.i, !dbg !588184 + %_0.i.i123.i.1 = load i64, ptr %119, align 8, !dbg !588184, !noalias !588181, !noundef !23 + %_3.i126.i.1 = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %_16456.i, !dbg !588185 + #dbg_value(i64 %_0.i.i123.i.1, !588027, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588186) + #dbg_value(i64 %_0.i5.i.i.1, !588027, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !588186) + #dbg_value(ptr %_3.i126.i.1, !588022, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i.1, !588023, !DIExpression(), !588188) + #dbg_value(i64 %_0.i.i123.i.1, !588010, !DIExpression(), !588189) + #dbg_value(i64 %_0.i5.i.i.1, !588024, !DIExpression(), !588188) + #dbg_value(i64 %_0.i5.i.i.1, !588011, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i.1, !588009, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i.1, !588036, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i123.i.1, !588001, !DIExpression(), !588193) + #dbg_value(i64 %_0.i5.i.i.1, !588002, !DIExpression(), !588193) + #dbg_value(i64 %_0.i.i123.i.1, !588067, !DIExpression(), !588195) + #dbg_value(i64 %_0.i.i123.i.1, !588073, !DIExpression(), !588197) + #dbg_value(i64 %_0.i5.i.i.1, !588070, !DIExpression(), !588195) + #dbg_value(i64 %_0.i5.i.i.1, !588076, !DIExpression(), !588197) + %_0.i3.i.i.1 = mul i64 %_0.i.i123.i.1, %_0.i5.i.i.1, !dbg !588199 + #dbg_value(i64 %_0.i.i123.i.1, !587992, !DIExpression(), !588200) + #dbg_value(i64 %_0.i.i123.i.1, !587994, !DIExpression(), !588202) + #dbg_value(i64 %_0.i5.i.i.1, !587993, !DIExpression(), !588200) + #dbg_value(i64 %_0.i5.i.i.1, !587995, !DIExpression(), !588202) + %_5.i.i.i.1 = zext i64 %_0.i.i123.i.1 to i128, !dbg !588203 + %_6.i.i.i.1 = zext i64 %_0.i5.i.i.1 to i128, !dbg !588204 + %_4.i1.i.i.1 = mul nuw i128 %_5.i.i.i.1, %_6.i.i.i.1, !dbg !588205 + %_3.i2.i.i.1 = lshr i128 %_4.i1.i.i.1, 64, !dbg !588206 + %_0.i.i128.i.1 = trunc nuw i128 %_3.i2.i.i.1 to i64, !dbg !588207 + #dbg_value(i64 %_0.i3.i.i.1, !588012, !DIExpression(), !588208) + #dbg_value(i64 %_0.i3.i.i.1, !588037, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i128.i.1, !588014, !DIExpression(), !588208) + store i64 %_0.i3.i.i.1, ptr %_3.i126.i.1, align 8, !dbg !588209, !alias.scope !588210, !noalias !587626 + #dbg_value(i64 %_0.i.i128.i.1, !561469, !DIExpression(), !587614) + %120 = or i64 %117, %_0.i.i128.i.1, !dbg !588213 + #dbg_value(i64 %120, !587504, !DIExpression(), !587773) + #dbg_value(i64 %_164.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + %_164.i.1 = add i64 %_16456.i, 2, !dbg !588214 + #dbg_value(i64 poison, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + %niter.next.1 = add i64 %niter, 2, !dbg !588108 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !588108 + br i1 %niter.ncmp.1, label %bb54.i.loopexit135.unr-lcssa, label %bb28.i, !dbg !588108 + +bb59.i: ; preds = %bb24.i + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(48) %111, ptr noundef nonnull align 8 dereferenceable(48) %_59.i, i64 48, i1 false), !dbg !588215, !noalias !587626 + call void @llvm.lifetime.end.p0(i64 48, ptr nonnull %_59.i), !dbg !587698, !noalias !587707 + %_49.sroa.4.0..sroa_idx.i = getelementptr inbounds nuw i8, ptr %_0, i64 16, !dbg !588109 + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(24) %_49.sroa.4.0..sroa_idx.i, ptr noundef nonnull align 8 dereferenceable(24) %_57.i, i64 24, i1 false), !dbg !587698, !noalias !587747 + call void @llvm.lifetime.end.p0(i64 24, ptr nonnull %_57.i), !dbg !587698, !noalias !587707 + br label %bb61.i, !dbg !587940 + +bb39.i: ; preds = %bb2.i109.i, %"_ZN12vortex_array9scalar_fn3row7element5tuple18ArgColumn$LT$T$GT$14addresses_rows17h8cb4442712b37c9aE.exit.i.i" + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md new file mode 100644 index 00000000000..0676296c7a7 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md @@ -0,0 +1,70 @@ + + + +# `candidate-u64-mul-dense.s` + +```s + .loc 524 92 26 + andq $-2, %r15 + leaq (%r12,%rdx), %r11 + addq $8, %r11 + xorl %ecx, %ecx + xorl %r10d, %r10d +.Ltmp117915: +.LBB1693_70: + .loc 564 62 9 + movq (%r14,%r10,8), %rax +.Ltmp117916: + .loc 565 175 44 + mulq (%r8,%r10,8) +.Ltmp117917: + movq %rdx, %rdi +.Ltmp117918: + .loc 207 475 9 + movq %rax, -8(%r11,%r10,8) +.Ltmp117919: + .loc 564 62 9 + movq 8(%r14,%r10,8), %rax +.Ltmp117920: + .loc 565 175 44 + mulq 8(%r8,%r10,8) +.Ltmp117921: + .loc 566 109 21 + orq %rcx, %rdi +.Ltmp117922: + .loc 207 475 9 + movq %rax, (%r11,%r10,8) +.Ltmp117923: + .loc 565 175 44 + movq %rdx, %rcx +.Ltmp117924: + .loc 566 109 21 + orq %rdi, %rcx +.Ltmp117925: + .loc 524 92 26 + addq $2, %r10 + cmpq %r10, %r15 + jne .LBB1693_70 +.Ltmp117926: +.LBB1693_71: + testb $1, %sil + je .LBB1693_87 +.Ltmp117927: + .loc 564 62 9 + movq (%r14,%r10,8), %rax +.Ltmp117928: + .loc 565 175 44 + mulq (%r8,%r10,8) +.Ltmp117929: +.LBB1693_73: + .loc 207 475 9 + movq %rax, (%r9,%r10,8) +.Ltmp117930: + .loc 566 109 21 + orq %rdx, %rcx +.Ltmp117931: + .loc 566 0 21 is_stmt 0 + jmp .LBB1693_87 +.Ltmp117932: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md new file mode 100644 index 00000000000..7513df3b465 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md @@ -0,0 +1,86 @@ + + + +# `final-i32-mul-constant.ll` + +```ll +bb27.preheader.i.split.us: ; preds = %bb27.preheader.i + br i1 %_3.i5.not.i.i, label %panic.i5.i5.i.invoke.i, label %bb27.i.us.preheader + +bb27.i.us.preheader: ; preds = %bb27.preheader.i.split.us + %57 = add nuw nsw i64 %len3.i.i.i, 1, !dbg !566996 + br label %bb27.i.us, !dbg !566996 + +bb27.i.us: ; preds = %bb27.i.us.preheader, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" + %_15854.i.us = phi i64 [ %_158.i.us, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ 1, %bb27.i.us.preheader ] + %iter.sroa.0.053.i.us = phi i64 [ %_15854.i.us, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ 0, %bb27.i.us.preheader ] + %accumulated.sroa.0.052.i.us = phi i1 [ %60, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ false, %bb27.i.us.preheader ] + #dbg_value(i64 %iter.sroa.0.053.i.us, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + #dbg_value(i64 %iter.sroa.0.053.i.us, !566732, !DIExpression(), !567057) + #dbg_value(ptr %columns.i, !545259, !DIExpression(), !567058) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545260, !DIExpression(), !567058) + #dbg_value(ptr %columns.i, !545249, !DIExpression(), !567059) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545250, !DIExpression(), !567059) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567060) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567062) + #dbg_value(ptr %columns.i, !545251, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567063) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545125, !DIExpression(), !567062) + %exitcond35.not = icmp eq i64 %_15854.i.us, %57, !dbg !566996 + br i1 %exitcond35.not, label %panic.i5.i5.i.invoke.i, label %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us", !dbg !566996 + +"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us": ; preds = %bb27.i.us + #dbg_value(ptr %columns.i, !545251, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567063) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567062) + %58 = getelementptr inbounds nuw i32, ptr %data.i6.i.i.i, i64 %iter.sroa.0.053.i.us, !dbg !566996 + %_0.sroa.0.0.i.i.i.us = load i32, ptr %58, align 4, !dbg !567000, !noalias !566784, !noundef !23 + #dbg_value(ptr %14, !545249, !DIExpression(), !567064) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545250, !DIExpression(), !567064) + #dbg_value(ptr %14, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567065) + #dbg_value(ptr %14, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567067) + #dbg_value(ptr %14, !545252, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567068) + #dbg_value(i64 0, !545125, !DIExpression(), !567065) + %_0.sroa.0.0.i9.i.i.us = load i32, ptr %data.i6.i7.i.i, align 4, !dbg !567005, !noalias !566784, !noundef !23 + #dbg_value(ptr poison, !567031, !DIExpression(), !567069) + #dbg_value(ptr poison, !567032, !DIExpression(), !567069) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567033, !DIExpression(DW_OP_LLVM_fragment, 0, 32), !567069) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567033, !DIExpression(DW_OP_LLVM_fragment, 32, 32), !567069) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567029, !DIExpression(), !567070) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567030, !DIExpression(), !567070) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567020, !DIExpression(), !567071) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567021, !DIExpression(), !567071) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567015, !DIExpression(), !567072) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567010, !DIExpression(), !567073) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567016, !DIExpression(), !567072) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567011, !DIExpression(), !567073) + %_0.i.i160.i.us = mul i32 %_0.sroa.0.0.i9.i.i.us, %_0.sroa.0.0.i.i.i.us, !dbg !567007 + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567039, !DIExpression(), !567074) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567041, !DIExpression(), !567075) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567040, !DIExpression(), !567074) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567042, !DIExpression(), !567075) + %_4.i1.i.i.us = sext i32 %_0.sroa.0.0.i.i.i.us to i64, !dbg !567035 + %_5.i.i.i.us = sext i32 %_0.sroa.0.0.i9.i.i.us to i64, !dbg !567046 + %product.i.i.i.us = mul nsw i64 %_5.i.i.i.us, %_4.i1.i.i.us, !dbg !567035 + #dbg_value(i64 %product.i.i.i.us, !567043, !DIExpression(), !567076) + %59 = add nsw i64 %product.i.i.i.us, -2147483648, !dbg !567047 + %_0.sroa.0.0.i.i161.i.us = icmp ult i64 %59, -4294967296, !dbg !567047 + #dbg_value(i1 %_0.sroa.0.0.i.i161.i.us, !566736, !DIExpression(DW_OP_LLVM_convert, 1, DW_ATE_unsigned, DW_OP_LLVM_convert, 8, DW_ATE_unsigned, DW_OP_stack_value), !567077) + #dbg_value(i32 %_0.i.i160.i.us, !566734, !DIExpression(), !567077) + %self34.i.us = getelementptr inbounds nuw i32, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.053.i.us, !dbg !567048 + #dbg_value(ptr %self34.i.us, !567052, !DIExpression(), !567078) + #dbg_value(i32 %_0.i.i160.i.us, !567053, !DIExpression(), !567078) + store i32 %_0.i.i160.i.us, ptr %self34.i.us, align 4, !dbg !567049, !noalias !566784 + #dbg_value(ptr undef, !541560, !DIExpression(), !566763) + #dbg_value(i1 %_0.sroa.0.0.i.i161.i.us, !541568, !DIExpression(DW_OP_LLVM_convert, 1, DW_ATE_unsigned, DW_OP_LLVM_convert, 8, DW_ATE_unsigned, DW_OP_stack_value), !566763) + %60 = or i1 %accumulated.sroa.0.052.i.us, %_0.sroa.0.0.i.i161.i.us, !dbg !567055 + #dbg_value(i8 poison, !566728, !DIExpression(), !566992) + #dbg_value(i64 %_15854.i.us, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + #dbg_value(ptr undef, !566753, !DIExpression(), !566756) + #dbg_value(ptr undef, !566744, !DIExpression(), !566749) + #dbg_value(ptr undef, !566757, !DIExpression(), !566761) + #dbg_value(ptr poison, !566760, !DIExpression(), !566761) + %_158.i.us = add i64 %_15854.i.us, 1, !dbg !567079 + #dbg_value(i64 poison, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + %exitcond.not.i.us = icmp eq i64 %_15854.i.us, %len3.i.i.i, !dbg !566994 + br i1 %exitcond.not.i.us, label %bb33.i, label %bb27.i.us, !dbg !566995 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md new file mode 100644 index 00000000000..f82f250cdd1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md @@ -0,0 +1,49 @@ + + + +# `final-i32-mul-constant.s` + +```s +.LBB1677_31: + .loc 564 47 9 is_stmt 1 + cmpq %rdi, %rdx + je .LBB1677_89 +.Ltmp110238: + .loc 564 47 9 is_stmt 0 + movslq (%r13,%rdi,4), %rcx +.Ltmp110239: + .loc 564 47 9 + movslq (%r12), %r8 +.Ltmp110240: + .loc 462 2133 13 is_stmt 1 + movl %r8d, %r10d + imull %ecx, %r10d +.Ltmp110241: + .loc 565 185 27 + imulq %rcx, %r8 +.Ltmp110242: + .loc 565 185 35 is_stmt 0 + addq $-2147483648, %r8 +.Ltmp110243: + cmpq %rax, %r8 + setb %cl +.Ltmp110244: + .loc 565 0 35 + movq -48(%rbp), %r8 +.Ltmp110245: + .loc 207 475 9 is_stmt 1 + movl %r10d, (%r8,%rdi,4) +.Ltmp110246: + .loc 566 821 53 + orb %cl, %r9b +.Ltmp110247: + .loc 182 1904 50 + incq %rdi +.Ltmp110248: + cmpq %rdi, %rdx +.Ltmp110249: + .loc 562 124 26 + jne .LBB1677_31 + jmp .LBB1677_64 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md new file mode 100644 index 00000000000..6b28482775c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md @@ -0,0 +1,96 @@ + + + +# `final-i64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb11.i, %bb15.i.i + %iter.sroa.0.012.i.i = phi i64 [ %_36.i.i, %bb15.i.i ], [ 0, %bb11.i ] + %failed.sroa.0.011.i.i = phi i64 [ %79, %bb15.i.i ], [ 0, %bb11.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !564425, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564479) + #dbg_value(i64 %failed.sroa.0.011.i.i, !564423, !DIExpression(), !564478) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564463, !DIExpression(), !564694) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564456, !DIExpression(), !564457) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564473, !DIExpression(), !564474) + %_36.i.i = add nuw i64 %iter.sroa.0.012.i.i, 1, !dbg !564695 + #dbg_value(i64 %_36.i.i, !564425, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564479) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564427, !DIExpression(), !564696) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564450, !DIExpression(), !564451) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564697, !DIExpression(), !564701) + #dbg_value(ptr undef, !547076, !DIExpression(), !564445) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547082, !DIExpression(), !564445) + #dbg_value(ptr poison, !547154, !DIExpression(), !564703) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547155, !DIExpression(), !564703) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547147, !DIExpression(), !564705) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547139, !DIExpression(), !564707) + #dbg_value(ptr %column.val.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564705) + #dbg_value(ptr %column.val.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564707) + #dbg_value(i64 %len3.i.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564705) + #dbg_value(i64 %len3.i.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564707) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564709 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !564710, !noalias !564711, !noundef !23 + #dbg_value(ptr poison, !547154, !DIExpression(), !564715) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547155, !DIExpression(), !564715) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547147, !DIExpression(), !564717) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547139, !DIExpression(), !564719) + #dbg_value(ptr %column5.val.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564717) + #dbg_value(ptr %column5.val.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564719) + #dbg_value(i64 %len3.i.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564717) + #dbg_value(i64 %len3.i.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564719) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !564721 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !564722 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564723 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !564724, !noalias !564711, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !564429, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564725) + #dbg_value(i64 %_0.i5.i.i.i, !564429, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564725) + #dbg_value(i64 %_0.i.i.i.i, !564726, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564734) + #dbg_value(i64 %_0.i5.i.i.i, !564726, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564734) + #dbg_value(ptr poison, !564315, !DIExpression(), !564736) + #dbg_value(ptr poison, !564316, !DIExpression(), !564736) + #dbg_value(i64 %_0.i.i.i.i, !564317, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564736) + #dbg_value(i64 %_0.i5.i.i.i, !564317, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564736) + #dbg_value(i64 %_0.i.i.i.i, !564313, !DIExpression(), !564738) + #dbg_value(i64 %_0.i5.i.i.i, !564314, !DIExpression(), !564738) + #dbg_value(i64 %_0.i.i.i.i, !564304, !DIExpression(), !564739) + #dbg_value(i64 %_0.i5.i.i.i, !564305, !DIExpression(), !564739) + #dbg_value(i64 %_0.i.i.i.i, !564293, !DIExpression(), !564741) + #dbg_value(i64 %_0.i.i.i.i, !564288, !DIExpression(), !564743) + #dbg_value(i64 %_0.i5.i.i.i, !564294, !DIExpression(), !564741) + #dbg_value(i64 %_0.i5.i.i.i, !564289, !DIExpression(), !564743) + %_0.i.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !564745 + #dbg_value(i64 %_0.i.i.i.i, !564325, !DIExpression(), !564746) + #dbg_value(i64 %_0.i.i.i.i, !564327, !DIExpression(), !564748) + #dbg_value(i64 %_0.i5.i.i.i, !564326, !DIExpression(), !564746) + #dbg_value(i64 %_0.i5.i.i.i, !564328, !DIExpression(), !564748) + %_4.i1.i.i.i.i = sext i64 %_0.i.i.i.i to i128, !dbg !564749 + %_5.i.i.i.i.i = sext i64 %_0.i5.i.i.i to i128, !dbg !564750 + %wide.i.i.i.i.i = mul nsw i128 %_5.i.i.i.i.i, %_4.i1.i.i.i.i, !dbg !564749 + #dbg_value(i128 %wide.i.i.i.i.i, !564329, !DIExpression(), !564751) + %kept.i.i.i.i.i = trunc i128 %wide.i.i.i.i.i to i64, !dbg !564752 + #dbg_value(i64 %kept.i.i.i.i.i, !564331, !DIExpression(), !564753) + %_8.i.i.i.i.i = lshr i128 %wide.i.i.i.i.i, 64, !dbg !564754 + %discarded.i.i.i.i.i = trunc nuw i128 %_8.i.i.i.i.i to i64, !dbg !564755 + #dbg_value(i64 %discarded.i.i.i.i.i, !564333, !DIExpression(), !564756) + %_10.i.i.i.i.i = ashr i64 %kept.i.i.i.i.i, 63, !dbg !564757 + %_9.i.i.i.i.i = xor i64 %_10.i.i.i.i.i, %discarded.i.i.i.i.i, !dbg !564758 + #dbg_value(i64 poison, !564431, !DIExpression(), !564759) + #dbg_value(i64 %_9.i.i.i.i.i, !564433, !DIExpression(), !564759) + #dbg_value(ptr undef, !564034, !DIExpression(), !564443) + #dbg_value(i64 %_9.i.i.i.i.i, !564040, !DIExpression(), !564443) + %79 = or i64 %_9.i.i.i.i.i, %failed.sroa.0.011.i.i, !dbg !564760 + #dbg_value(i64 %79, !564423, !DIExpression(), !564478) + #dbg_value(i64 %_0.i.i.i.i.i, !564431, !DIExpression(), !564759) + #dbg_value(ptr %_4.sroa.10.0.i.i, !564700, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564701) + #dbg_value(i64 %index.i, !564700, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564701) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564761 + #dbg_value(ptr %self4.i.i, !564762, !DIExpression(), !564766) + #dbg_value(i64 %_0.i.i.i.i.i, !564765, !DIExpression(), !564766) + store i64 %_0.i.i.i.i.i, ptr %self4.i.i, align 8, !dbg !564768, !alias.scope !564439, !noalias !564769 + #dbg_value(ptr undef, !564467, !DIExpression(), !564480) + #dbg_value(ptr undef, !564462, !DIExpression(), !564481) + #dbg_value(ptr undef, !564482, !DIExpression(), !564486) + #dbg_value(ptr poison, !564485, !DIExpression(), !564486) + %exitcond.not.i.i = icmp eq i64 %_36.i.i, %len3.i4.i.i.fr, !dbg !564770 + br i1 %exitcond.not.i.i, label %bb33.i, label %bb15.i.i, !dbg !564488 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md new file mode 100644 index 00000000000..56206a6ec40 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md @@ -0,0 +1,34 @@ + + + +# `final-i64-mul-dense.s` + +```s +.LBB1675_25: + .loc 567 39 18 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp109477: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp109478: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp109479: + .loc 156 717 17 + incq %rsi +.Ltmp109480: + .loc 565 198 26 + sarq $63, %rax +.Ltmp109481: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp109482: + .loc 566 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp109483: + .loc 182 1904 50 + cmpq %rsi, %r9 + jne .LBB1675_25 + jmp .LBB1675_60 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md new file mode 100644 index 00000000000..4e40cfb8f2c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md @@ -0,0 +1,322 @@ + + + +# `final-u64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb15.i.i, %bb15.i.i.preheader.new + %iter.sroa.0.012.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %116, %bb15.i.i ] + %niter = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %niter.next.1, %bb15.i.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %failed.sroa.0.011.i.i, !569873, !DIExpression(), !569927) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569912, !DIExpression(), !570143) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569905, !DIExpression(), !569906) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569922, !DIExpression(), !569923) + %_36.i.i = or disjoint i64 %iter.sroa.0.012.i.i, 1, !dbg !570144 + #dbg_value(i64 %_36.i.i, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569877, !DIExpression(), !570145) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569899, !DIExpression(), !569900) + #dbg_value(i64 %iter.sroa.0.012.i.i, !570146, !DIExpression(), !570150) + #dbg_value(ptr undef, !551471, !DIExpression(), !569894) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551477, !DIExpression(), !569894) + #dbg_value(ptr poison, !551549, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551550, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551542, !DIExpression(), !570154) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570158 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(ptr poison, !551549, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551550, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551542, !DIExpression(), !570166) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !570171 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570172 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(ptr poison, !569758, !DIExpression(), !570185) + #dbg_value(ptr poison, !569759, !DIExpression(), !570185) + #dbg_value(i64 %_0.i.i.i.i, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i = zext i64 %_0.i.i.i.i to i128, !dbg !570198 + %_6.i.i.i.i.i = zext i64 %_0.i5.i.i.i to i128, !dbg !570199 + %_4.i1.i.i.i.i = mul nuw i128 %_6.i.i.i.i.i, %_5.i.i.i.i.i, !dbg !570200 + %_3.i2.i.i.i.i = lshr i128 %_4.i1.i.i.i.i, 64, !dbg !570201 + %_0.i.i.i.i.i = trunc nuw i128 %_3.i2.i.i.i.i to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i, !569883, !DIExpression(), !570203) + #dbg_value(ptr undef, !564034, !DIExpression(), !569892) + #dbg_value(i64 %_0.i.i.i.i.i, !564040, !DIExpression(), !569892) + %115 = or i64 %failed.sroa.0.011.i.i, %_0.i.i.i.i.i, !dbg !570204 + #dbg_value(i64 %115, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570205 + #dbg_value(ptr %self4.i.i, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i, ptr %self4.i.i, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + #dbg_value(ptr undef, !569916, !DIExpression(), !569929) + #dbg_value(ptr undef, !569911, !DIExpression(), !569930) + #dbg_value(ptr undef, !569931, !DIExpression(), !569935) + #dbg_value(ptr poison, !569934, !DIExpression(), !569935) + #dbg_value(i64 %_36.i.i, !569912, !DIExpression(), !570143) + #dbg_value(i64 %_36.i.i, !569905, !DIExpression(), !569906) + #dbg_value(i64 %_36.i.i, !569922, !DIExpression(), !569923) + %_36.i.i.1 = add nuw i64 %iter.sroa.0.012.i.i, 2, !dbg !570144 + #dbg_value(i64 %_36.i.i.1, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %_36.i.i, !569877, !DIExpression(), !570145) + #dbg_value(i64 %_36.i.i, !569899, !DIExpression(), !569900) + #dbg_value(i64 %_36.i.i, !570146, !DIExpression(), !570150) + #dbg_value(i64 %_36.i.i, !551477, !DIExpression(), !569894) + #dbg_value(i64 %_36.i.i, !551550, !DIExpression(), !570152) + #dbg_value(i64 %_36.i.i, !551542, !DIExpression(), !570154) + #dbg_value(i64 %_36.i.i, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_36.i.i, !dbg !570158 + %_0.i.i.i.i.1 = load i64, ptr %_4.i.i.i.i.1, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(i64 %_36.i.i, !551550, !DIExpression(), !570164) + #dbg_value(i64 %_36.i.i, !551542, !DIExpression(), !570166) + #dbg_value(i64 %_36.i.i, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i.1 = icmp ult i64 %_36.i.i, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.1), !dbg !570171 + %_4.i4.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_36.i.i, !dbg !570172 + %_0.i5.i.i.i.1 = load i64, ptr %_4.i4.i.i.i.1, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.1, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i.1, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i.1, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i.1, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(i64 %_0.i.i.i.i.1, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i.1, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i.1, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i.1, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i.1, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i.1, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i.1, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i.1, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i.1, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i.1, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i.1 = mul i64 %_0.i5.i.i.i.1, %_0.i.i.i.i.1, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i.1, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i.1, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i.1, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i.1, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i.1 = zext i64 %_0.i.i.i.i.1 to i128, !dbg !570198 + %_6.i.i.i.i.i.1 = zext i64 %_0.i5.i.i.i.1 to i128, !dbg !570199 + %_4.i1.i.i.i.i.1 = mul nuw i128 %_6.i.i.i.i.i.1, %_5.i.i.i.i.i.1, !dbg !570200 + %_3.i2.i.i.i.i.1 = lshr i128 %_4.i1.i.i.i.i.1, 64, !dbg !570201 + %_0.i.i.i.i.i.1 = trunc nuw i128 %_3.i2.i.i.i.i.1 to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.1, !569883, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.1, !564040, !DIExpression(), !569892) + %116 = or i64 %115, %_0.i.i.i.i.i.1, !dbg !570204 + #dbg_value(i64 %116, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_36.i.i, !dbg !570205 + #dbg_value(ptr %self4.i.i.1, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i.1, ptr %self4.i.i.1, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + %niter.next.1 = add i64 %niter, 2, !dbg !569937 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !569937 + br i1 %niter.ncmp.1, label %bb33.i.loopexit144.unr-lcssa, label %bb15.i.i, !dbg !569937 + +bb33.thread.i: ; preds = %bb26.preheader.i.thread, %bb11.i, %bb26.preheader.i + #dbg_value(i64 0, !569430, !DIExpression(), !570214) + #dbg_value(i64 %index.i, !569423, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !569651) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !570215, !noalias !569589 + #dbg_value(i64 0, !569401, !DIExpression(), !570217) + #dbg_declare(ptr poison, !569405, !DIExpression(), !570218) + #dbg_declare(ptr %value.i.i, !570219, !DIExpression(), !570222) + #dbg_value(ptr undef, !564775, !DIExpression(), !570225) + #dbg_value(ptr undef, !564776, !DIExpression(), !570225) + br label %bb36.i, !dbg !570226 + +bb33.i.loopexit.unr-lcssa: ; preds = %bb27.us.i.us, %bb27.us.i.us.preheader + %.lcssa.ph = phi i64 [ poison, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %iter.sroa.0.046.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %_158.us.i.us, %bb27.us.i.us ] + %accumulated.sroa.0.045.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %lcmp.mod148.not = icmp eq i64 %xtraiter147, 0, !dbg !569720 + br i1 %lcmp.mod148.not, label %bb33.i, label %bb27.us.i.us.epil, !dbg !569720 + +bb27.us.i.us.epil: ; preds = %bb33.i.loopexit.unr-lcssa + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + #dbg_value(i64 %accumulated.sroa.0.045.us.i.us.unr, !569447, !DIExpression(), !569717) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !569451, !DIExpression(), !569793) + #dbg_value(ptr %columns.i, !551276, !DIExpression(), !569794) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551277, !DIExpression(), !569794) + #dbg_value(ptr %columns.i, !551266, !DIExpression(), !569795) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551267, !DIExpression(), !569795) + #dbg_value(ptr %columns.i, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569798) + #dbg_value(i64 0, !551142, !DIExpression(), !569796) + #dbg_value(ptr %columns.i, !551269, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569827) + #dbg_value(ptr %columns.i, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569796) + %_0.sroa.0.0.i.i.us.i.us.epil = load i64, ptr %data.i.i.i.us.i, align 8, !dbg !569725, !noalias !569509, !noundef !23 + #dbg_value(ptr %14, !551266, !DIExpression(), !569800) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551267, !DIExpression(), !569800) + #dbg_value(ptr %14, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569801) + #dbg_value(ptr %14, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569803) + #dbg_value(ptr %14, !551269, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569804) + #dbg_value(i64 0, !551142, !DIExpression(), !569801) + %_0.sroa.0.0.i9.i.us.i.us.epil = load i64, ptr %data.i6.i7.i.us.i, align 8, !dbg !569730, !noalias !569509, !noundef !23 + #dbg_value(ptr poison, !569758, !DIExpression(), !569805) + #dbg_value(ptr poison, !569759, !DIExpression(), !569805) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569805) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !569805) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569756, !DIExpression(), !569806) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569757, !DIExpression(), !569806) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569747, !DIExpression(), !569807) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569748, !DIExpression(), !569807) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569740, !DIExpression(), !569808) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569735, !DIExpression(), !569809) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569741, !DIExpression(), !569808) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569736, !DIExpression(), !569809) + %_0.i3.i.us.i.us.epil = mul i64 %_0.sroa.0.0.i9.i.us.i.us.epil, %_0.sroa.0.0.i.i.us.i.us.epil, !dbg !569732 + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569766, !DIExpression(), !569810) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569768, !DIExpression(), !569811) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569767, !DIExpression(), !569810) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569769, !DIExpression(), !569811) + %_5.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i.i.us.i.us.epil to i128, !dbg !569762 + %_6.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i9.i.us.i.us.epil to i128, !dbg !569771 + %_4.i1.i.us.i.us.epil = mul nuw i128 %_6.i.i.us.i.us.epil, %_5.i.i.us.i.us.epil, !dbg !569772 + %_3.i2.i.us.i.us.epil = lshr i128 %_4.i1.i.us.i.us.epil, 64, !dbg !569773 + %_0.i.i159.us.i.us.epil = trunc nuw i128 %_3.i2.i.us.i.us.epil to i64, !dbg !569774 + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !569455, !DIExpression(), !569812) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !569453, !DIExpression(), !569812) + %self34.us.i.us.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.046.us.i.us.unr, !dbg !569775 + #dbg_value(ptr %self34.us.i.us.epil, !569779, !DIExpression(), !569813) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !569780, !DIExpression(), !569813) + store i64 %_0.i3.i.us.i.us.epil, ptr %self34.us.i.us.epil, align 8, !dbg !569776, !noalias !569509 + #dbg_value(ptr undef, !564034, !DIExpression(), !569482) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !564040, !DIExpression(), !569482) + %117 = or i64 %accumulated.sroa.0.045.us.i.us.unr, %_0.i.i159.us.i.us.epil, !dbg !569782 + #dbg_value(i64 %117, !569447, !DIExpression(), !569717) + #dbg_value(i64 poison, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + #dbg_value(ptr undef, !569472, !DIExpression(), !569475) + #dbg_value(ptr undef, !569463, !DIExpression(), !569468) + #dbg_value(ptr undef, !569476, !DIExpression(), !569480) + #dbg_value(ptr poison, !569479, !DIExpression(), !569480) + #dbg_value(i64 poison, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + br label %bb33.i, !dbg !570227 + +bb33.i.loopexit144.unr-lcssa: ; preds = %bb15.i.i, %bb15.i.i.preheader + %.lcssa145.ph = phi i64 [ poison, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %iter.sroa.0.012.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %lcmp.mod.not = icmp eq i64 %xtraiter, 0, !dbg !569937 + br i1 %lcmp.mod.not, label %bb33.i, label %bb15.i.i.epil, !dbg !569937 + +bb15.i.i.epil: ; preds = %bb33.i.loopexit144.unr-lcssa + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %failed.sroa.0.011.i.i.unr, !569873, !DIExpression(), !569927) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569912, !DIExpression(), !570143) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569905, !DIExpression(), !569906) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569922, !DIExpression(), !569923) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569875, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569877, !DIExpression(), !570145) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569899, !DIExpression(), !569900) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !570146, !DIExpression(), !570150) + #dbg_value(ptr undef, !551471, !DIExpression(), !569894) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551477, !DIExpression(), !569894) + #dbg_value(ptr poison, !551549, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551550, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551542, !DIExpression(), !570154) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570158 + %_0.i.i.i.i.epil = load i64, ptr %_4.i.i.i.i.epil, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(ptr poison, !551549, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551550, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551542, !DIExpression(), !570166) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i.epil = icmp ult i64 %iter.sroa.0.012.i.i.unr, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.epil), !dbg !570171 + %_4.i4.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570172 + %_0.i5.i.i.i.epil = load i64, ptr %_4.i4.i.i.i.epil, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.epil, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i.epil, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i.epil, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(ptr poison, !569758, !DIExpression(), !570185) + #dbg_value(ptr poison, !569759, !DIExpression(), !570185) + #dbg_value(i64 %_0.i.i.i.i.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i.epil, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i.epil, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i.epil, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i.epil, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i.epil = mul i64 %_0.i5.i.i.i.epil, %_0.i.i.i.i.epil, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i.epil, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i.epil, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i.epil = zext i64 %_0.i.i.i.i.epil to i128, !dbg !570198 + %_6.i.i.i.i.i.epil = zext i64 %_0.i5.i.i.i.epil to i128, !dbg !570199 + %_4.i1.i.i.i.i.epil = mul nuw i128 %_6.i.i.i.i.i.epil, %_5.i.i.i.i.i.epil, !dbg !570200 + %_3.i2.i.i.i.i.epil = lshr i128 %_4.i1.i.i.i.i.epil, 64, !dbg !570201 + %_0.i.i.i.i.i.epil = trunc nuw i128 %_3.i2.i.i.i.i.epil to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !569883, !DIExpression(), !570203) + #dbg_value(ptr undef, !564034, !DIExpression(), !569892) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !564040, !DIExpression(), !569892) + %118 = or i64 %failed.sroa.0.011.i.i.unr, %_0.i.i.i.i.i.epil, !dbg !570204 + #dbg_value(i64 %118, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570205 + #dbg_value(ptr %self4.i.i.epil, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i.epil, ptr %self4.i.i.epil, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + #dbg_value(ptr undef, !569916, !DIExpression(), !569929) + #dbg_value(ptr undef, !569911, !DIExpression(), !569930) + #dbg_value(ptr undef, !569931, !DIExpression(), !569935) + #dbg_value(ptr poison, !569934, !DIExpression(), !569935) + br label %bb33.i, !dbg !570227 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md new file mode 100644 index 00000000000..e30db12f65a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md @@ -0,0 +1,71 @@ + + + +# `final-u64-mul-dense.s` + +```s +.LBB1679_67: + .loc 181 765 12 + andq $-2, %r10 + xorl %edi, %edi + xorl %ecx, %ecx + movq -48(%rbp), %r8 +.Ltmp111183: +.LBB1679_68: + .loc 567 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp111184: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp111185: + movq %rdx, %rsi +.Ltmp111186: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp111187: + .loc 567 39 18 + movq 8(%r13,%rdi,8), %rax +.Ltmp111188: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp111189: + .loc 566 821 53 + orq %rcx, %rsi +.Ltmp111190: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp111191: + .loc 156 717 17 + addq $2, %rdi +.Ltmp111192: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp111193: + .loc 566 821 53 + orq %rsi, %rcx +.Ltmp111194: + .loc 181 765 12 + cmpq %r10, %rdi + jne .LBB1679_68 +.Ltmp111195: +.LBB1679_69: + testb $1, %r9b + je .LBB1679_84 +.Ltmp111196: + .loc 567 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp111197: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp111198: + .loc 566 821 53 + orq %rdx, %rcx +.Ltmp111199: + .loc 566 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp111200: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp111201: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md new file mode 100644 index 00000000000..4cd73bddfad --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md @@ -0,0 +1,96 @@ + + + +# `indexed-i64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb11.i, %bb15.i.i + %iter.sroa.0.012.i.i = phi i64 [ %_36.i.i, %bb15.i.i ], [ 0, %bb11.i ] + %failed.sroa.0.011.i.i = phi i64 [ %79, %bb15.i.i ], [ 0, %bb11.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !576781, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !576835) + #dbg_value(i64 %failed.sroa.0.011.i.i, !576779, !DIExpression(), !576834) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576819, !DIExpression(), !577050) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576812, !DIExpression(), !576813) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576829, !DIExpression(), !576830) + %_36.i.i = add nuw i64 %iter.sroa.0.012.i.i, 1, !dbg !577051 + #dbg_value(i64 %_36.i.i, !576781, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !576835) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576783, !DIExpression(), !577052) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576806, !DIExpression(), !576807) + #dbg_value(i64 %iter.sroa.0.012.i.i, !577053, !DIExpression(), !577057) + #dbg_value(ptr undef, !553722, !DIExpression(), !576801) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553728, !DIExpression(), !576801) + #dbg_value(ptr poison, !553800, !DIExpression(), !577059) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553801, !DIExpression(), !577059) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553793, !DIExpression(), !577061) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553785, !DIExpression(), !577063) + #dbg_value(ptr %column.val.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577061) + #dbg_value(ptr %column.val.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577063) + #dbg_value(i64 %len3.i.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577061) + #dbg_value(i64 %len3.i.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577063) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577065 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !577066, !noalias !577067, !noundef !23 + #dbg_value(ptr poison, !553800, !DIExpression(), !577071) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553801, !DIExpression(), !577071) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553793, !DIExpression(), !577073) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553785, !DIExpression(), !577075) + #dbg_value(ptr %column5.val.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577073) + #dbg_value(ptr %column5.val.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577075) + #dbg_value(i64 %len3.i.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577073) + #dbg_value(i64 %len3.i.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577075) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !577077 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !577078 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577079 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !577080, !noalias !577067, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !576785, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577081) + #dbg_value(i64 %_0.i5.i.i.i, !576785, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577081) + #dbg_value(i64 %_0.i.i.i.i, !577082, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577090) + #dbg_value(i64 %_0.i5.i.i.i, !577082, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577090) + #dbg_value(ptr poison, !576671, !DIExpression(), !577092) + #dbg_value(ptr poison, !576672, !DIExpression(), !577092) + #dbg_value(i64 %_0.i.i.i.i, !576673, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577092) + #dbg_value(i64 %_0.i5.i.i.i, !576673, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577092) + #dbg_value(i64 %_0.i.i.i.i, !576669, !DIExpression(), !577094) + #dbg_value(i64 %_0.i5.i.i.i, !576670, !DIExpression(), !577094) + #dbg_value(i64 %_0.i.i.i.i, !576660, !DIExpression(), !577095) + #dbg_value(i64 %_0.i5.i.i.i, !576661, !DIExpression(), !577095) + #dbg_value(i64 %_0.i.i.i.i, !576649, !DIExpression(), !577097) + #dbg_value(i64 %_0.i.i.i.i, !576644, !DIExpression(), !577099) + #dbg_value(i64 %_0.i5.i.i.i, !576650, !DIExpression(), !577097) + #dbg_value(i64 %_0.i5.i.i.i, !576645, !DIExpression(), !577099) + %_0.i.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !577101 + #dbg_value(i64 %_0.i.i.i.i, !576681, !DIExpression(), !577102) + #dbg_value(i64 %_0.i.i.i.i, !576683, !DIExpression(), !577104) + #dbg_value(i64 %_0.i5.i.i.i, !576682, !DIExpression(), !577102) + #dbg_value(i64 %_0.i5.i.i.i, !576684, !DIExpression(), !577104) + %_4.i1.i.i.i.i = sext i64 %_0.i.i.i.i to i128, !dbg !577105 + %_5.i.i.i.i.i = sext i64 %_0.i5.i.i.i to i128, !dbg !577106 + %wide.i.i.i.i.i = mul nsw i128 %_5.i.i.i.i.i, %_4.i1.i.i.i.i, !dbg !577105 + #dbg_value(i128 %wide.i.i.i.i.i, !576685, !DIExpression(), !577107) + %kept.i.i.i.i.i = trunc i128 %wide.i.i.i.i.i to i64, !dbg !577108 + #dbg_value(i64 %kept.i.i.i.i.i, !576687, !DIExpression(), !577109) + %_8.i.i.i.i.i = lshr i128 %wide.i.i.i.i.i, 64, !dbg !577110 + %discarded.i.i.i.i.i = trunc nuw i128 %_8.i.i.i.i.i to i64, !dbg !577111 + #dbg_value(i64 %discarded.i.i.i.i.i, !576689, !DIExpression(), !577112) + %_10.i.i.i.i.i = ashr i64 %kept.i.i.i.i.i, 63, !dbg !577113 + %_9.i.i.i.i.i = xor i64 %_10.i.i.i.i.i, %discarded.i.i.i.i.i, !dbg !577114 + #dbg_value(i64 poison, !576787, !DIExpression(), !577115) + #dbg_value(i64 %_9.i.i.i.i.i, !576789, !DIExpression(), !577115) + #dbg_value(ptr undef, !576390, !DIExpression(), !576799) + #dbg_value(i64 %_9.i.i.i.i.i, !576396, !DIExpression(), !576799) + %79 = or i64 %_9.i.i.i.i.i, %failed.sroa.0.011.i.i, !dbg !577116 + #dbg_value(i64 %79, !576779, !DIExpression(), !576834) + #dbg_value(i64 %_0.i.i.i.i.i, !576787, !DIExpression(), !577115) + #dbg_value(ptr %_4.sroa.10.0.i.i, !577056, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577057) + #dbg_value(i64 %index.i, !577056, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577057) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577117 + #dbg_value(ptr %self4.i.i, !577118, !DIExpression(), !577122) + #dbg_value(i64 %_0.i.i.i.i.i, !577121, !DIExpression(), !577122) + store i64 %_0.i.i.i.i.i, ptr %self4.i.i, align 8, !dbg !577124, !alias.scope !576795, !noalias !577125 + #dbg_value(ptr undef, !576823, !DIExpression(), !576836) + #dbg_value(ptr undef, !576818, !DIExpression(), !576837) + #dbg_value(ptr undef, !576838, !DIExpression(), !576842) + #dbg_value(ptr poison, !576841, !DIExpression(), !576842) + %exitcond.not.i.i = icmp eq i64 %_36.i.i, %len3.i4.i.i.fr, !dbg !577126 + br i1 %exitcond.not.i.i, label %bb33.i, label %bb15.i.i, !dbg !576844 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md new file mode 100644 index 00000000000..4f95e7d0668 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md @@ -0,0 +1,35 @@ + + + +# `indexed-i64-mul-dense.s` + +```s + .p2align 4 +.LBB1685_25: + .loc 566 39 18 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp113564: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp113565: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp113566: + .loc 156 717 17 + incq %rsi +.Ltmp113567: + .loc 565 198 26 + sarq $63, %rax +.Ltmp113568: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp113569: + .loc 568 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp113570: + .loc 182 1904 50 + cmpq %rsi, %r9 + jne .LBB1685_25 + jmp .LBB1685_60 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md new file mode 100644 index 00000000000..4b254e7d98a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md @@ -0,0 +1,322 @@ + + + +# `indexed-u64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb15.i.i, %bb15.i.i.preheader.new + %iter.sroa.0.012.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %116, %bb15.i.i ] + %niter = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %niter.next.1, %bb15.i.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %failed.sroa.0.011.i.i, !580571, !DIExpression(), !580625) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580610, !DIExpression(), !580841) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580603, !DIExpression(), !580604) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580620, !DIExpression(), !580621) + %_36.i.i = or disjoint i64 %iter.sroa.0.012.i.i, 1, !dbg !580842 + #dbg_value(i64 %_36.i.i, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580575, !DIExpression(), !580843) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580597, !DIExpression(), !580598) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580844, !DIExpression(), !580848) + #dbg_value(ptr undef, !563670, !DIExpression(), !580592) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563676, !DIExpression(), !580592) + #dbg_value(ptr poison, !563748, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563749, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563741, !DIExpression(), !580852) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580856 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(ptr poison, !563748, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563749, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563741, !DIExpression(), !580864) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !580869 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580870 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(ptr poison, !580456, !DIExpression(), !580883) + #dbg_value(ptr poison, !580457, !DIExpression(), !580883) + #dbg_value(i64 %_0.i.i.i.i, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i = zext i64 %_0.i.i.i.i to i128, !dbg !580896 + %_6.i.i.i.i.i = zext i64 %_0.i5.i.i.i to i128, !dbg !580897 + %_4.i1.i.i.i.i = mul nuw i128 %_6.i.i.i.i.i, %_5.i.i.i.i.i, !dbg !580898 + %_3.i2.i.i.i.i = lshr i128 %_4.i1.i.i.i.i, 64, !dbg !580899 + %_0.i.i.i.i.i = trunc nuw i128 %_3.i2.i.i.i.i to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i, !580581, !DIExpression(), !580901) + #dbg_value(ptr undef, !576390, !DIExpression(), !580590) + #dbg_value(i64 %_0.i.i.i.i.i, !576396, !DIExpression(), !580590) + %115 = or i64 %failed.sroa.0.011.i.i, %_0.i.i.i.i.i, !dbg !580902 + #dbg_value(i64 %115, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580903 + #dbg_value(ptr %self4.i.i, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i, ptr %self4.i.i, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + #dbg_value(ptr undef, !580614, !DIExpression(), !580627) + #dbg_value(ptr undef, !580609, !DIExpression(), !580628) + #dbg_value(ptr undef, !580629, !DIExpression(), !580633) + #dbg_value(ptr poison, !580632, !DIExpression(), !580633) + #dbg_value(i64 %_36.i.i, !580610, !DIExpression(), !580841) + #dbg_value(i64 %_36.i.i, !580603, !DIExpression(), !580604) + #dbg_value(i64 %_36.i.i, !580620, !DIExpression(), !580621) + %_36.i.i.1 = add nuw i64 %iter.sroa.0.012.i.i, 2, !dbg !580842 + #dbg_value(i64 %_36.i.i.1, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %_36.i.i, !580575, !DIExpression(), !580843) + #dbg_value(i64 %_36.i.i, !580597, !DIExpression(), !580598) + #dbg_value(i64 %_36.i.i, !580844, !DIExpression(), !580848) + #dbg_value(i64 %_36.i.i, !563676, !DIExpression(), !580592) + #dbg_value(i64 %_36.i.i, !563749, !DIExpression(), !580850) + #dbg_value(i64 %_36.i.i, !563741, !DIExpression(), !580852) + #dbg_value(i64 %_36.i.i, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_36.i.i, !dbg !580856 + %_0.i.i.i.i.1 = load i64, ptr %_4.i.i.i.i.1, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(i64 %_36.i.i, !563749, !DIExpression(), !580862) + #dbg_value(i64 %_36.i.i, !563741, !DIExpression(), !580864) + #dbg_value(i64 %_36.i.i, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i.1 = icmp ult i64 %_36.i.i, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.1), !dbg !580869 + %_4.i4.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_36.i.i, !dbg !580870 + %_0.i5.i.i.i.1 = load i64, ptr %_4.i4.i.i.i.1, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.1, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i.1, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i.1, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i.1, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(i64 %_0.i.i.i.i.1, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i.1, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i.1, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i.1, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i.1, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i.1, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i.1, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i.1, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i.1, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i.1, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i.1 = mul i64 %_0.i5.i.i.i.1, %_0.i.i.i.i.1, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i.1, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i.1, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i.1, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i.1, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i.1 = zext i64 %_0.i.i.i.i.1 to i128, !dbg !580896 + %_6.i.i.i.i.i.1 = zext i64 %_0.i5.i.i.i.1 to i128, !dbg !580897 + %_4.i1.i.i.i.i.1 = mul nuw i128 %_6.i.i.i.i.i.1, %_5.i.i.i.i.i.1, !dbg !580898 + %_3.i2.i.i.i.i.1 = lshr i128 %_4.i1.i.i.i.i.1, 64, !dbg !580899 + %_0.i.i.i.i.i.1 = trunc nuw i128 %_3.i2.i.i.i.i.1 to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.1, !580581, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.1, !576396, !DIExpression(), !580590) + %116 = or i64 %115, %_0.i.i.i.i.i.1, !dbg !580902 + #dbg_value(i64 %116, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_36.i.i, !dbg !580903 + #dbg_value(ptr %self4.i.i.1, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i.1, ptr %self4.i.i.1, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + %niter.next.1 = add i64 %niter, 2, !dbg !580635 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !580635 + br i1 %niter.ncmp.1, label %bb33.i.loopexit144.unr-lcssa, label %bb15.i.i, !dbg !580635 + +bb33.thread.i: ; preds = %bb26.preheader.i.thread, %bb11.i, %bb26.preheader.i + #dbg_value(i64 0, !580128, !DIExpression(), !580912) + #dbg_value(i64 %index.i, !580121, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !580349) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !580913, !noalias !580287 + #dbg_value(i64 0, !580099, !DIExpression(), !580915) + #dbg_declare(ptr poison, !580103, !DIExpression(), !580916) + #dbg_declare(ptr %value.i.i, !580917, !DIExpression(), !580920) + #dbg_value(ptr undef, !577131, !DIExpression(), !580923) + #dbg_value(ptr undef, !577132, !DIExpression(), !580923) + br label %bb36.i, !dbg !580924 + +bb33.i.loopexit.unr-lcssa: ; preds = %bb27.us.i.us, %bb27.us.i.us.preheader + %.lcssa.ph = phi i64 [ poison, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %iter.sroa.0.046.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %_157.us.i.us, %bb27.us.i.us ] + %accumulated.sroa.0.045.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %lcmp.mod148.not = icmp eq i64 %xtraiter147, 0, !dbg !580418 + br i1 %lcmp.mod148.not, label %bb33.i, label %bb27.us.i.us.epil, !dbg !580418 + +bb27.us.i.us.epil: ; preds = %bb33.i.loopexit.unr-lcssa + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + #dbg_value(i64 %accumulated.sroa.0.045.us.i.us.unr, !580145, !DIExpression(), !580415) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !580149, !DIExpression(), !580491) + #dbg_value(ptr %columns.i, !563475, !DIExpression(), !580492) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563476, !DIExpression(), !580492) + #dbg_value(ptr %columns.i, !563465, !DIExpression(), !580493) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563466, !DIExpression(), !580493) + #dbg_value(ptr %columns.i, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580496) + #dbg_value(i64 0, !563341, !DIExpression(), !580494) + #dbg_value(ptr %columns.i, !563468, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580525) + #dbg_value(ptr %columns.i, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580494) + %_0.sroa.0.0.i.i.us.i.us.epil = load i64, ptr %data.i.i.i.us.i, align 8, !dbg !580423, !noalias !580207, !noundef !23 + #dbg_value(ptr %14, !563465, !DIExpression(), !580498) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563466, !DIExpression(), !580498) + #dbg_value(ptr %14, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580499) + #dbg_value(ptr %14, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580501) + #dbg_value(ptr %14, !563468, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580502) + #dbg_value(i64 0, !563341, !DIExpression(), !580499) + %_0.sroa.0.0.i9.i.us.i.us.epil = load i64, ptr %data.i6.i7.i.us.i, align 8, !dbg !580428, !noalias !580207, !noundef !23 + #dbg_value(ptr poison, !580456, !DIExpression(), !580503) + #dbg_value(ptr poison, !580457, !DIExpression(), !580503) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580503) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580503) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580454, !DIExpression(), !580504) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580455, !DIExpression(), !580504) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580445, !DIExpression(), !580505) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580446, !DIExpression(), !580505) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580438, !DIExpression(), !580506) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580433, !DIExpression(), !580507) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580439, !DIExpression(), !580506) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580434, !DIExpression(), !580507) + %_0.i3.i.us.i.us.epil = mul i64 %_0.sroa.0.0.i9.i.us.i.us.epil, %_0.sroa.0.0.i.i.us.i.us.epil, !dbg !580430 + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580464, !DIExpression(), !580508) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580466, !DIExpression(), !580509) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580465, !DIExpression(), !580508) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580467, !DIExpression(), !580509) + %_5.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i.i.us.i.us.epil to i128, !dbg !580460 + %_6.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i9.i.us.i.us.epil to i128, !dbg !580469 + %_4.i1.i.us.i.us.epil = mul nuw i128 %_6.i.i.us.i.us.epil, %_5.i.i.us.i.us.epil, !dbg !580470 + %_3.i2.i.us.i.us.epil = lshr i128 %_4.i1.i.us.i.us.epil, 64, !dbg !580471 + %_0.i.i159.us.i.us.epil = trunc nuw i128 %_3.i2.i.us.i.us.epil to i64, !dbg !580472 + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !580151, !DIExpression(), !580510) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !580153, !DIExpression(), !580510) + #dbg_value(ptr undef, !576390, !DIExpression(), !580180) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !576396, !DIExpression(), !580180) + %117 = or i64 %accumulated.sroa.0.045.us.i.us.unr, %_0.i.i159.us.i.us.epil, !dbg !580473 + #dbg_value(i64 %117, !580145, !DIExpression(), !580415) + %self34.us.i.us.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.046.us.i.us.unr, !dbg !580474 + #dbg_value(ptr %self34.us.i.us.epil, !580478, !DIExpression(), !580511) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !580479, !DIExpression(), !580511) + store i64 %_0.i3.i.us.i.us.epil, ptr %self34.us.i.us.epil, align 8, !dbg !580475, !noalias !580207 + #dbg_value(i64 poison, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + #dbg_value(ptr undef, !580170, !DIExpression(), !580173) + #dbg_value(ptr undef, !580161, !DIExpression(), !580166) + #dbg_value(ptr undef, !580174, !DIExpression(), !580178) + #dbg_value(ptr poison, !580177, !DIExpression(), !580178) + #dbg_value(i64 poison, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + br label %bb33.i, !dbg !580925 + +bb33.i.loopexit144.unr-lcssa: ; preds = %bb15.i.i, %bb15.i.i.preheader + %.lcssa145.ph = phi i64 [ poison, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %iter.sroa.0.012.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %lcmp.mod.not = icmp eq i64 %xtraiter, 0, !dbg !580635 + br i1 %lcmp.mod.not, label %bb33.i, label %bb15.i.i.epil, !dbg !580635 + +bb15.i.i.epil: ; preds = %bb33.i.loopexit144.unr-lcssa + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %failed.sroa.0.011.i.i.unr, !580571, !DIExpression(), !580625) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580610, !DIExpression(), !580841) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580603, !DIExpression(), !580604) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580620, !DIExpression(), !580621) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580573, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580575, !DIExpression(), !580843) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580597, !DIExpression(), !580598) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580844, !DIExpression(), !580848) + #dbg_value(ptr undef, !563670, !DIExpression(), !580592) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563676, !DIExpression(), !580592) + #dbg_value(ptr poison, !563748, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563749, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563741, !DIExpression(), !580852) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580856 + %_0.i.i.i.i.epil = load i64, ptr %_4.i.i.i.i.epil, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(ptr poison, !563748, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563749, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563741, !DIExpression(), !580864) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i.epil = icmp ult i64 %iter.sroa.0.012.i.i.unr, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.epil), !dbg !580869 + %_4.i4.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580870 + %_0.i5.i.i.i.epil = load i64, ptr %_4.i4.i.i.i.epil, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.epil, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i.epil, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(ptr poison, !580456, !DIExpression(), !580883) + #dbg_value(ptr poison, !580457, !DIExpression(), !580883) + #dbg_value(i64 %_0.i.i.i.i.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i.epil, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i.epil, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i.epil, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i.epil, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i.epil = mul i64 %_0.i5.i.i.i.epil, %_0.i.i.i.i.epil, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i.epil, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i.epil, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i.epil = zext i64 %_0.i.i.i.i.epil to i128, !dbg !580896 + %_6.i.i.i.i.i.epil = zext i64 %_0.i5.i.i.i.epil to i128, !dbg !580897 + %_4.i1.i.i.i.i.epil = mul nuw i128 %_6.i.i.i.i.i.epil, %_5.i.i.i.i.i.epil, !dbg !580898 + %_3.i2.i.i.i.i.epil = lshr i128 %_4.i1.i.i.i.i.epil, 64, !dbg !580899 + %_0.i.i.i.i.i.epil = trunc nuw i128 %_3.i2.i.i.i.i.epil to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !580581, !DIExpression(), !580901) + #dbg_value(ptr undef, !576390, !DIExpression(), !580590) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !576396, !DIExpression(), !580590) + %118 = or i64 %failed.sroa.0.011.i.i.unr, %_0.i.i.i.i.i.epil, !dbg !580902 + #dbg_value(i64 %118, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580903 + #dbg_value(ptr %self4.i.i.epil, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i.epil, ptr %self4.i.i.epil, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + #dbg_value(ptr undef, !580614, !DIExpression(), !580627) + #dbg_value(ptr undef, !580609, !DIExpression(), !580628) + #dbg_value(ptr undef, !580629, !DIExpression(), !580633) + #dbg_value(ptr poison, !580632, !DIExpression(), !580633) + br label %bb33.i, !dbg !580925 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md new file mode 100644 index 00000000000..f0981969a0e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md @@ -0,0 +1,71 @@ + + + +# `indexed-u64-mul-dense.s` + +```s +.LBB1688_67: + .loc 181 765 12 + andq $-2, %r10 + xorl %edi, %edi + xorl %ecx, %ecx + movq -48(%rbp), %r8 +.Ltmp114798: +.LBB1688_68: + .loc 566 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp114799: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp114800: + movq %rdx, %rsi +.Ltmp114801: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp114802: + .loc 566 39 18 + movq 8(%r13,%rdi,8), %rax +.Ltmp114803: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp114804: + .loc 568 821 53 + orq %rcx, %rsi +.Ltmp114805: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp114806: + .loc 156 717 17 + addq $2, %rdi +.Ltmp114807: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp114808: + .loc 568 821 53 + orq %rsi, %rcx +.Ltmp114809: + .loc 181 765 12 + cmpq %r10, %rdi + jne .LBB1688_68 +.Ltmp114810: +.LBB1688_69: + testb $1, %r9b + je .LBB1688_84 +.Ltmp114811: + .loc 566 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp114812: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp114813: + .loc 568 821 53 + orq %rdx, %rcx +.Ltmp114814: + .loc 568 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp114815: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp114816: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md new file mode 100644 index 00000000000..c4a917571df --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md @@ -0,0 +1,84 @@ + + + +# `owned-i64-mul-dense.ll` + +```ll +terminate.i: ; preds = %bb57.i + %78 = landingpad { ptr, i32 } + filter [0 x ptr] zeroinitializer +; call core::panicking::panic_in_cleanup + call void @_ZN4core9panicking16panic_in_cleanup17h8f68387bb6cbbf54E() #88, !dbg !573712, !noalias !573567 + unreachable, !dbg !573712 + +bb18.i: ; preds = %bb18.i, %bb18.lr.ph.i + %_15552.i = phi i64 [ 1, %bb18.lr.ph.i ], [ %_155.i, %bb18.i ] + %iter.sroa.0.051.i = phi i64 [ 0, %bb18.lr.ph.i ], [ %_15552.i, %bb18.i ] + %failed.sroa.0.050.i = phi i64 [ 0, %bb18.lr.ph.i ], [ %81, %bb18.i ] + #dbg_value(i64 %iter.sroa.0.051.i, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + #dbg_value(i64 %failed.sroa.0.050.i, !573466, !DIExpression(), !573746) + #dbg_value(i64 %iter.sroa.0.051.i, !573479, !DIExpression(), !574113) + #dbg_value(ptr undef, !552190, !DIExpression(), !573546) + #dbg_value(i64 %iter.sroa.0.051.i, !552196, !DIExpression(), !573546) + #dbg_value(ptr poison, !552716, !DIExpression(), !574114) + #dbg_value(i64 %iter.sroa.0.051.i, !552717, !DIExpression(), !574114) + #dbg_value(ptr poison, !552716, !DIExpression(), !574116) + #dbg_value(i64 %iter.sroa.0.051.i, !552717, !DIExpression(), !574116) + %79 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.051.i, !dbg !574118 + %_0.i.i97.i = load i64, ptr %79, align 8, !dbg !574118, !noalias !574119, !noundef !23 + %80 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.051.i, !dbg !574122 + %_0.i5.i.i = load i64, ptr %80, align 8, !dbg !574122, !noalias !574119, !noundef !23 + #dbg_value(ptr poison, !573820, !DIExpression(), !574123) + #dbg_value(ptr poison, !573821, !DIExpression(), !574123) + #dbg_value(i64 %_0.i.i97.i, !573822, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !574123) + #dbg_value(i64 %_0.i5.i.i, !573822, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !574123) + #dbg_value(i64 %_0.i.i97.i, !573818, !DIExpression(), !574125) + #dbg_value(i64 %_0.i5.i.i, !573819, !DIExpression(), !574125) + #dbg_value(i64 %_0.i.i97.i, !573809, !DIExpression(), !574126) + #dbg_value(i64 %_0.i5.i.i, !573810, !DIExpression(), !574126) + #dbg_value(i64 %_0.i.i97.i, !573798, !DIExpression(), !574128) + #dbg_value(i64 %_0.i.i97.i, !573793, !DIExpression(), !574130) + #dbg_value(i64 %_0.i5.i.i, !573799, !DIExpression(), !574128) + #dbg_value(i64 %_0.i5.i.i, !573794, !DIExpression(), !574130) + %_0.i.i111.i = mul i64 %_0.i5.i.i, %_0.i.i97.i, !dbg !574132 + #dbg_value(i64 %_0.i.i97.i, !573830, !DIExpression(), !574133) + #dbg_value(i64 %_0.i.i97.i, !573832, !DIExpression(), !574135) + #dbg_value(i64 %_0.i5.i.i, !573831, !DIExpression(), !574133) + #dbg_value(i64 %_0.i5.i.i, !573833, !DIExpression(), !574135) + %_4.i1.i.i = sext i64 %_0.i.i97.i to i128, !dbg !574136 + %_5.i.i.i = sext i64 %_0.i5.i.i to i128, !dbg !574137 + %wide.i.i.i = mul nsw i128 %_5.i.i.i, %_4.i1.i.i, !dbg !574136 + #dbg_value(i128 %wide.i.i.i, !573834, !DIExpression(), !574138) + %kept.i.i.i = trunc i128 %wide.i.i.i to i64, !dbg !574139 + #dbg_value(i64 %kept.i.i.i, !573836, !DIExpression(), !574140) + %_8.i.i.i = lshr i128 %wide.i.i.i, 64, !dbg !574141 + %discarded.i.i.i = trunc nuw i128 %_8.i.i.i to i64, !dbg !574142 + #dbg_value(i64 %discarded.i.i.i, !573838, !DIExpression(), !574143) + %_10.i.i.i = ashr i64 %kept.i.i.i, 63, !dbg !574144 + %_9.i.i.i = xor i64 %_10.i.i.i, %discarded.i.i.i, !dbg !574145 + #dbg_value(i64 %_0.i.i111.i, !573481, !DIExpression(), !574146) + #dbg_value(i64 %_9.i.i.i, !573483, !DIExpression(), !574146) + #dbg_value(ptr undef, !573548, !DIExpression(), !573557) + #dbg_value(i64 %_9.i.i.i, !573554, !DIExpression(), !573557) + %81 = or i64 %_9.i.i.i, %failed.sroa.0.050.i, !dbg !574147 + #dbg_value(i64 %81, !573466, !DIExpression(), !573746) + %self32.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.051.i, !dbg !574148 + #dbg_value(ptr %self32.i, !573852, !DIExpression(), !574149) + #dbg_value(i64 %_0.i.i111.i, !573853, !DIExpression(), !574149) + store i64 %_0.i.i111.i, ptr %self32.i, align 8, !dbg !574151, !noalias !573567 + #dbg_value(i64 %_15552.i, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + #dbg_value(ptr undef, !573517, !DIExpression(), !573539) + #dbg_value(ptr undef, !573505, !DIExpression(), !573535) + #dbg_value(ptr undef, !573521, !DIExpression(), !573540) + #dbg_value(ptr poison, !573524, !DIExpression(), !573540) + %_155.i = add i64 %_15552.i, 1, !dbg !574152 + #dbg_value(i64 poison, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + %exitcond.not.i = icmp eq i64 %_15552.i, %len3.i4.i.i.fr, !dbg !574153 + br i1 %exitcond.not.i, label %bb38.i, label %bb18.i, !dbg !573891 + +bb38.thread.i: ; preds = %bb31.preheader.i.thread, %bb17.preheader.split.i, %bb31.preheader.i + #dbg_value(i64 0, !573466, !DIExpression(), !573746) + #dbg_value(i64 %index.i, !573459, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !573709) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !574154, !noalias !573647 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md new file mode 100644 index 00000000000..db24d697b4e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md @@ -0,0 +1,46 @@ + + + +# `owned-i64-mul-dense.s` + +```s + .loc 562 112 26 is_stmt 1 + je .LBB1685_70 +.Ltmp114190: + .loc 562 0 26 is_stmt 0 + movq -128(%rbp), %r13 +.Ltmp114191: + xorl %esi, %esi +.Ltmp114192: + xorl %ecx, %ecx + movq -48(%rbp), %rdi +.Ltmp114193: + .p2align 4 +.LBB1685_25: + .loc 564 62 9 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp114194: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp114195: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp114196: + .loc 565 198 26 + sarq $63, %rax +.Ltmp114197: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp114198: + .loc 566 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp114199: + .loc 182 1904 50 + incq %rsi +.Ltmp114200: + cmpq %rsi, %r9 + jne .LBB1685_25 + jmp .LBB1685_60 +.Ltmp114201: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md new file mode 100644 index 00000000000..0659a92c119 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md @@ -0,0 +1,121 @@ + + + +# `owned-u64-mul-dense.ll` + +```ll +bb18.i: ; preds = %bb18.i, %bb18.lr.ph.i.new + %_15552.i = phi i64 [ 1, %bb18.lr.ph.i.new ], [ %_155.i.1, %bb18.i ] + %iter.sroa.0.051.i = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %_155.i, %bb18.i ] + %failed.sroa.0.050.i = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %120, %bb18.i ] + %niter = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %niter.next.1, %bb18.i ] + #dbg_value(i64 %iter.sroa.0.051.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %failed.sroa.0.050.i, !576953, !DIExpression(), !577231) + #dbg_value(i64 %iter.sroa.0.051.i, !576966, !DIExpression(), !577601) + #dbg_value(ptr undef, !561311, !DIExpression(), !577032) + #dbg_value(i64 %iter.sroa.0.051.i, !561317, !DIExpression(), !577032) + #dbg_value(ptr poison, !561836, !DIExpression(), !577602) + #dbg_value(i64 %iter.sroa.0.051.i, !561837, !DIExpression(), !577602) + #dbg_value(ptr poison, !561836, !DIExpression(), !577604) + #dbg_value(i64 %iter.sroa.0.051.i, !561837, !DIExpression(), !577604) + %115 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.051.i, !dbg !577606 + %_0.i.i92.i = load i64, ptr %115, align 8, !dbg !577606, !noalias !577607, !noundef !23 + %116 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.051.i, !dbg !577610 + %_0.i5.i.i = load i64, ptr %116, align 8, !dbg !577610, !noalias !577607, !noundef !23 + #dbg_value(ptr poison, !577301, !DIExpression(), !577611) + #dbg_value(ptr poison, !577302, !DIExpression(), !577611) + #dbg_value(i64 %_0.i.i92.i, !577303, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577611) + #dbg_value(i64 %_0.i5.i.i, !577303, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577611) + #dbg_value(i64 %_0.i.i92.i, !577299, !DIExpression(), !577613) + #dbg_value(i64 %_0.i5.i.i, !577300, !DIExpression(), !577613) + #dbg_value(i64 %_0.i.i92.i, !577290, !DIExpression(), !577614) + #dbg_value(i64 %_0.i5.i.i, !577291, !DIExpression(), !577614) + #dbg_value(i64 %_0.i.i92.i, !577283, !DIExpression(), !577616) + #dbg_value(i64 %_0.i.i92.i, !577278, !DIExpression(), !577618) + #dbg_value(i64 %_0.i5.i.i, !577284, !DIExpression(), !577616) + #dbg_value(i64 %_0.i5.i.i, !577279, !DIExpression(), !577618) + %_0.i3.i.i = mul i64 %_0.i5.i.i, %_0.i.i92.i, !dbg !577620 + #dbg_value(i64 %_0.i.i92.i, !577309, !DIExpression(), !577621) + #dbg_value(i64 %_0.i.i92.i, !577311, !DIExpression(), !577623) + #dbg_value(i64 %_0.i5.i.i, !577310, !DIExpression(), !577621) + #dbg_value(i64 %_0.i5.i.i, !577312, !DIExpression(), !577623) + %_5.i.i.i = zext i64 %_0.i.i92.i to i128, !dbg !577624 + %_6.i.i.i = zext i64 %_0.i5.i.i to i128, !dbg !577625 + %_4.i1.i.i = mul nuw i128 %_6.i.i.i, %_5.i.i.i, !dbg !577626 + %_3.i2.i.i = lshr i128 %_4.i1.i.i, 64, !dbg !577627 + %_0.i.i106.i = trunc nuw i128 %_3.i2.i.i to i64, !dbg !577628 + #dbg_value(i64 %_0.i3.i.i, !576968, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i, !576970, !DIExpression(), !577629) + #dbg_value(ptr undef, !573548, !DIExpression(), !577036) + #dbg_value(i64 %_0.i.i106.i, !573554, !DIExpression(), !577036) + %117 = or i64 %failed.sroa.0.050.i, %_0.i.i106.i, !dbg !577630 + #dbg_value(i64 %117, !576953, !DIExpression(), !577231) + %self32.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.051.i, !dbg !577631 + #dbg_value(ptr %self32.i, !577323, !DIExpression(), !577632) + #dbg_value(i64 %_0.i3.i.i, !577324, !DIExpression(), !577632) + store i64 %_0.i3.i.i, ptr %self32.i, align 8, !dbg !577634, !noalias !577052 + #dbg_value(i64 %_15552.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(ptr undef, !577003, !DIExpression(), !577025) + #dbg_value(ptr undef, !576991, !DIExpression(), !577021) + #dbg_value(ptr undef, !577007, !DIExpression(), !577026) + #dbg_value(ptr poison, !577010, !DIExpression(), !577026) + %_155.i = add i64 %_15552.i, 1, !dbg !577635 + #dbg_value(i64 poison, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %_15552.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %_15552.i, !576966, !DIExpression(), !577601) + #dbg_value(i64 %_15552.i, !561317, !DIExpression(), !577032) + #dbg_value(i64 %_15552.i, !561837, !DIExpression(), !577602) + #dbg_value(i64 %_15552.i, !561837, !DIExpression(), !577604) + %118 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_15552.i, !dbg !577606 + %_0.i.i92.i.1 = load i64, ptr %118, align 8, !dbg !577606, !noalias !577607, !noundef !23 + %119 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_15552.i, !dbg !577610 + %_0.i5.i.i.1 = load i64, ptr %119, align 8, !dbg !577610, !noalias !577607, !noundef !23 + #dbg_value(i64 %_0.i.i92.i.1, !577303, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577611) + #dbg_value(i64 %_0.i5.i.i.1, !577303, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577611) + #dbg_value(i64 %_0.i.i92.i.1, !577299, !DIExpression(), !577613) + #dbg_value(i64 %_0.i5.i.i.1, !577300, !DIExpression(), !577613) + #dbg_value(i64 %_0.i.i92.i.1, !577290, !DIExpression(), !577614) + #dbg_value(i64 %_0.i5.i.i.1, !577291, !DIExpression(), !577614) + #dbg_value(i64 %_0.i.i92.i.1, !577283, !DIExpression(), !577616) + #dbg_value(i64 %_0.i.i92.i.1, !577278, !DIExpression(), !577618) + #dbg_value(i64 %_0.i5.i.i.1, !577284, !DIExpression(), !577616) + #dbg_value(i64 %_0.i5.i.i.1, !577279, !DIExpression(), !577618) + %_0.i3.i.i.1 = mul i64 %_0.i5.i.i.1, %_0.i.i92.i.1, !dbg !577620 + #dbg_value(i64 %_0.i.i92.i.1, !577309, !DIExpression(), !577621) + #dbg_value(i64 %_0.i.i92.i.1, !577311, !DIExpression(), !577623) + #dbg_value(i64 %_0.i5.i.i.1, !577310, !DIExpression(), !577621) + #dbg_value(i64 %_0.i5.i.i.1, !577312, !DIExpression(), !577623) + %_5.i.i.i.1 = zext i64 %_0.i.i92.i.1 to i128, !dbg !577624 + %_6.i.i.i.1 = zext i64 %_0.i5.i.i.1 to i128, !dbg !577625 + %_4.i1.i.i.1 = mul nuw i128 %_6.i.i.i.1, %_5.i.i.i.1, !dbg !577626 + %_3.i2.i.i.1 = lshr i128 %_4.i1.i.i.1, 64, !dbg !577627 + %_0.i.i106.i.1 = trunc nuw i128 %_3.i2.i.i.1 to i64, !dbg !577628 + #dbg_value(i64 %_0.i3.i.i.1, !576968, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i.1, !576970, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i.1, !573554, !DIExpression(), !577036) + %120 = or i64 %117, %_0.i.i106.i.1, !dbg !577630 + #dbg_value(i64 %120, !576953, !DIExpression(), !577231) + %self32.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_15552.i, !dbg !577631 + #dbg_value(ptr %self32.i.1, !577323, !DIExpression(), !577632) + #dbg_value(i64 %_0.i3.i.i.1, !577324, !DIExpression(), !577632) + store i64 %_0.i3.i.i.1, ptr %self32.i.1, align 8, !dbg !577634, !noalias !577052 + #dbg_value(i64 %_155.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + %_155.i.1 = add i64 %_15552.i, 2, !dbg !577635 + #dbg_value(i64 poison, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + %niter.next.1 = add i64 %niter, 2, !dbg !577379 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !577379 + br i1 %niter.ncmp.1, label %bb38.i.loopexit144.unr-lcssa, label %bb18.i, !dbg !577379 + +bb38.thread.i: ; preds = %bb31.preheader.i.thread, %bb17.preheader.split.i, %bb31.preheader.i + #dbg_value(i64 0, !576953, !DIExpression(), !577231) + #dbg_value(i64 %index.i, !576946, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !577194) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !577636, !noalias !577132 + #dbg_value(i64 0, !576924, !DIExpression(), !577638) + #dbg_declare(ptr poison, !576928, !DIExpression(), !577639) + #dbg_declare(ptr %value.i.i, !577640, !DIExpression(), !577643) + #dbg_value(ptr undef, !574157, !DIExpression(), !577646) + #dbg_value(ptr undef, !574158, !DIExpression(), !577646) + br label %bb41.i, !dbg !577647 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md new file mode 100644 index 00000000000..47957a445c2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md @@ -0,0 +1,72 @@ + + + +# `owned-u64-mul-dense.s` + +```s +.LBB1688_67: + .loc 562 112 26 + andq $-2, %r10 + xorl %ecx, %ecx + xorl %edi, %edi + movq -48(%rbp), %r8 +.Ltmp115437: +.LBB1688_68: + .loc 564 62 9 + movq (%r13,%rdi,8), %rax +.Ltmp115438: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp115439: + movq %rdx, %rsi +.Ltmp115440: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp115441: + .loc 564 62 9 + movq 8(%r13,%rdi,8), %rax +.Ltmp115442: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp115443: + .loc 566 821 53 + orq %rcx, %rsi +.Ltmp115444: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp115445: + .loc 566 821 53 + orq %rsi, %rcx +.Ltmp115446: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp115447: + .loc 562 112 26 + addq $2, %rdi + cmpq %rdi, %r10 + jne .LBB1688_68 +.Ltmp115448: +.LBB1688_69: + testb $1, %r9b + je .LBB1688_84 +.Ltmp115449: + .loc 564 62 9 + movq (%r13,%rdi,8), %rax +.Ltmp115450: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp115451: + .loc 566 821 53 + orq %rdx, %rcx +.Ltmp115452: + .loc 566 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp115453: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp115454: +.LBB1688_84: + .loc 182 1868 54 + testq %rcx, %rcx + +``` From d0a3186502ee230ac1d90c8e421c4faaff08c11b Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 19:39:06 -0400 Subject: [PATCH 11/12] Clarify RowFn compiler ablation evidence Signed-off-by: "Connor Tsui" --- research/rowfn-x86-2026-08-07/README.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/research/rowfn-x86-2026-08-07/README.md b/research/rowfn-x86-2026-08-07/README.md index e1df97740ba..fefa6f973d0 100644 --- a/research/rowfn-x86-2026-08-07/README.md +++ b/research/rowfn-x86-2026-08-07/README.md @@ -109,8 +109,9 @@ exposed the next compiler-sensitive detail. ## Store placement and the `Copy` ablation Moving the output store before the failure OR changed `mul_i32_constant` from about 32.38 to -18.68 microseconds. Final assembly records overflow `setb`, output store, then loop-carried OR. This -is compiler scheduling sensitivity, not a semantic difference. +18.68 microseconds. Final assembly records overflow `setb`, output store, then loop-carried OR. The +source-order ablation therefore changes whole-function code generation, but the final instruction +order is **not** a confirmed explanation for the throughput change. Adding the descriptive `Output: Copy` bound regressed that case to 29.88/29.86 microseconds. Replacing it with compile-time `!needs_drop::()` returned it to 18.65/18.67. A generic store @@ -118,6 +119,13 @@ helper did not repair the `Copy` case. The executor needs only the no-drop prope cleanup; it never copies an output. The API therefore enforces the actual requirement without the measured optimizer-visible bound. Re-test this workaround whenever LLVM changes. +An isolated exact-loop ablation on the same CPU contradicted the simple scheduling story. LLVM-MCA +estimated both final instruction orders at 2.7 cycles per iteration. Direct CPU 8 measurements made +OR-before-store slightly faster at 0.75-0.77 nanoseconds per row than store-before-OR at +0.823-0.825 nanoseconds per row. The production source-order effect is therefore an unresolved +whole-function compiler interaction, such as layout, surrounding control flow, or another +optimization decision. Do not justify the chosen source order as intrinsically better scheduling. + ## Final results Order: baseline, final, candidate, repeated twice. Values are median microseconds. @@ -165,6 +173,12 @@ AVX features, and LLVM selected the same essential scalar high-half strategy as [`final i64 assembly`](codegen/final-i64-mul-dense-s.md), and [`final u64 assembly`](codegen/final-u64-mul-dense-s.md). +A separate minimal `target-cpu=native` experiment did form `<8 x i128>` operations in LLVM IR for +the widened `u64` product. The x86 backend still scalarized them into eight `mulq`/`imulq` +instructions, then used ZMM registers only to pack and reduce the scalar results. x86 has no true +wide 64-by-64-to-128 integer multiply here. Seeing a vector IR type or ZMM instruction is therefore +not evidence that the expensive multiply itself executed as SIMD. + `-C remark=loop-vectorize` emitted no remark attributable to the exact dense production loop. The constant fallback source line had successes for other monomorphs and duplicated cost-model misses, but diagnostics lacked function identity. Exact IR proves the measured specialization is scalar; @@ -179,7 +193,8 @@ Confirmed: - `SinkResult` already reduced to a register OR and did not impose a per-row `Result`. - Output ownership materially helped but was insufficient alone. - A typed indexed source restored stable parity for varying primitive tuples. -- Store-before-OR and omission of a `Copy` bound materially affect LLVM 21.1.2 constant codegen. +- Source store/OR order and omission of a `Copy` bound materially affect LLVM 21.1.2 production + codegen; the mechanism behind the source-order effect remains unresolved. - The default x86 target prefers scalar high-half 64-bit multiply; SIMD is not the recovered speed. Still inference: From 53fff3325046ac24d1d6a5e5d595ebf1f6502f5b Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 7 Aug 2026 19:58:32 -0400 Subject: [PATCH 12/12] Record completed RowFn Copy ablation Signed-off-by: "Connor Tsui" --- research/rowfn-x86-2026-08-07/README.md | 64 +++++++++++++------ .../codegen/copy-ablation.md | 52 +++++++++++++++ 2 files changed, 95 insertions(+), 21 deletions(-) create mode 100644 research/rowfn-x86-2026-08-07/codegen/copy-ablation.md diff --git a/research/rowfn-x86-2026-08-07/README.md b/research/rowfn-x86-2026-08-07/README.md index fefa6f973d0..5a657f47fa0 100644 --- a/research/rowfn-x86-2026-08-07/README.md +++ b/research/rowfn-x86-2026-08-07/README.md @@ -106,25 +106,46 @@ execution validates both varying lengths once. The generic owned executor calls The indexed source closed the varying and nullable gap. It did not affect mixed constants, which exposed the next compiler-sensitive detail. -## Store placement and the `Copy` ablation - -Moving the output store before the failure OR changed `mul_i32_constant` from about 32.38 to -18.68 microseconds. Final assembly records overflow `setb`, output store, then loop-carried OR. The -source-order ablation therefore changes whole-function code generation, but the final instruction -order is **not** a confirmed explanation for the throughput change. - -Adding the descriptive `Output: Copy` bound regressed that case to 29.88/29.86 microseconds. -Replacing it with compile-time `!needs_drop::()` returned it to 18.65/18.67. A generic store -helper did not repair the `Copy` case. The executor needs only the no-drop property for safe panic -cleanup; it never copies an output. The API therefore enforces the actual requirement without the -measured optimizer-visible bound. Re-test this workaround whenever LLVM changes. - -An isolated exact-loop ablation on the same CPU contradicted the simple scheduling story. LLVM-MCA -estimated both final instruction orders at 2.7 cycles per iteration. Direct CPU 8 measurements made -OR-before-store slightly faster at 0.75-0.77 nanoseconds per row than store-before-OR at -0.823-0.825 nanoseconds per row. The production source-order effect is therefore an unresolved -whole-function compiler interaction, such as layout, surrounding control flow, or another -optimization decision. Do not justify the chosen source order as intrinsically better scheduling. +## Compiler ablations: `Copy`, source order, and whole-function sensitivity + +The completed ablation matrix isolates the public `Output: Copy` bound as a reliable trigger, while +falsifying the simpler explanations considered during the initial investigation: + +| Variant | `mul_i32_constant` run 1 / 2 | +| --- | ---: | +| No `Copy` bound | 18.77 / 18.72 us | +| Inert private marker bound | 18.77 / 18.72 us | +| `Output: Copy` | 29.94 / 29.93 us | +| `Output: Copy`, `codegen-units=1` | 29.87 / 29.89 us | + +The `i64` and `u64` controls did not move. The inert private marker is important: an arbitrary +where-clause or source perturbation is insufficient to trigger the loss. The result is specific to +the optimizer-visible `Copy` constraint, though the mechanism is not yet known. + +The default-CGU DWARF ranges show large whole-function differences for the exact `i32 CheckedMul` +monomorph. The `Copy` function spans `0xe58c90..0xe59adc` (`0xe4c` bytes); no-Copy spans +`0xe7a1c0..0xe7b6d0` (`0x1510` bytes). The `Copy` hot loop at `0xe58f90` is only 16-byte aligned and +computes the low multiply before the widened chain. The no-Copy loop at `0xe7b260` is 32-byte +aligned, computes the widened chain first, and delays the low multiply. LLVM-MCA nevertheless +predicts the smaller `Copy` loop slightly better, 2.5 versus 2.7 cycles. Alignment and final loop +scheduling therefore do not explain the measured direction. + +A fresh `Copy` plus `codegen-units=1` build makes this conclusion stronger: its optimized IR already +has store-before-OR, yet the linked benchmark remains at about 29.9 microseconds. Store-before-OR is +neither sufficient nor established as causal. The earlier source-order edit changed production +performance, but it must be described only as another trigger for a whole-function compiler +interaction. An isolated exact-loop hardware ablation also found OR-before-store slightly faster +(0.75-0.77 ns/row) than store-before-OR (0.823-0.825 ns/row), while LLVM-MCA rated both at 2.7 +cycles. The loop's local instruction order cannot explain the production result. + +A standalone generic `MaybeUninit` loop emits identical optimized IR and assembly with and without +`Copy`. The sensitivity therefore needs the real trait, closure, `Vec`, and monomorphization context. +This is currently evidence of compiler phase-order or code-quality sensitivity, not enough to claim +a rustc correctness bug or a specific LLVM bug. The next upstream step is to reduce the real +monomorph while retaining both the timing and whole-function delta, then bisect MIR/LLVM passes and +compiler versions. The executor needs only the no-drop property, so the selected API continues to +enforce `!needs_drop::()` without exposing the harmful, unnecessary `Copy` bound. See the +compact [Copy-ablation evidence](codegen/copy-ablation.md). ## Final results @@ -193,8 +214,9 @@ Confirmed: - `SinkResult` already reduced to a register OR and did not impose a per-row `Result`. - Output ownership materially helped but was insufficient alone. - A typed indexed source restored stable parity for varying primitive tuples. -- Source store/OR order and omission of a `Copy` bound materially affect LLVM 21.1.2 production - codegen; the mechanism behind the source-order effect remains unresolved. +- An `Output: Copy` bound reliably triggers slower LLVM 21.1.2 production codegen; an inert marker + does not. Source store/OR order is neither sufficient nor established as causal. The mechanism is + an unresolved whole-function compiler interaction. - The default x86 target prefers scalar high-half 64-bit multiply; SIMD is not the recovered speed. Still inference: diff --git a/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md b/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md new file mode 100644 index 00000000000..185018ab6a9 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md @@ -0,0 +1,52 @@ + + + +# `Copy`-bound compiler ablation + +This note records the compact evidence behind the no-drop assertion in owned RowFn execution. + +## Timings + +All public `binary_ops` runs used CPU 8 and the same default repository flags. + +```text +no Copy bound mul_i32_constant 18.77 / 18.72 us +inert private marker bound mul_i32_constant 18.77 / 18.72 us +Output: Copy mul_i32_constant 29.94 / 29.93 us +Output: Copy, CGU=1 mul_i32_constant 29.87 / 29.89 us +i64/u64 controls unchanged +``` + +The inert marker rules out a generic “any where-clause/source change perturbs codegen” explanation. +`codegen-units=1` rules out the default partitioning choice as a repair. + +## Exact production functions + +Default-CGU DWARF identified the measured `i32 CheckedMul` monomorphs: + +```text +Copy: 0xe58c90..0xe59adc, size 0xe4c +no-Copy: 0xe7a1c0..0xe7b6d0, size 0x1510 + +Copy hot loop: 0xe58f90, 16-byte but not 32-byte aligned +no-Copy hot loop: 0xe7b260, 32-byte aligned +``` + +The Copy loop schedules the low `imul` before the widened-product chain. No-Copy schedules the +widened chain first and delays the low multiply. LLVM-MCA predicts Copy slightly better at 2.5 +cycles versus 2.7, contradicting scheduling as the cause of its 1.6x wall-time loss. + +Fresh Copy-plus-CGU1 optimized IR contains store-before-OR and still runs at 29.9 microseconds. +Therefore store-before-OR is not sufficient. Exact isolated loops also contradict causality: + +```text +LLVM-MCA: both orders 2.7 cycles +OR before store: 0.75-0.77 ns/row +store before OR: 0.823-0.825 ns/row +``` + +The standalone generic `MaybeUninit` loop produces identical Copy/no-Copy IR and assembly. The +remaining hypothesis is phase-order or code-quality sensitivity requiring the real trait, closure, +`Vec`, and monomorphization context. Do not label this a correctness bug or assign it to a specific +rustc/LLVM pass without a reduced reproducer. Reduce the real monomorph while retaining timing and +whole-function changes, then bisect MIR/LLVM passes and compiler versions.