diff --git a/AGENTS.md b/AGENTS.md index a0e4c6558cd..30d67e18cb5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,12 @@ Notes: Avoid hidden-cost per-element accessors in hot loops, follow the performance guidance in `STYLE.md`, and benchmark changes to hot paths. +Treat branchless indexing as a code-generation hypothesis, not as an optimization by itself. A +runtime expression such as `index & mask` can make a slice index non-affine, retain bounds checks, +and block vectorization. Inspect generated code before replacing a loop-invariant enum match because +LLVM can unswitch the match into specialized loops. For binary kernels, benchmark varying x varying, +varying x constant, constant x varying, and nullable constant shapes separately. + ## Tests - Strongly consider `rstest` cases when parameterizing repetitive test logic. diff --git a/Cargo.toml b/Cargo.toml index 36aa5b2ac9e..5328905cc95 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -160,7 +160,13 @@ flatbuffers = "25.2.10" fsst-rs = "0.6.0" futures = { version = "0.3.31", default-features = false } fuzzy-matcher = "0.3" -geo = "0.31.0" +# `vortex-spatial`'s `contains_route` transcribes geo's `impl_contains_from_relate!` dispatch table, so +# any bump that moves a row silently changes containment verdicts — the tests stay green wherever +# relate and the direct algorithm agree. Pinned exactly so that taking any new geo, patch releases +# included, is a deliberate edit of this line that re-verifies the table; a caret requirement would +# let `cargo update` (or automated lockfile maintenance) take 0.31.x with no diff to review. See +# `vortex-spatial/src/scalar_fn/contains.rs`. +geo = "=0.31.0" geo-traits = "0.3.0" geo-types = "0.7.19" geoarrow = "0.8.0" diff --git a/NUMERIC_ROWFN_PLAN.md b/NUMERIC_ROWFN_PLAN.md new file mode 100644 index 00000000000..6fb337afc04 --- /dev/null +++ b/NUMERIC_ROWFN_PLAN.md @@ -0,0 +1,275 @@ + + + +# Plan: fit the numeric binary operators onto `RowFn` + +> The later x86 follow-up found that the sink-only API regressed varying `i64`/`u64` multiply and +> added separate owned-output and stateful-sink capabilities. Its complete evidence is in +> [`research/rowfn-x86-2026-08-07/README.md`](research/rowfn-x86-2026-08-07/README.md). Treat that +> record as authoritative where it supersedes the pre-x86 conclusions below. + +Working note, branch-only, like `SCALAR_FN_HANDOFF.md`. Written so this survives a conversation +compaction: everything needed to start is here, and nothing below depends on chat history. + +## Where things stand + +Branch `ct/row-fn`, at `4becc863ae` after the final API +simplification. Issues #9128, #9129, and #9130 match the implementation. The public-path benchmark +baseline from #9136 is now in the repository. + +This document preserves the original spike plan and the measurements that answered it. The current +API has no witnesses, persistence is function-owned, executor-only helper traits are sealed, and +filtered decode cost is additive per input. Read the outcome and final API/codegen sections before +following an earlier step literally. + +`byte_length` is no longer a row function, and `Bytes`/`BytesLen` are deleted. It measured 7.6-7.7x +slower than develop and is the case #9128 already excludes. + +## Goal of this spike + +Prove, or disprove, that the four arithmetic operators can move onto `RowFn` without changing the +`RowFn` API, without a second scalar function ID, and without touching serialization. Doing this +first is deliberate: it is the change most likely to force an API change, and discovering that after +tensor and geo are ported would mean reworking them. + +## The design + +`Binary` keeps everything and delegates only execution: + +```rust +Operator::Add => ScalarFnVTable::execute(&NumericBinary, &NumericOperator::Add, args, ctx), +``` + +`NumericBinary` is a `RowFn` with `Options = NumericOperator` and `FALLIBLE = true`. It is **not** +registered as a public scalar function, so it needs no ID in the registry and appears in no +serialized expression. It is reached through the `ScalarFnVTable::execute` that the blanket impl +already provides. + +Why this works, and each of these was verified against the code rather than assumed: + +- **Nothing is lost.** `BooleanKernel` and `CompareKernel` exist with per-encoding pushdown; there is + no `NumericKernel`. Unlike `not`, a numeric port gives up no encoding fast path. +- **The seam is already numeric-only.** All four arithmetic arms of `Binary::execute` funnel into + `execute_numeric(lhs, rhs, NumericOperator, ctx)`, and `NumericOperator` is already its own enum in + `crate::scalar`, so it is a ready-made `RowFn::Options`. +- **Fallibility is uniform.** `Binary::is_fallible` is false for the six comparisons plus `And`/`Or` + and true for exactly the four arithmetic operators, so `FALLIBLE = true` on a numeric-only `RowFn` + is exactly right. The options-independence of `RowFn::is_fallible` only bites when one function + spans both families. +- **Strictness stays where it belongs.** `Binary::is_strict` is `!matches!(op, And | Or)` because + Kleene `false AND null` is a valid `false`. `Binary` keeps owning that; `NumericBinary` never sees + a boolean operator. +- **Decimal fits.** `OutputSink::sink_dtype(args)` sees the input dtypes, which is what + `numeric_op_result_decimal_dtype(decimal_dtype, op)` needs. + +## Steps + +1. **Primitive path only, `Add` only.** A `NumericBinary` `RowFn` over `(T, T)` for one integer + width, with a deferred-error sink that writes the wrapping sum and ORs an overflow bit. Delegate + only `Operator::Add` from `Binary::execute` and leave the other three on `execute_numeric`. + Success is: the existing `binary/numeric/tests.rs` suite passes unchanged. +2. **Widen to every primitive ptype**, through `match_each_native_ptype!` in `dispatch`. Confirm the + compile-time witness check tolerates it, as it does for tensor widths. +3. **Add `Sub`, `Mul`, `Div`.** `Div` is the awkward one: see the risk below. +4. **Decide decimal.** Either a decimal input element plus a sink that carries the result precision + and scale, or leave `DType::Decimal` on `execute_numeric` and delegate only the primitive path. + Leaving it is a legitimate outcome for the spike and possibly for the first PR. +5. **Delete the replaced code** only once benchmarks agree, not before. + +## Risks, in the order they are likely to bite + +- **`Div` already has a per-type strategy.** `primitive.rs` carries `CHECKED_VALUE_LOOP` and + `DIV_CHECKS_IN_VALUE_LOOP`, set per type, so division checking is not uniform. A single row closure + may not express it, and `Div` may have to stay behind. +- **The existing implementation is tuned, not naive.** `checked.rs` has `checked_lanes` and + `checked_apply_lanes` taking a `valid_rows: &Mask` and returning `Result, usize>` with the + failing index. The port is replacing real engineering, so parity is not a given. This is the reason + the CodSpeed gate on the `binary_ops` names from #9136 matters. +- **Two declarations of the result dtype must agree.** `Binary::return_dtype` is what the expression + layer uses, while `reconcile_return` checks the kernel output against `NumericBinary`'s + sink-derived dtype. Cover every operator and dtype pair with a test that asserts they match. +- **Error messages are part of the contract.** `primitive.rs` defines `ERROR` per operator, such as + `"integer overflow in checked add"`, and `numeric/tests.rs` asserts on failures. The deferred-error + sink reports once from `finish`, so the message must be preserved and the error must still be + raised for the same inputs. +- **Overflow behind a null row must stay invisible.** `numeric/tests.rs` has + `test_decimal_overflow_on_null_lane_ignored`. The lifting's deferred-error retry over valid rows is + exactly this behavior, so the test should pass, but it is the first thing to check. + +## Verification + +```bash +cargo nextest run -p vortex-array +cargo clippy --all-targets --all-features -p vortex-array +cargo +nightly fmt --all +cargo test --doc -p vortex-array +``` + +The numeric suite specifically: + +```bash +cargo nextest run -p vortex-array scalar_fn::fns::binary +``` + +Performance gate is CodSpeed on the stable `binary_ops` names from #9136. Locally, use +`cargo bench -p vortex-array --bench binary_ops` with two runs, fastest and median, machine stated. + +## What this spike is not + +Not a PR. Not a deletion of `execute_numeric`. Not decimal support unless step 4 turns out easy. The +output is an answer to "does this fit cleanly", plus whatever the answer implies for #9129's API. + +## Outcome + +It fits, with no change to the `RowFn` API and one change to the machinery. + +Steps 1 through 3 landed together rather than in sequence: once the sink existed, widening it through +`match_each_native_ptype!` and adding the other three operators was the same code. Step 4 leaves +decimal on `execute_numeric_decimal`, which the delegation makes easy since `execute_numeric` still +owns the dtype split. Step 5 deleted the replaced primitive execution, which the measurements below +justify. + +### What the design turned out to be + +`Binary::execute` is untouched. `execute_numeric` keeps its validation, its error messages, its empty +short circuit, and its primitive/decimal split, and only `execute_numeric_primitive` changed: it +builds a `VecExecutionArgs` and calls `ScalarFnVTable::execute(&NumericBinary, &op, ..)`. Everything +the old implementation did around the arithmetic (decoding, the constant-operand collapse, the +all-constant fold, the null-constant short circuit, output allocation, nullability widening, masking, +and the valid-row retry after an overflow behind a null) is now the lifting's. + +`NumericOperator` became the options type. `NumericBinary` is unregistered and deliberately has no +serialization implementation. Persistence now belongs to each `RowFn`, so reusing an options type +does not silently assign the helper a wire contract. `Binary` retains its existing ID and options +serialization, and only primitive execution delegates to `NumericBinary`. + +Three things the old code carried are gone because the row framework removes the distinction they +existed for: + +- `CHECKED_VALUE_LOOP` and `DIV_CHECKS_IN_VALUE_LOOP` chose between a split value/error scan and a + one-pass early-exit kernel, because for integer division the split loop only added a second scan. + A row kernel produces the value and the error bit in the same pass, so there is one loop shape and + no choice to make. `div_i64` got 1.11x faster. +- `checked_apply_lanes` had no caller left. `checked_lanes` stays for decimal. +- `PrimitiveOperand` moved to `compare/primitive.rs`, its only remaining user. + +### The machinery change: the reduction is a word the kernel chooses + +`SinkResult` gained `Accumulated`, the word the executor OR-reduces in a loop-local. Two properties +of that reduction are load-bearing, and each was got wrong once before the numbers made it obvious. + +- **Width no greater than the element.** `DeferredError` held an `i64`, which bounds how many rows a + vector of the reduction covers whatever the element width. That cost `Mul` 3.5x at `i8`, 2.05x at + `i16` and 1.28x at `i32`, and nothing at `i64` where the widths already agree. +- **It lives in a loop-local, not in the sink.** Holding the accumulator as a sink field, reached + through a `&mut` for every row, is a loop-carried memory dependence. It cost the boolean kernels + 2.5x to 10x while leaving the three unsigned multiply kernels untouched. + +Naming the word is also what lets multiplication report the discarded high half of its product +rather than a comparison, which is what recovers its vectorization. `OutputSink` is unchanged and no +sink names the word. + +### Results + +Against the hand-written kernels, divan medians, best of two runs, 65536 rows, Apple M4 Max, with +the decimal, boolean and comparison benchmarks held as controls and moving under 2%: + +| benchmark | hand-written | row framework | | +| --- | --- | --- | --- | +| `mul_u8_nonnull` | 22.91 us | 1.854 us | 12.4x faster | +| `mul_u16_nonnull` | 22.20 us | 3.791 us | 5.9x faster | +| `mul_u32_nonnull` | 24.62 us | 7.124 us | 3.5x faster | +| `div_i64_nonnull` | 40.41 us | 34.83 us | 1.16x faster | +| `mul_i64_nonnull` | 27.37 us | 28.66 us | 1.05x slower | +| `mul_i32_constant` | 7.583 us | 8.041 us | 1.06x slower | + +Everything else lands within 3%, which is inside this host's drift between sessions. The unsigned +multiply win is not attributable to the port: the same defect exists in the hand-written kernels and +is fixed for `develop` separately in vortex-data/vortex#9210, stacked on vortex-data/vortex#9211. +Re-measure the port against `develop` once that lands, because the comparison above flatters it. + +### Measured dead ends + +Recorded so they are not retried. The entries that predate the broadcast-index-mask experiment are +also in vortex-data/vortex#9130. + +- Bounds-check elimination in the row loop is not available. Narrowing the varying view to the row + count buys nothing, and `get_unchecked` is not uniformly a win: about 10% on `mul_u16` and + `mul_u32`, and 22% slower on `mul_u8`. +- A per-argument row source that keeps the `Varying` view when another argument is batch-constant is + 4x slower than the `ArgColumn` branch it replaces, which already vectorizes. +- Pairing each varying view with a runtime index mask also fails. Commit `ad24700088` used + `index & usize::MAX` for varying inputs and `index & 0` for constants. On x86 with + `RUSTFLAGS="-C target-feature=+avx2"`, the constant numeric cases became approximately 4x to 7x + slower in wall time while non-constant cases stayed at parity. CodSpeed reported smaller but + consistent regressions: `add_i64_constant` 31.34%, `sub_i64_constant` 32.38%, and + `mul_i32_constant` 46.24%. +- The disassembly explains the mask result. Each `index & mask` remained behind a slice bounds + check, so LLVM saw a non-affine index and emitted a scalar loop. The old enum match was + loop-invariant, and LLVM unswitched it into constant-pattern loops with affine varying + accesses. Removing a branch removed information that the vectorizer needed. + +Commit `ad24700088` was removed from `ct/row-fn` history. The clean head after the rewrite is +`ea58061b5d`. A compile-time varying x constant or constant x varying specialization remains a +possible design, but it is not work for the first PR. Implement it only after the clean branch has +a stable mixed-constant regression against the current merge base. + +The earlier `mul_i32_constant` result was within Apple host drift and predates the mask experiment. +It does not establish parity against current `develop`. Rerun the clean candidate and current merge +base on x86 before deleting the hand-written kernels in a mergeable PR. + +### What this implies for #9129 and #9130 + +- The `RowFn` API needed nothing. No new visit method, no options-aware `sink_dtype`, no return + witness. `NumericBinary::FALLIBLE = true` is conservative for every dispatch arm, and each + concrete result type supplies the precise loop behavior. +- `SinkResult::Accumulated` and its two constraints belong in #9130, and are recorded there. +- On kernels this close to the vectorizer's decision boundary, the emitted IR is the reliable gate + and wall clock on one host is not. Two separate interventions here moved a benchmark the wrong + way, and host drift between sessions exceeded the effects under measurement. + +### Final API cleanup and generated code + +The later simplification did not add numeric-specific surface: + +- `NumericBinary` declares `ARG_NAMES = &["lhs", "rhs"]` instead of repeating an argument witness. +- Its `Options = NumericOperator` has no persistence bound or implementation. The registered + `Binary` function remains the sole owner of the serialized `vortex.binary` contract. +- The selected input tuple carries arity, dense-safety, decode fallibility, and filtered-decode + cost. The selected sink and `SinkResult` carry output and deferred-error facts. +- `SinkResult` is sealed, but a numeric function does not need to implement it. It chooses the + supplied unsigned evidence width that matches the primitive element width. +- `OutputSink` remains one abstraction. A later numeric function with multiple logical outputs + should put both builders in one sink rather than add a pair-of-sinks framework type. + +The final cleanup was checked against its parent by cross-compiling the optimized +`row_fn_executor` benchmark for `x86_64-apple-darwin` with `target-cpu=x86-64-v3`. After normalizing +symbol names and metadata, the vector/reduction block for checked `i64` add matched exactly. It +retains `<4 x i64>` loads and adds, vector overflow detection through xor/and/compare operations, +`<4 x i1>` OR accumulation, and a reduction after the loop. The vector body has no call or panic +path, and the scalar tail is unchanged. + +The ordinary `ElementSink` and custom-sink wrapping-add monomorphs also matched exactly. Native +Apple M4 Max measurements over 65,536 rows found RowFn median changes between 1.11% faster and 0.94% +slower, with fastest changes within about 0.17%. Specialized controls drifted more than the RowFn +arms, so there is no measurable native regression from the cleanup. + +This is not an x86 runtime result. It proves that the API edits preserved the optimized x86_64-v3 +loop shape. Runtime confirmation for numeric changes should use the stable public benchmark names +from #9136 on the target host. + +The next session will run on x86 and must perform that confirmation. The #9136 `binary_ops` +benchmark is on `develop` at `9a482c0230`, so compare this branch with the latest +`origin/develop` using the same public benchmark names. Record both exact commits and run each +revision at least twice in alternating order. If possible, pin one core. Report fastest and median +values with the CPU and timer configuration. If a stable case regresses, compare its optimized LLVM +IR before changing the row API or restoring hand-written execution. + +### Verification + +The whole of `binary/numeric/tests.rs` passed unchanged, including +`test_decimal_overflow_on_null_lane_ignored` and the integer-error tests that pin the valid-row +retry. Decimal is untouched and stays on `execute_numeric_decimal`. The final API state also +recorded 67 focused RowFn tests, 179 tensor tests, 230 geo tests, nightly formatting, and full +workspace clippy. Clippy needed `PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1` because the host +Python is 3.9 while the workspace targets the Python 3.11 stable ABI. diff --git a/SCALAR_FN_HANDOFF.md b/SCALAR_FN_HANDOFF.md new file mode 100644 index 00000000000..10224704829 --- /dev/null +++ b/SCALAR_FN_HANDOFF.md @@ -0,0 +1,459 @@ + + + +# Handoff: the row scalar-function framework + +This is the concise source of truth for the branch. `STRICT_SCALAR_FN_RESEARCH.md` keeps the full +design history, rejected alternatives, measurements, and generated-code evidence. +`NUMERIC_ROWFN_PLAN.md` records the numeric-binary migration and its narrower performance boundary. +`research/rowfn-x86-2026-08-07/README.md` records the later x86 regression reproduction, the +owned-output and indexed-source experiments, raw benchmark logs, and exact production IR/assembly. +All three are branch-only working notes for agents. They are not intended to land with the API. + +The public design lives in these tracking issues, which now match the implementation: + +- [#9128, Row-oriented scalar functions](https://github.com/vortex-data/vortex/issues/9128) +- [#9129, Define the `RowFn` API](https://github.com/vortex-data/vortex/issues/9129) +- [#9130, Execute `RowFn` over Vortex arrays](https://github.com/vortex-data/vortex/issues/9130) + +The branch is `ct/row-fn`, and draft PR #9255 remains the integration and research branch. Its +history was rewritten at `ea58061b5d` to remove the regressing broadcast-index-mask experiment. +Do not use the draft PR as the first mergeable change. Cut the first PR from the latest +`origin/develop`, and keep this branch as the source for later tensor and spatial ports. Push or +rewrite either branch only when explicitly requested. + +## Next action: cut the vortex-array PR + +The first mergeable PR must stay within `vortex-array` and contain: + +1. the `RowFn` API, lifting, executor, and focused behavioral tests. +2. the primitive `NumericBinary` port as its production consumer. +3. only the executor and numeric benchmarks needed to support its performance claim. + +Do not include the tensor or spatial ports, these branch-only working notes, the unrelated `like` +benchmark additions, or the fixed-size-list test. `NumericBinary` is the only `RowFn` consumer in +`vortex-array` on this branch. It already exercises varying and constant inputs, all-constant +folding, null constants, nullable execution, deferred overflow evidence, and the valid-row retry. +Do not add another consumer only to make the PR appear broader. + +The numeric commit deletes the now-unused `vortex-compute::lane_kernels::map_into` helper. Leave +that helper in place for a strictly `vortex-array`-only PR, and remove it in a separate cleanup. + +The first PR must establish parity against the latest `origin/develop`, not the integration +branch's old merge base. Run the public `binary_ops` benchmark on x86 with identical build flags at +both revisions. Cover varying x varying, varying x constant, constant x varying, and nullable plus +constant inputs. Run each revision at least twice in alternating order. Record the exact commits, +CPU, timer, pinning, fastest values, and medians. Inspect optimized LLVM IR or assembly for every +stable regression before changing the row API. + +The production benchmark commands across the staged work are: + +```bash +cargo bench -p vortex-array --bench binary_ops +cargo bench -p vortex-array --bench like +cargo bench -p vortex-tensor --bench l2_norm +cargo bench -p vortex-tensor --bench inner_product +cargo bench -p vortex-tensor --bench cosine_similarity +cargo bench -p vortex-tensor --bench normalized +cargo bench -p vortex-spatial --bench binary_predicates +cargo bench -p vortex-spatial --bench distance +cargo bench -p vortex-spatial --bench envelope +cargo bench -p vortex-spatial --bench predicate_bbox +``` + +For the spatial PR, also run the branch-only `vortex-spatial` `null_strategies` diagnostic. It +forces branch-and-skip and filter-and-scatter for the measured nullable geometry shapes. Confirm +that automatic selection uses the faster mechanism for one costly decode at 50% survivors and for +two costly decodes at about 81% survivors. + +```bash +cargo bench -p vortex-spatial --bench null_strategies +``` + +The public benchmark names are shared with `develop`, so cross-revision comparisons do not need a +frozen benchmark-local implementation as their primary control. + +## The API in one screen + +`RowFn` is the author-facing function trait. A function gives the framework its exact argument +names, a conservative fallibility declaration, function-owned persistence, and a value-blind +dispatch over concrete input and sink types: + +```rust +impl RowFn for Example { + type Options = ExampleOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.example"); + *ID + } + + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + Ok(Some(encode(options)?)) + } + + fn deserialize( + &self, + metadata: &[u8], + session: &VortexSession, + ) -> VortexResult { + decode(metadata, session) + } + + fn dispatch( + &self, + options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + validate_options(options, args)?; + visitor.visit_prepared_into::<(InputA, InputB), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| { + *output = compute(lhs, rhs); + }, + ) + } +} +``` + +There are no argument or return witness types. The dispatched tuple is the argument declaration, +the sink owns the output representation, and the row result names the error behavior. Planning +runs the same dispatch as execution and checks the selected types against the function constants. + +## The extension boundary + +The framework is deliberately not sealed wholesale. Function authors need to add decode and output +primitives for their own scalar functions. Only the executor mechanics are closed. + +| API | Boundary | Why | +| --- | --- | --- | +| `RowFn` | open | Defines a scalar function and selects concrete execution types. | +| `InputElement` | open | Adds a new scalar decode primitive, including crate-local domain types. | +| `OutputElement` | open | Adds an ordinary one-value-per-row output primitive. | +| `OutputSink` | open | Adds a custom output representation or builder. | +| `RowVisitor` | sealed | Executor-owned dispatch mechanism with one supported implementation. | +| `ElementTuple` | sealed | Executor-owned tuple recursion, with built-ins through arity 12. | +| `SinkResult` | sealed | Executor-owned loop and error facts trusted by the blanket vtable. | + +`ElementTuple` being sealed does not prevent a function from adding a decode primitive. Implement +`InputElement` and use it inside one of the supplied tuples. Likewise, a function with two logical +outputs should define one `OutputSink` whose state has two fields. The framework does not need a +second tuple or composite-sink abstraction. + +The supplied `SinkResult` forms are: + +- `()` for infallible rows; +- `VortexResult<()>` for an error that must stop immediately; and +- `bool`, `u8`, `u16`, `u32`, or `u64` for error evidence OR-reduced after the loop. + +The unsigned evidence widths let each kernel choose a word no wider than its element type. That is +load-bearing for vectorization, particularly for checked unsigned multiplication. + +## Function-owned persistence + +Persistence belongs to the function ID, not to the Rust options type. `RowFn::Options` has no +serialization supertrait. The `RowFn::serialize` and `RowFn::deserialize` hooks have conservative +defaults, and registered functions override them when their existing wire contract requires it. + +This has three useful consequences: + +- two functions may reuse an options type while choosing different formats; +- a function may deliberately be nonserializable even if another function serializes the same + options type; and +- an unregistered helper such as `NumericBinary` needs no dummy persistence implementation. + +Tensor and geo functions keep their explicit existing formats. Do not introduce a blanket options +wire format or infer serializability from `Options`. + +## One sink abstraction + +`OutputSink` is the complete output contract. It owns the output dtype, allocation, row storage, +row lookup, length proof, and final array construction. `ElementSink` covers the common case. Its +row type is `&mut T`, so the closure writes with ordinary assignment. + +Custom sinks remain available for a real output shape that cannot use `ElementSink`. The unused +public `TensorSink` was removed. No current tensor row function returns tensor-valued rows, and a +90-line public runtime-shaped sink was not justified without a user. Add a custom sink when a real +function needs one, using one sink struct even when it owns several builders. + +Every current sink produces an all-valid child column. The blanket vtable can therefore derive the +function result validity from the input validities. Nullable row outputs remain out of scope. A +sink that emits its own nulls must change that derivation in the same change. + +`OutputSink::sink_dtype` must return a non-nullable dtype. `SUPPORTS_SKIPPED_ROWS` says whether +branch-and-skip may leave placeholder rows behind the result validity. `ERRORS_ARE_DEFERRED` says +whether the sink accepts accumulated error evidence at `finish`. + +## Dispatch and fallibility + +`dispatch` must be pure in `(options, args)`. It sees dtypes, not array values. Planning and +execution both call it, so value-dependent preparation belongs inside `visit_prepared_into`. + +The executor statically checks each dispatched visit: + +- the tuple arity equals `ARG_NAMES.len()`; +- a fallible decoder, early-error result, or deferred result implies `RowFn::FALLIBLE`; +- deferred evidence requires both `RowFn::FALLIBLE` and a sink with + `ERRORS_ARE_DEFERRED = true`; and +- the sink and result agree about their error contract. + +The implications are intentionally one-way. `FALLIBLE = true` is a conservative function-level +claim, while a particular dtype dispatch arm may be infallible. + +`prepare` must not be load-bearing for validation. Empty batches may bypass value preparation, and +the executor needs its safety and fallibility facts before it runs the closure. + +## Null execution policy + +The old public `NullHandling` enum and argument witness were removed. Authors do not select an +execution mechanism. The executor derives a private row policy from the dispatched input and result +types: + +- `Dense` may execute over garbage behind nulls and masks afterward; +- `DenseWithRetry` may execute densely, then retry valid rows when deferred evidence reports an + error; and +- `ValidOnly { filtered_decode_cost }` guarantees that the row closure sees only valid rows. + +An early-failing row or a decoder that is not dense-safe must use valid-only execution. A deferred +kernel may use dense execution because it writes a legal provisional value for every row. If only +garbage behind nulls reports an error, the valid-row retry discards it. + +Valid-only execution has two mechanisms. Filter-and-scatter shrinks inputs before decoding. +Branch-and-skip decodes the original batch and visits set bits from the conjoined validity mask. A +sink that does not support skipped rows automatically falls back to filter-and-scatter. + +The selector needs more than a boolean "decode shrinks" flag. Every `InputElement` declares an +additive `FILTERED_DECODE_COST`, defaulting to zero. `ElementTuple` sums the costs across arguments: + +- cost 0 always prefers branch-and-skip; +- cost 1 prefers branch-and-skip at 50% or more surviving rows; and +- cost 2 or greater prefers branch-and-skip at 85% or more surviving rows. + +This distinction comes from the x86 measurement in #9128. One nullable geometry input at 50% nulls +favored branching, while two independently nullable geometry inputs at 10% nulls each, about 81% +survivors, favored filtering. OR-ing a per-argument flag loses exactly that distinction. + +The values are still a coarse heuristic. There is no evidence yet to separate cost 2 from cost 3, +and the batch-size crossover has not been measured. `NullStrategy` remains only as a test-harness +seam for forcing a mechanism. Do not expose the private row policy as an author contract. + +## Performance and generated-code evidence + +The older Ryzen 9 7950X AVX-512 measurements remain the production-performance record in the +[#9128 follow-up](https://github.com/vortex-data/vortex/issues/9128#issuecomment-5151831802). They +also supplied the per-argument null-selection evidence above. + +The final API cleanup was checked separately against its parent, `53c51d803c`, by cross-compiling +the optimized `row_fn_executor` benchmark for `x86_64-apple-darwin` with `target-cpu=x86-64-v3`. +After normalizing symbol names and metadata, the vector and reduction blocks were identical for all +three executor shapes: + +- ordinary wrapping add through `ElementSink`; +- checked add with deferred evidence; and +- wrapping add through a custom sink. + +The wrapping loops retain 256-bit `<4 x i64>` loads, adds, and stores. The checked loop retains the +same vector loads and adds, derives overflow with vector xor/and/compare operations, accumulates +`<4 x i1>` with vector OR, and reduces after the loop. None of the vector bodies contains a call or +panic path. Scalar tails are unchanged. + +The production tensor benchmarks were also cross-compiled before and after the cleanup. Normalized +arithmetic sequences and counts match for `l2_norm`, inner product, and cosine similarity. Their +ordered floating-point reductions are scalar-unrolled in both revisions because LLVM preserves the +strict reduction order. The cleanup did not remove vectorization because those reductions were not +vectorized before it. + +Native Apple M4 Max timings used 65,536 rows, two alternating before/after runs, 100 samples, and a +0.5-second minimum per arm. RowFn median deltas ranged from 1.11% faster to 0.94% slower. Fastest +deltas stayed within about 0.17%, while specialized controls drifted by as much as 3.7% in their +medians. There is no measurable native regression from the API cleanup. + +This does not replace the required x86 runtime run above. Cross-target IR proves that the hot loop +shape survived, not that the revised null selector has the expected branch-predictor behavior on +x86. + +## Current implementation and checks + +The implementation includes production users in `vortex-array`, `vortex-tensor`, and +`vortex-spatial`. +`NumericBinary` is an unregistered `RowFn` used only for primitive arithmetic execution. Decimal +arithmetic keeps its existing path. The stable public-path benchmark baseline landed as #9136. + +The checks recorded for the final API state are: + +- 67 focused RowFn tests; +- 179 `vortex-tensor` tests; +- 230 `vortex-spatial` tests; +- `cargo +nightly fmt --all`; and +- full workspace clippy, with `PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1` because the host + `/usr/bin/python3` is 3.9 while the workspace requires the Python 3.11 stable ABI. + +The generated-code comparison and native timing evidence are described above and in the final +section of `STRICT_SCALAR_FN_RESEARCH.md`. + +## Review pass: what changed and what was deliberately left + +A review of the three parts (API, execution, implementations). **The author-facing API is +unchanged**: every proposal that would have altered it was backed out, for the reasons below, and +what landed is cleanup, corrected documentation, and test coverage. The emitted IR of every +`visit_prepared_into` monomorph is identical to the pre-review commit. + +API: + +- `InputElement::decode_null_tolerant` overrides that only restated the default were deleted from + the primitive, bool and `TensorRow` elements. `GeometryRow`'s override is the only real one. The + doc now says a dense-safe element should *not* override. +- `ElementTuple` now records why it carries arities past the widest function in tree: it is sealed, + so a downstream crate cannot add the one it needs, and an uninstantiated arity costs only its own + macro expansion. + +Execution: + +- `execute_filtered` and the forced-strategy test seam now share `resolve_validity`, so the mask + materialization and the all-true/all-false shortcuts cannot drift apart between them. +- The dense-retry path's comment was wrong and is corrected. It filters unconditionally because + `execute_dense` is not handed the `branch` closure, **not** because a deferred sink cannot skip + rows: `ERRORS_ARE_DEFERRED` and `SUPPORTS_SKIPPED_ROWS` are independent consts and a sink may + legally set both. + +Implementations: + +- `l2_norm_row` had two copies, in `l2_norm.rs` and `cosine_similarity.rs`. Cosine's prepared and + per-row arms must agree bit for bit, which only holds while both accumulate in the same order, so + the duplicate was an invitation to break exactly the property the comments defend. One copy now + lives in `utils.rs` beside the other shared tensor helpers. +- `CosineSimilarity::reduce_encoded` zips its three slices instead of indexing `0..len` three times + per row, and documents why it materializes where `InnerProduct::reduce_encoded` stays lazy (the + zero-norm guard is a conditional, not an arithmetic factor). +- `IndexedSourceExt::map_checked_into` was deleted from vortex-compute. `CheckedSink` replaced the + split value/evidence pass it served, and it had no caller left. +- `contains_route` and the workspace `geo` dependency both record that the table transcribes geo's + `impl_contains_from_relate!` and must be re-verified on a version bump. `geo` is pinned to + `=0.31.0`: a caret requirement would admit 0.31.x patches, which `cargo update` (or automated + lockfile maintenance) takes with no diff to review, and a patch is free to reshuffle the dispatch + without any API change. The agreement tests stay green wherever relate and the direct algorithm + agree, so the pin, not the suite, is what makes the coupling break only deliberately. + +Split out onto `develop` instead of landing here: + +- **The checked-arithmetic macro collapse.** `primitive.rs` on this branch and on `develop` both + carry four near-identical `CheckedArithmetic` bodies that differ only in `mul_failure`, so the + collapse into one `impl_checked_integer!` belongs on `develop` where every caller benefits. It is + on `claude/collapse-checked-arith-macros`. This branch's `primitive.rs` keeps its four bodies + until `develop` is merged, at which point the collapse arrives with it and the merge conflict is + a member deletion rather than two competing macro structures. +- **The `mul_failure` kernel tests.** The exhaustive 8-bit sweep and the 64-bit probe grid already + exist on `develop` from vortex-data/vortex#9210 and arrive with the same merge. + +Deliberately **not** done: + +- **No `DeferredElementSink`.** `CheckedSink` exists largely because `ElementSink` cannot name an + error at `finish`. A framework sink combining an element output with a type-level message would + remove ~100 lines per function, but there is exactly one deferred-error function. Build it when a + second appears, rather than copying `CheckedSink`. +- **No change to `reduce_encoded`'s probe semantics.** Hoisting the probe out of the strategy paths + and masking a full-length result looks like a simplification and is not one: + `normalized_readthrough_survives_null_rows` pins that a filtered input is no longer `Normalized`, + so which arrays reach `reduce_encoded` is load-bearing and differs per strategy. +- **No mixed-constant specialization without a failing benchmark.** The broadcast-index-mask + experiment regressed numeric constants by 4x to 7x on x86 and was removed. Keep the current + executor for the first PR. Add a specialized varying x constant or constant x varying loop only + after the clean branch has a stable regression against the current merge base. + +### Three API changes proposed, and why none of them landed + +All three were implemented, run against the suite, and backed out. None prevents a bug, and this +branch's open work is *settling* the API rather than churning it, so they belong in #9129 as +questions decided alongside the rest of the surface: + +- **Should `reduce_encoded` take an explicit `row_count`?** The filtered-count requirement is real + and easy to miss, but `args` are filtered to match, so `args[0].len()` is already both the natural + thing to write and correct. The parameter is documentation, and it costs every implementor a + signature change. What survived is the test: + `reduce_encoded_is_probed_before_and_after_filtering` pins that the rewrite is offered the + original arrays at full length and then the filtered ones at the surviving count. +- **Should `OutputSink::row_count_matches` become `rows_len`?** A length reads cleaner and lets the + executor name what it found. Against that, `row_count_matches` lets a sink fold in its own + invariants, which `SpreadSink` uses for its width check; narrowing it turns that into a panic. + Neither spelling prevents a bug. +- **Should the nullary path go?** A function with no inputs has no validity to lift, which is the + lifting's whole job. But `RowFn` would still give it sink allocation and dtype derivation, so + `random()` or `now()` is not obviously better hand-written, and the path is ~70 lines and tested. + +Trimming `ElementTuple` to arity four was proposed on the same reasoning and backed out for a +stronger one: the trait is sealed, so the arities are the only ones a downstream crate can ever +have. + +### Two changes this pass made and then reverted + +Both were proposed, implemented, reviewed, and backed out on evidence. They are recorded because +each is an attractive idea that a later reader will have again. + +**Making `CheckedSink` safe with `BufferMut::zeroed` costs 1.65 to 1.71x.** Replacing the +`MaybeUninit` storage removes an `unsafe set_len` and reads as a clear win, and `ElementSink`'s own +comment appears to bless it by routing a zeroable placeholder to `alloc_zeroed`. Measured, it is +not: allocate-zeroed-then-fill against allocate-then-fill, interleaved in one process over `u64` +outputs, ran **1.221x** slower at 8 KiB, **1.71x** at 64 KiB, **1.66x** at 512 KiB and **1.71x** at +2 MiB, stable to within 2% across two runs. `alloc_zeroed` does not avoid the write: below glibc's +mmap threshold `calloc` recycles a dirty chunk and memsets it, and above it every fresh page faults +on first touch. The row loop overwrites every slot regardless, so this is a duplicated pass over +the output of the hottest kernel in the system. + +Note the corollary, which is a real optimization nobody has taken: `ElementSink::with_capacity` +pays exactly this on every batch, and only branch-and-skip ever reads a placeholder back. A sink +that allocated uninitialized on the dense and filter paths would recover it. + +**Hoisting `OutputSink::SUPPORTS_SKIPPED_ROWS` into the plan is not sound as an optimization.** +#9130 records "avoid probing `reduce_encoded` twice when branch execution is unsupported" as a +follow-up. It reads as free, and is not, because the branch path probes `reduce_encoded` against +the _original_ arrays before it consults the sink, and that is the only probe that ever sees them +still encoded. Skipping the path early leaves such a function with only the filtered probe, whose +canonical arrays match no encoding fast path. For a function whose reduction is _defined_ to answer +differently from its row loop, which is exactly what `L2Norm` over `Normalized` is, that is a wrong +answer rather than a slow one. Nothing in tree is reachable today only because every `ValidOnly` +dispatch happens to use `ElementSink`. **#9130's follow-up should be struck, not implemented.** +`reduce_encoded_is_probed_before_and_after_filtering` now pins the two probes and their row +counts. + +### On measurement, and what the IR gate does and does not cover + +Wall-clock benchmarking of the row loops was attempted first and abandoned on evidence. Two runs of +the *same* baseline binary, pinned with `taskset -c 2`, 100 samples, disagreed by up to 4x +(`row_wrapping_add_nullable`: 198.8 us then 52.9 us median; `specialized_checked_add`: 185.5 us then +34.4 us). The 4-vCPU shared VM drifts more within a session than any effect being measured, which is +the same conclusion this branch already reached on a dedicated 7950X. + +The gate used instead is the emitted optimized IR of every `visit_prepared_into` monomorph in +`vortex-array`, profiled by vector width, reduction count, overflow-intrinsic survival and bounds +checks, then compared as a multiset before and after. Reproduce with: + +```bash +RUSTFLAGS="--emit=llvm-ir -C codegen-units=1" cargo rustc -p vortex-array --release --lib +``` + +**Its blind spot is worth stating, because it nearly landed a regression.** The IR of a row loop +cannot show an allocator call outside it, so the `BufferMut::zeroed` substitution above passed this +gate cleanly while costing 1.7x. An allocation-strategy change needs its own targeted A/B, which is +cheap to write and immune to the host drift above because both arms run interleaved in one process. +Use the IR gate for loop shape and a focused microbenchmark for anything the loop does not contain. + +## Remaining boundaries + +- Complete the required x86 production and forced-null-strategy benchmark run above before treating + the thresholds or overall performance as settled. +- Keep nullable outputs separate until the first real function can define the validity contract. +- Do not add another sink composition abstraction. Put multiple builders in one custom sink. +- Do not add a general runtime-shaped sink until a production function needs one. +- Keep pattern compilation and other state shared across rows outside `RowFn` when it cannot be + represented as batch preparation. +- Use emitted optimized IR as a gate for numeric changes near LLVM's vectorization boundary, then + use the stable #9136 benchmark names for runtime confirmation. + +## Repository rules for the next agent + +Follow `AGENTS.md`. Keep public APIs small, run narrow checks before workspace-wide checks, and +report blocked checks separately from passing ones. Preserve unrelated working-tree and staging +state. Every commit must include the required `Signed-off-by` trailer. diff --git a/STRICT_SCALAR_FN_RESEARCH.md b/STRICT_SCALAR_FN_RESEARCH.md new file mode 100644 index 00000000000..b5125ffc953 --- /dev/null +++ b/STRICT_SCALAR_FN_RESEARCH.md @@ -0,0 +1,1830 @@ + + + +# A layered authoring API for strict scalar functions + +**Status: historical design record, with the final API review recorded at the end.** This document +keeps the experiments in the order they happened, including APIs and ports that were later removed. +The current architecture is one `RowFn` authoring trait, private lifting, one sink-backed +`RowVisitor::visit_prepared_into` primitive, and a deliberately open input/output vocabulary. Read +[`SCALAR_FN_HANDOFF.md`](SCALAR_FN_HANDOFF.md) for orientation, then the final section here before +using an earlier sketch. + +> **Later architecture decisions:** `StrictScalarFnVTable`, the columnar ports, returning visits, +> both witness types, `PersistableOptions`, the public `NullHandling`, the aggregate decode-shrinks +> flag, and the unused `TensorSink` were deleted. Framework-only visitor, tuple, and result traits +> are sealed. `InputElement`, `OutputElement`, and `OutputSink` remain open so functions can add +> their own decode and output primitives. Sections below remain the evidence that led to those +> decisions, not the API to implement. + +--- + +## Current benchmark and codegen record + +The authoritative current comparison is the +[x86 AVX-512 re-measurement on issue #9128](https://github.com/vortex-data/vortex/issues/9128#issuecomment-5151831802). +It records the machine, exact refs, stabilized governor, two-run fastest and median results, control +limitations, geo fix, adaptive-null diagnostics, and native LLVM IR/assembly in folded sections. +It supersedes every older shared-VM or pre-#9076 figure in these notes for claims about the current +branch versus `develop`. + +The run used candidate `d293d3cdd59e` plus the recorded geo bbox widening, baseline +`876996fe7846`, and a Ryzen 9 7950X pinned to CPU 4 with the TSC timer and performance governor. +The conclusions that survive into the implementation plan are: + +- sink-only checked add is 1.018-1.226x faster by median than its benchmark-local specialized + control, depending on constant and null shape; +- cosine is 1.40-30.13x faster than current develop; +- prepared overlapping `contains` is 8.60-8.77x faster by median, while the widened bbox gate + restores disjoint polygons to parity with #9076; +- point/constant geo still has real 8.6-14.2% and 10.9-13.2% median regressions; +- `BytesLen` is 1.410-1.411x faster by median on long strings and 1.097x on short strings; +- the global 75% survivor threshold mispredicts both one-input/50%-null and + two-input/10%-null geo cases, so adaptive selection needs element/arity-aware cost data; +- checked-add codegen has AVX-512 error-word accumulation and post-loop vector reduction with no + per-row error branch; current `l2_norm` remains a strict-order scalar reduction. + +The stabilized cosine median ratios preserve the shape and width dependence instead of collapsing +the result into one headline range: + +| shape | width 2 | width 32 | width 256 | +| --- | ---: | ---: | ---: | +| column x column | 5.77-5.79x | 1.93-2.19x | 2.54-2.58x | +| column x constant | 12.44-12.48x | 28.87-28.96x | 30.08-30.13x | +| column x extension constant | 3.05x | 1.40-1.41x | 1.77-1.78x | + +The final geo median ratios, including the bbox patch, are: + +| predicate and shape | develop / row branch | +| --- | ---: | +| contains, column x column points | 0.951-0.964x | +| contains, column x column polygons | 0.999-1.003x | +| contains, constant x points | 0.876-0.921x | +| contains, disjoint polygons | 0.993-0.997x | +| contains, overlapping polygons | 8.60-8.77x | +| intersects, column x column polygons | 0.983-0.995x | +| intersects, points x constant | 0.884-0.902x | +| intersects, disjoint polygons | 1.011-1.019x | +| intersects, overlapping polygons | 1.011-1.023x | + +Here, as above, ratios greater than 1x favor the row branch. The issue comment contains the paired +fastest and median observations rather than only these compact ranges. + +The historical measurements below remain because they explain design decisions and experiments made +while building the prototype; they are not the current before/after performance record. + +### Later broadcast-index-mask experiment + +Commit `ad24700088` tried to preserve a varying neighbor's decoded slice when another input was +constant. Every argument exposed `(Varying, mask)`, where the mask was `usize::MAX` for a varying +column and `0` for a one-row constant, and the fallback loop indexed each input with +`index & mask`. The all-varying loop was unchanged. + +The design regressed the numeric mixed-constant cases. The comparison used baseline `fed7038` and +candidate `edb3953`, with `RUSTFLAGS="-C target-feature=+avx2"` at both revisions: + +| benchmark | `fed7038` | `edb3953` | result | +| --- | ---: | ---: | ---: | +| `add_i64_constant` | 9.5-9.7 us | 37-71 us | approximately 4x to 7x slower | +| `sub_i64_constant` | 9.4-9.7 us | 37-45 us | approximately 4x slower | +| `mul_i32_constant` | 10.7-11.1 us | 42-43 us | approximately 4x slower | +| `add_i64_nonnull` | 11.1 us | 11.2 us | parity | +| `mul_i32_nonnull` | 13.9 us | 13.4 us | parity | + +The x86 report did not record the CPU, timer, pinning, or fastest and median values separately, so +these wall-clock values diagnose the code-generation failure rather than satisfy the release gate. +CodSpeed reported the same direction at a smaller magnitude: `add_i64_constant` 31.34%, +`sub_i64_constant` 32.38%, and `mul_i32_constant` 46.24% slower. + +The generated assembly kept two bounds checks per row and performed scalar loads. The runtime mask +made each varying index non-affine, so LLVM could not prove it in bounds or vectorize the loop. The +previous `ArgColumnKind` match was loop-invariant, which allowed LLVM to unswitch the numeric loop +into constant-pattern variants. The experiment optimized the branch count and discarded the +information that enabled vectorization. + +The commit was removed from `ct/row-fn` history. The clean integration head is `ea58061b5d`. Two +unpinned Divan runs on an Apple M4 Max, with 41 ns timer precision, restored the constant medians to +8.71-8.73 us for `add_i64`, 8.79-9.00 us for `sub_i64`, and 5.71-5.75 us for `mul_i32`. These +values prove that the mask regression is gone. They are not an x86 comparison against current +`develop`. + +Do not reintroduce a runtime mask or another runtime-shaped per-argument source. A later +mixed-constant optimization must monomorphize the loop over the constant pattern and must first be +justified by a stable benchmark against the current merge base. + +--- + +## The design in one screen + +```text +RowFn ──────────blanket──▶ StrictScalarFnVTable ──────blanket──▶ ScalarFnVTable +(row at a time, types (null / constant / validity (full control) + chosen per batch) lifting for a columnar kernel) +``` + +Two authoring traits, one for each axis a strict function actually varies on, plus a third axis (*how +a row is typed, and how its output is delivered*) factored into an open element and sink vocabulary that +neither trait mentions. + +### `StrictScalarFnVTable`, the null/validity lifting + +Write the structural metadata plus one **columnar** kernel that ignores validity. A blanket impl +derives: + +- `is_strict = true`, and a mirrored `validity` a kernel can answer with the conjunction of its child + validities when it never turns a wholly non-null row into a null (see + [Strictness is not totality](#strictness-is-not-totality)), so the planner knows which rows are null + without executing the function. +- `return_dtype` = `return_element_dtype` widened to nullable iff any input is nullable, so the + strictness dtype contract holds by construction rather than per function. +- `execute` = the shared cases before the kernel runs: a null-constant input short-circuits to an + all-null constant, all-constant inputs evaluate one row and broadcast, and partially-null inputs + are handled per `NullHandling` (`Dense` masks after a full pass, `Filter` filters then scatters). +- Options serde, from `PersistableOptions` on the options type. + +This is the layer for a function whose kernel is columnar rather than row-at-a-time: `not` (one `!` +per 64-bit word), `list_length` (a difference of offset buffers), `list_sum` (a grouped accumulator over +the elements child). See [Why three concepts and not fewer](#why-three-concepts-and-not-fewer) for why it +cannot be folded away. + +### `RowFn`, one row with element types chosen per batch + +Name a witness argument tuple and return type, then in `dispatch` pick the concrete element types for +a batch and hand the framework a row closure through a rank-2 visitor. A blanket impl derives the +whole `StrictScalarFnVTable` from it. When the element types are fixed, `dispatch` is a single +`visit` at those types. When one ID spans several widths (`l2_norm` accepts f16/f32/f64), `dispatch` +matches on the input dtypes and visits at the chosen width. + +Everything structural follows from the argument tuple and return type: arity, per-argument dtype +validation, the output dtype, null handling, and fallibility. There is nothing for an implementor to +declare twice or get wrong, because the framework reads it off the types (see +[Properties, not conventions](#properties-not-conventions)). A constant operand is decoded once and +read at stride 0, so a broadcast argument costs one decode rather than one per row. + +Output takes one of two forms, chosen per visit. `visit` takes a closure that **returns** an +`OutputElement`, one owned value per row whose dtype is a property of its Rust type. `visit_into` takes +one that **writes** into an `OutputSink`, allocated once per batch knowing the output dtype and handing +out a place to write. Orthogonally, `visit_prepared` runs a once-per-batch prepare step over the +element values of whichever operands are constant for the batch, and threads its result to every row +by shared reference; plain `visit` is that with unit state (see +[Constant compute](#constant-compute-the-last-quadrant-of-the-lifting)). The sink carries what an owned per-row value cannot: `l2_denorm` writes each row +into a slice of one flat buffer, so its output width comes from the arguments and it allocates once +rather than per row. The executor holds the sink and passes the handle in, so the closure stays `Fn` +and the returning path pays nothing. + +Note that `RowFn` does not *require* totality, it just cannot currently express its absence: both output +forms build an all-valid column, so a row kernel has no way to say "this row is null". An +`impl OutputElement for Option`, or a sink that can push a null, would lift that, at the cost of +revisiting the `validity` law that reads the output validity off the inputs. No function needs it yet, so +it is not there. + +### The element vocabulary, how a row is typed + +`InputElement`, `OutputElement` and `OutputSink` are open traits. A `NativePType`, `bool`, `Bytes` (a +resolved `&[u8]`), and `BytesLen` (a length read from a view without resolving it) ship in the framework, +and `vortex-tensor` adds `TensorRow`, reaching through the extension wrapper into flat storage, plus +`TensorSink` on the output side, in its own crate. Adding `&str`, decimals, or a list row is one impl +that every row function gains, with no framework change. + +--- + +## Why three concepts and not fewer + +The standard applied here: every trait, and every member of every trait, has to have a purpose +nothing else can provide. Testing each against that standard is what the bulk of this research was. + +### `RowFn` and the witnesses are forced, not chosen + +A scalar function's *signature*, meaning its arity and fallibility, is a property of +`(function, options)` with **no input dtypes**: `ScalarFnVTable::arity(&self, options)` and +`is_fallible(&self, options)`, and `ScalarFnSignature` above them, take none. So any framework that +derives arity and fallibility from element types has to be able to name element types *without seeing +dtypes*, which is exactly what `ArgsWitness` / `RetWitness` are. Because `dispatch` *does* see dtypes +and could choose otherwise, some check has to tie the two together, which is the compile-time witness +check below. This cost is not a consequence of the rank-2 encoding: **any** design that derives a +dtype-free signature from per-batch types pays it. + +A previous iteration made the width choice a generic-associated-type family generated by a +`row_family!` macro. Rust cannot abstract over a GAT's bound (`type Args` is +rejected), so that approach needed a trait *and* an adapter per width class, hand-written or +macro-stamped. The rank-2 visitor sidesteps the limit rather than writing around it: the kernel owns +the width `match`, where `T: Float` appears literally inside a `match_each_*_ptype!` arm, and the +framework method `RowVisitor::visit` is generic only over bounds it +owns. The macro, its family traits, and its generated adapters are all deleted. Note that `dispatch` +is not even per-*width*: it can pick different element *kinds* per dtype, which no +bound-parameterized family could. + +### `ElementwiseFn` was not forced, so it is gone + +An earlier revision had a third trait, `ElementwiseFn`, for the fixed-element-type case: name `Args` +and `Ret`, write `apply`. It read cleanly, but it failed the standard. `RowFn` already covers the +fixed case (the dispatch is a single constant `visit`), so `ElementwiseFn` bought roughly seven lines +on exactly one production function (`byte_length`) at the cost of 114 framework lines and a third +link in the blanket-impl chain. The probes settled it: of the functions examined, `not` and `list_sum` +turned out not to be row functions at all, and `list_length` needed the encoding-aware +`reduce_encoded` hook that `ElementwiseFn` never exposed. So the constituency I expected it to have +never materialized, and it is deleted. `byte_length` writes a two-line `dispatch` instead. + +The one-trait-with-defaults alternative (a single `RowFn` with `dispatch` defaulted to visit the +witnesses and `apply` defaulted to `unimplemented!()`) was rejected because it converts a compile +error into a runtime panic: a type implementing neither method compiles, registers, and answers +signature queries with a plausible shape, then panics on first execution. `dispatch` is therefore +required. + +### `StrictScalarFnVTable` cannot be folded into `RowFn` + +`RowFn`'s type surface is *closed*. The output dtype is `OutputElement::element_dtype()`, drawn from +the finite set of `OutputElement` impls, `ElementTuple` exists only for arities 1 to 3, and the loop +is one `apply` per row. Three whole classes of strict function are therefore inexpressible as a +`RowFn` at any cost: + +- **Output dtype outside the element set.** `ext_storage`'s output is an extension array's storage + dtype, so `vortex.st.box` is a struct and `vortex.uuid` is a `FixedSizeList(u8,16)`. + Zone-map pruning in `vortex-spatial` calls `ext_storage` on an `st.box` statistic, and a + row-function port breaks it at plan time. +- **Variadic arity.** `merge` and `select` take an unbounded number of children, while `RowFn` fixes + `Arity::Exact(n <= 3)`. +- **Sub-row-granular kernels.** `not` negates one 64-bit word at a time, so a row loop over `bool` is + ~64x the memory traffic and, measured, 406x slower at a 64Ki batch (see + [Measurements](#measurements)). + +So the middle layer has a genuine, disjoint constituency: `not`, `list_length`, `list_sum`, and +prospectively `select`, `merge`, `json_to_variant`. "Just a visitor" collapses three concepts to two +rather than to one. + +### Every remaining member earns its place + +A member-by-member audit, with call sites found by grep rather than by guess, turned up nothing +deletable. The non-obvious cases are worth recording: + +- **`RowVisitor::Out`** is what lets one `dispatch` `match` serve both plan time (`Out = DType`, + validate and name the output dtype) and run time (`Out = ArrayRef`, decode and run the loop). The + alternatives, a `{DType, ArrayRef}` enum unwrapped at each site or two separate dispatch hooks, + either add unwrap-panics or duplicate the width `match` in every width-polymorphic function with no + compiler check that the two copies agree. +- **A plan-time visit is unavoidable.** `l2_norm` declares `RetWitness = f64` but dispatches over + f16/f32/f64, so the output dtype read off the witness would be wrong for two of three widths. Also + `TensorRow::validate` rejects an `f32` column against an `f64` witness, and the visit is what + gives cross-argument uniformity for free (`int_max(i16_col, i64_col)` is rejected by + `(T, T)::validate`, not by any `dispatch` body, which only inspects `args[0]`). +- **`ApplyResult` distinct from `OutputElement`** is what lets one trait serve both infallible + (`Ret = f64`) and fallible (`Ret = VortexResult`) kernels without a wrapper. `f64` cannot be + simultaneously fallible and infallible, so the fallibility bit lives on the return *shape* rather + than on the element. + +--- + +## Properties, not conventions + +The framework's real value beyond line count is that two invariants an implementor used to have to +get right are now derived from the types, so an unsound combination cannot be written. + +### Null handling follows from the arguments and the return type + +`NullHandling::Dense` runs the kernel over every row including those behind nulls, then masks. It is +cheaper than filtering and the only option that leaves inputs at their original encoding, so it is +right whenever it is sound. Soundness needs two things, every argument readable behind a null row and +an infallible computation, and both are already visible in the types: + +```rust +const fn row_null_handling() -> NullHandling { + if A::DENSE_SAFE && !row_is_fallible::() { NullHandling::Dense } else { NullHandling::Filter } +} +``` + +Whether a dense read is safe is a property of the *element*, not of the function: reading a whole +value out of a flat buffer is safe (`NativePType`, `bool`, `TensorRow`, `BytesLen`), while following a +stored offset into a data buffer is not (`Bytes`), because arrays only validate the views of their +*valid* rows. This caught a real bug in this branch's own `byte_length`, see +[Problems to extract](#problems-to-extract-onto-develop). + +### Fallibility comes from the return type *and* the element decode + +A function is fallible if its computation can fail (`Ret = VortexResult`) **or** if decoding an +argument can fail on legal data. The second source is real and was missing: `geo_distance`'s row +computation cannot fail, but parsing WKB bytes into a geometry can, for a *valid* row holding +malformed bytes. So `InputElement` carries `DECODE_FALLIBLE`, and fallibility is the disjunction: + +```rust +const fn row_is_fallible() -> bool { A::DECODE_FALLIBLE || R::FALLIBLE } +``` + +`is_fallible` gates dict-value pushdown (`arrays/dict/compute/rules.rs`), which speculatively +evaluates a function over *unreferenced* dictionary values, so a function that under-reports +fallibility fails a query on rows it never needed. + +### The witness is checked at compile time + +Arity, dense-safety and fallibility must not vary between the choices `dispatch` makes, because the +framework acts on them before dispatching. Since (with `ElementwiseFn` gone) *every* function names +its element tuple twice, once as `ArgsWitness` and once in the `visit`, the check that the two agree +is load-bearing, and it is a compile-time `const` assert inside each visit: + +```rust +const fn assert_witness_agrees() { + assert!(A::ARITY == ::ARITY, "…"); + assert!(A::DENSE_SAFE == ::DENSE_SAFE, "…"); + assert!(row_is_fallible::() == row_is_fallible::(), "…"); +} +``` + +Monomorphizing any dispatch arm evaluates it, so even a `match` arm that never runs at a given width +is checked, and a disagreement fails the build pointing at the exact `visit::<…>` call. It compares +the raw arity/dense-safety/fallibility rather than the derived `NullHandling`, which collapses +dense-safety and fallibility together and would miss an arm that flipped both. A `compile_fail` +doctest pins that a lying witness does not compile. This replaced a runtime check that ran three +times per array (plan, execute, deserialize). + +--- + +## Strictness is not totality + +This is the finding that decides what the middle layer may derive. Note that +[#9033](https://github.com/vortex-data/vortex/pull/9033) reached the same conclusion independently and +has since landed, so this section is no longer the argument for the finding, only for the API that +follows from it. + +Before #9033, the `is_strict` documentation stated the validity-equivariance law, +`f(…, mask(aⱼ, m), …) == mask(f(…, aⱼ, …), m)`, and then asserted as "consequence 1" that output +validity is the conjunction of input validities. **Consequence 1 does not follow from the law.** It +needs an extra premise: that the kernel never turns a wholly non-null row into a null. #9033 replaced +that equality with a one-sided bound, `valid(f(a₁, …, aₖ)) ⊆ valid(a₁) ∧ … ∧ valid(aₖ)`, which is the +vocabulary this branch uses. `docs/strictness-and-validity-pushdown.typ` proves the law and the +null-propagation reading are the same property, and separates what does not follow from either. + +`list_sum` is the counterexample. Summing a valid *empty* list yields null. It still satisfies the law +(a null it introduces at a valid row appears identically on both sides of the equation and cancels), +so it is genuinely strict, but its output validity is *narrower* than its input validity. + +Two properties, then, not one: + +| property | what needs it | +| --- | --- | +| **strict** (null propagation, equivalently validity equivariance) | every validity push-down, the thing we actually want | +| **total** (non-null in implies non-null out) | upgrading the `⊆` bound to `=`, so validity is precomputable | + +The old blanket impl derived `validity = union_child_validities` for *every* implementor, which needs +totality while the trait only requires strictness. Every current implementor happens to be total, so +nothing was broken, but a partial function joining the layer would get a `validity` that contradicts +what it computes: `arr.validity()` would report all-valid while `arr.execute()` yields the null, since +`ValidityVTable::validity` evaluates the derived expression. `list_sum` was about to be +exactly that, and is now ported onto the layer as the first non-total member. + +#9033 says a function satisfying the stronger equality "can advertise that through +`ScalarFnVTable::validity`". That is the same idea as `is_total`, moved from a hand-written method to a +boolean, because a blanket impl cannot hand-write `validity` per function: it needs the property as +data in order to decide whether to derive one. + +The fix needs no new property. `validity` is mirrored on `StrictScalarFnVTable` alongside `reduce`, +defaulting to `None`, and a kernel that satisfies the equality answers it with +`union_child_validities`. The unsound direction is the one that now takes work, and the safe default is +what a function gets for free. + +An earlier revision of this branch instead added an `is_total` method and derived `validity` from it. +That was strictly worse: it introduced a concept the codebase did not have, in order to compute +something a function can just say directly. It is gone. The `RowFn` blanket impl answers `validity` +for every row function, justified by its own output vocabulary (no `OutputElement` is nullable, so no +row kernel can introduce a null), which keeps the row layer at zero boilerplate. + +Note that strictness rather than totality gates membership either way: `is_null` is total but +disqualified, because it inspects validity and so does not propagate nulls. That is also why the trait +is not called `TotalFnVTable`. + +> **A related latent issue, deliberately not fixed here.** Four functions declare `is_strict = true` +> and are strict-but-not-total: `get_item` (a nullable field under a non-null struct), `mask`, +> `variant_get`, `geo_envelope`. None is broken today, since `get_item` leaves `validity` at the +> default and `mask` overrides it correctly, but any that grows a conjunction-shaped `validity` +> derivation would be wrong. This predates the branch and belongs in its own investigation. + +--- + +## Problems to extract onto develop + +The framework surfaced three problems that are not really about the framework. Each is filed +separately and I think each should land as its own PR rather than riding in on this one. Note that +none of them is a live miscompute on `develop` today, which is worth saying plainly, because the +branch's own commit messages describe fixes to *this branch's* code. + +1. **Strict-but-non-total validity derivation ([#9091]).** The `is_strict` documentation presents + totality as a consequence of strictness when it is an independent premise (see above). Nothing + derives validity from `is_strict` automatically, so nothing is wrong today, but the doc invites the + next strict-but-partial function to write `validity: union_child_validities` and be silently wrong. + **Superseded by [#9033], which lands the documentation correction on `develop`.** This branch needs + nothing beyond that, since it now mirrors `validity` rather than deriving it from a property. + +2. **Views behind null rows are unvalidated ([#9090]).** `VarBinViewArray::validate_views` only + validates the views of *valid* rows, so a legal array can hold a view behind a null row naming a + buffer that does not exist, and resolving it densely panics (`index out of bounds: the len is 1 but + the index is 9`). On this branch, expressing byte length as "a function of the row's bytes" quietly + changed *what gets decoded* and hit that panic. The fix here reads the length out of the view + (`BytesLen`) and never resolves the row, and + `test_byte_length_ignores_unresolvable_views_behind_nulls` pins it (verified to panic without the + fix). `develop`'s `byte_length` was already immune, since it also read `view.len()`, so the + extraction is that regression test rather than a code change. The doc half is also covered by + [#9033], which deletes the dense-evaluation "consequence 2" outright rather than narrowing it. That + leaves `InputElement::DENSE_SAFE` as the only place the licence is written down, per element rather + than as a blanket claim, which is where it belongs. + +3. **Bit-at-a-time bool packing ([#9092]).** `OutputElement for bool` used `BitBuffer::from_iter`, + where the `Vec` is already owned and contiguous so `BitBuffer::from` routes to the + multiversioned SIMD packer. Measured **6.6 to 7.9x faster** on the packing step, for every + bool-returning row function. Note that `OutputElement` only exists on this branch, so the + develop-side instance of the same pattern is a different call site: + `encodings/sequence/src/compute/compare.rs` builds an n-bit result with a per-row predicate when it + already knows the single set index. I have not benchmarked that site. + +[#9033]: https://github.com/vortex-data/vortex/pull/9033 +[#9090]: https://github.com/vortex-data/vortex/issues/9090 +[#9091]: https://github.com/vortex-data/vortex/issues/9091 +[#9092]: https://github.com/vortex-data/vortex/issues/9092 + +--- + +## Audit: can the four `StrictScalarFnVTable` impls really not be `RowFn`? + +There were exactly four in production when this audit ran. Auditing each against the two questions that +matter, rather than repeating the earlier verdicts, **not one of them was structurally impossible**. Every +"cannot" in this document was really "cannot with the trait signed as it is today". One of the four, +`l2_denorm`, has since moved onto `RowFn`, so three remain. Recording the distinction because it is the +difference between a limit and a decision. + +| function | signature expressible? | kernel row-shaped? | what it would take | +| --- | --- | --- | --- | +| `not` | **yes**, `(bool,) -> bool`, both elements exist | **no** | nothing. It can be a `RowFn` today and should not be: `!bits` is one `!` per 64-bit word, in place when unshared, against 16k closure calls and a `Vec` repack | +| `list_length` | output is a fixed `U64`; input needs a `ListLen` element | **no** | one new element. Still should not: the answer is a child array or one constant | +| `list_sum` | output is one number per row, so nearly: only the *nullability* is unexpressible | **no** | `impl OutputElement for Option` and a list element, but the kernel is the real blocker | +| `l2_denorm` | **yes, now**: an `OutputSink` names its dtype from the arguments | yes, per-row scaling | **done**, see below | + +**A varying output dtype was already supported, and listing it as a blocker was wrong.** `dispatch` +chooses element types per batch and `return_element_dtype` routes through it, so `R::Out::element_dtype()` +is already answered per dispatch arm. `l2_norm` relies on this today, visiting `::<(TensorRow,), T>` +with `T` ranging over the float widths. The compile-time witness check pins only arity, dense-safety and +fallibility, deliberately leaving the output type free to vary. What `l2_denorm` needed was different and +narrower: its output dtype depends on the input *dtype* in a way no choice of element type can express, +because the extension dtype carries a shape. That is what `OutputSink::sink_dtype(args)` supplies. + +**`list_sum`'s output side is the easy part; its kernel is not.** One number per row means it needs only +a nullable output element, no write-into-buffer machinery. But `execute_strict` is not a per-row sum: it +builds a `GroupedAccumulator` over `Sum`, calls `accumulate_list`, and then `mask_empty_lists` computes +per-group emptiness with `count_range` popcounts, with all-true and all-none fast paths and an early +return when nothing needs masking. Porting it to a row loop would hand-roll the shared aggregate +framework, lose the overflow modes that `NumericalAggregateOpts` selects, and trade SIMD popcounts for +per-row checks. That puts it in the same category as `not`: expressible, and worse. + +So `l2_denorm` was the only one of the four whose kernel actually wants to be a row loop, which is why it +was the right first target despite needing the larger output-side change. + +Two readings follow. + +**The honest framing is "can, and here is whether it is worth it."** For `not` and `list_length` the +answer is a flat no on performance grounds, and those are settled. For `list_sum` the answer is +yes-with-changes, and the change it wants is a nullable output, which the sink could supply but which the +`validity` law argues against (see below). + +**`l2_denorm` was the one worth doing, and it is done.** Its kernel genuinely is per-row scaling, and +it carried the `unsafe` the other three tensor ports removed. What it needed was a second visit method +whose closure *writes* its row instead of returning it, generalized to an `OutputSink` rather than +hardcoding `&mut [T]`, because the same mechanism covers three gaps recorded separately in these notes: + +- **runtime-shaped output**: the sink is a preallocated flat buffer and the per-row handle a + `&mut [T]` slice of it, so `l2_denorm` allocates once per batch rather than once per row. This is what + shipped. +- **`str -> str` without the double copy**: the sink is one growing byte buffer plus views, and + `upper`/`lower`/`replace` push into it. Strictly better than the `Cow` output element considered + above, which still copies each row once. Not built, but the trait admits it unchanged. +- **nullable output**: a sink *could* push a null, which would remove the need for + `impl OutputElement for Option` as a separate patch. Deliberately **not** taken: both output forms + build an all-valid column today, and that is exactly what lets the blanket `validity` return + `union_child_validities`. Adding nulls to either form has to come with that law being revisited. + +### What shipped + +```rust +pub trait OutputSink: 'static + Sized { + type Row<'a> where Self: 'a; + fn sink_dtype(args: &[DType]) -> VortexResult; + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult; + fn row(&mut self, index: usize) -> Self::Row<'_>; + fn finish(self) -> VortexResult; +} + +fn visit_into( + self, + apply: impl Fn(A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult; +``` + +**The executor threads the sink, not the closure**, so `apply` stays `Fn` and the existing `visit` pays +nothing. That was the design constraint, not an accident: relaxing `visit` itself to `FnMut` measured at +8 to 11% (see the `like` discussion), and a handle passed in per row avoids captured mutable state +entirely. Measured after the fact, `l2_norm` is unchanged at 69.05 µs against the 69.44 µs recorded +before the sink landed. + +**Step 1 of the earlier plan turned out to be unnecessary.** The plan called for widening +`OutputElement::element_dtype()` to take `args`. It never happened, because `sink_dtype(args)` puts the +argument-dependence on the *sink* instead, leaving all three existing `OutputElement` impls untouched. +That is the better split: an element's dtype genuinely is a property of its Rust type, and only the +thing that needs the arguments asks for them. + +**The `RetWitness` split resolved as predicted.** It carried two roles, *what dtype* and *is it +fallible*, and only the second is readable before `dispatch` picks a form. So `RowResult` now holds just +`const FALLIBLE`, with `ApplyResult: RowResult` adding the output element and `SinkResult: RowResult` +adding nothing but the error, and `RowFn::RetWitness` is bounded by `RowResult`. A returning dispatch +names `f64` or `VortexResult`; a writing one names `()` or `VortexResult<()>`. Coherence permits +this: `impl RowResult for ()` does not overlap `impl RowResult for T` because +`(): OutputElement` does not hold and no downstream crate can make it hold, the same negative reasoning +the pre-existing `ApplyResult` impls already relied on. + +**A new limit, worth naming.** `sink_dtype` sees the input dtypes but **not** the function's options, +because `OutputSink` does not know the `RowFn`'s `Options` type. A function whose output dtype depends +on an option value therefore still drops to `StrictScalarFnVTable`, whose `return_element_dtype` sees +both. Nothing in the repository needs it, and threading options through later is additive. + +### Results + +`unsafe` in `l2_denorm.rs` went from 8 blocks to 6. The two removed are the memory-safety ones on the +kernel path: `FixedSizeListArray::new_unchecked` in the constant-norms path, now `try_new` (the norm is +cast to the element dtype first, so the product stays non-nullable and the check passes), and +`PrimitiveArray::new_unchecked` in `build_tensor_array`, now `new`. That second one is an independent +cleanup rather than something the port forced. + +The 6 remaining are not of that kind and are not the row layer's business: four are calls to +`L2Denorm::new_array_unchecked`, an `unsafe fn` whose contract is the *semantic* unit-norm invariant and +not memory safety, and two are buffer pushes inside `normalize_as_l2_denorm`, a helper that builds the +normalized child and is not a scalar function at all. + +**Performance: the sink is faster than the kernel it replaced**, which was not the expected outcome. +`vortex-tensor/benches/l2_denorm.rs`, `fastest` column, both configurations run twice, 16384 rows, +non-nullable. The control implements `StrictScalarFnVTable` with the pre-port body, so it shares the +strict lifting and the gap is the row layer alone: + +| width | sink | pre-port kernel | ratio | +| --- | --- | --- | --- | +| 2 | 88.02 / 88.16 µs | 60.19 / 60.45 µs | sink 1.46x slower *(since fixed, see below)* | +| 32 | 482.0 / 515.5 µs | 1.175 / 1.014 ms | sink **2.1x faster** | +| 256 | 10.23 / 10.43 ms | 20.41 / 22.48 ms | sink **2.0x faster** | + +The likely cause of the win is that the pre-port kernel collected a `flat_map` over rows into a fresh +`Buffer`, and `flat_map` is not `TrustedLen`, so that `collect` grew the buffer with a capacity check +per element. The sink allocates once with `BufferMut::zeroed` and each row writes a slice of it, which +vectorizes. The zeroing is not a separate pass at these sizes, since large allocations come back zeroed +from the allocator. This is a hypothesis consistent with the width scaling rather than something +profiled. + +Width 2 showed the same regression as `l2_norm`'s, and for the same reason: both read tensor rows through +`TensorRow`, whose `get` re-derived a typed slice per row. Typing the column at decode time took +`l2_denorm` from 88.0 µs to **48.9 µs** at width 2, ahead of this control rather than behind it. See +[the like-for-like comparison](#the-like-for-like-comparison-and-the-per-row-cost-that-was-hiding-in-it) +for the measurement and for the wrong diagnosis it corrects. + +The constant-norms fast path moved to `reduce_encoded`, which sees the argument arrays before the row +loop. It keeps both of its cases (unit norms return the normalized child untouched, any other constant +rewrites the storage elements through one multiply), and it still fires for a filtered batch because +filtering a constant yields a constant. + +**Two visit methods do not cover everything, and it is worth being precise about the residue.** They +cover every function whose output is *computed* per row, returned or written. What stays columnar is +output that *aliases* its input, since `trim` and `substring` want to keep the input's data buffer and +rewrite only views, copying nothing, and a sink still copies bytes into itself. Likewise kernels whose +natural unit is not a row (`not`'s word-at-a-time negation, `binary`'s slice kernels) gain nothing. + +The sink is also what a `str -> str` string library needs. After reclassifying `L2Denorm` as an +encoding, that string library becomes the prospective first production user rather than a second +one. The experiment still demonstrates that the generic sink can carry runtime-shaped and +builder-backed outputs without making the returning path pay, but it should not be stabilized from +the tensor experiment alone. + +--- + +## Constant compute: the last quadrant of the lifting + +The lifting's constant handling was complete on the data side and absent on the compute side. A +null-constant input short-circuits, all-constant inputs fold to one row, and a constant operand is +decoded once and read at stride 0. What nothing owned was kernel computation that depends only on a +constant argument: `cosine_similarity(rows, query)` with a broadcast query re-accumulated +`norm(query)`, an O(width) pass plus a sqrt, once per row, and the geo predicates rebuilt the +constant side's topology graph, R-tree, or bounding box once per row. `cosine_similarity` escaped +partially by hand-writing a `reduce_encoded` rewrite, and the survey found that rewrite already +wrong for the literal shape, which is the argument for framework ownership stated as a correctness +fact: one hand-written constant path per function is one place per function to rot on +encoding-normalization details. + +### Where the hook can live, and where it cannot + +The hoist needs three things at once: knowing which arguments are constant, having their decoded +values, and a typed place for the function to compute from them. Constness is a per-batch value +fact (a RunEnd slice landing inside one run, a per-chunk compression decision), so: + +- **`dispatch` cannot see it.** It runs at plan time and run time and must choose identical element + types at both; values do not exist at plan time. +- **Element types cannot encode it.** A `Const` wrapper element would need value-aware dispatch + to be chosen, splitting plan/run monomorphizations in exactly the way the witness deliberately + does not pin, and costing 2^arity dispatch arms. The salvageable half of the idea, + framework-internal value-driven specialization, already exists as the stride-0 `ArgColumn`. +- **The closure cannot memoize it.** An `unsync::OnceCell` capture compiles under `Fn`, but without + constness information it is wrong (it would cache row 0 of a varying operand), and with that + information it saves nothing over a prepare step while planting an unhoistable load inside the + loop. + +That leaves one point: inside the visit, after decode, where `ArgColumn` already knows each +column's stride. `ElementTuple` gains `ConstElems<'a>`, the element tuple with every slot wrapped +in `Option` (`Some` iff that operand is batch-constant), and the visitor gains: + +```rust +fn visit_prepared( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> R, +) -> VortexResult; +``` + +`prepare` runs once per batch; its result reaches every row by `&P`, so `apply` stays `Fn` and the +loop keeps the shape the FnMut measurement forbids changing. `P` names no column lifetime, so +prepared state provably cannot alias the columns the loop reads. Plain `visit` is now a *provided* +method, `visit_prepared` with unit state: the ZST erases under monomorphization (measured, l2_norm +non_nullable at 33.38 us against the 32.83 us hand-written control, parity), the duplicate row loop +is deleted, and the visitor's method count grows with genuine axes (how output is delivered) rather +than with feature combinations. + +`prepare` is infallible in v1: it refines values the row loop could compute itself, and fallibility +is read off the witnesses before dispatch, so a failing prepare would have nowhere to be declared. +The extension (prepare returning `VortexResult

`, riding the existing fallibility axis) is +documented next to the method and deliberately unbuilt, because no adopter needs it. + +Three boundary facts worth stating because they will bite someone: + +- **Prepare must never be load-bearing for validation.** An empty batch decodes every operand as + non-constant (there is no row 0 to slice), so a prepare that validated its constant would + silently not run. Validation belongs to `validate` and the dtype rules. +- **What counts as a batch constant is wider than the constant encoding.** The stride-0 decode sees + one level through two wrappers that spell "the same value in every row" without being it: + `MaskedArray(ConstantArray)`, how the compressor spells an all-same-with-nulls chunk (sound + because the lifting owns validity entirely, so the value the loop reads behind a null row is + unobservable), and `Extension` over constant storage, the shape extension builders produce before + `ExtensionConstantRule` normalizes it. +- **`P` having no `Send`/`Sync` bound is load-bearing.** geo's `PreparedGeometry` carries + `Rc`/`RefCell` and could not be prepared state otherwise. The flip side, recorded so it is a + decision rather than a surprise: adding such bounds later (a parallel row loop, say) is a + breaking change to real adopters, not a relaxation. + +### What it bought, measured + +**cosine_similarity, and a lesson in ILP.** The closure accumulated the rhs norm per element and +sqrt'd it per row, a third of the arithmetic plus one of two sqrts. Hoisting it moved the benchmark +by only ~5% at width 32 and ~3% at 256 (16384 rows, fastest column), far under the flop count, +because the loop is latency-bound on the serial dot-product FMA chain (FP reassociation is illegal) +and the removed accumulation was executing in the chain's spare ILP slots. The measurable saving is +the hoisted sqrt. The row is bit-identical either way, each arm accumulating in the same order as +the unprepared kernel. + +The lesson generalizes and is the honest scoping of the feature: **"removes an O(width) pass per +row" is not "saves time" when that pass rides in ILP slack.** The work that collects the full +saving is work that extends the dependency chain: parses, tree builds, prepared structures. Which +is exactly what the geo numbers then showed. + +**The geo predicates, where the win lives.** `contains` substitutes an owned +`PreparedGeometry<'static>` of the constant operand (r-tree plus self-noded topology, built lazily +inside `P` through a `OnceCell` so point-row batches never pay for it) into relate exactly where +geo routes `Contains` through relate, argument order preserved including the `MultiPolygon` +reversal; direct pairings keep geo's own algorithms untouched. `intersects` hoists the constant +side's `bounding_rect` and replays geo's own disjoint-bboxes early-out, gated to fire only where +geo makes exactly that comparison first. `distance` was investigated and left alone: geo builds +R-trees for both sides inside a private helper on every call, so there is no seam to reuse one, and +the finding is recorded as a doc comment on its dispatch. 16384 rows, fastest column, two runs: + +| arm | before | after | change | +| --- | --- | --- | --- | +| contains, constant x polygons, overlapping | 457.5 / 458.0 ms | 50.88 / 50.00 ms | **9.1x** | +| contains, constant x polygons, disjoint | 7.04 / 7.05 ms | 3.97 / 3.74 ms | **1.9x** | +| contains, constant x points (direct route) | 3.15 / 3.08 ms | 3.22 / 3.15 ms | unchanged | +| contains, column x column | 3.56 / 6.29 ms | 3.68 / 6.40 ms | unchanged | +| intersects, polygons disjoint x constant | 6.81 / 6.72 ms | 3.20 / 3.14 ms | **2.1x** | +| intersects, polygons overlapping x constant | 9.57 / 9.48 ms | 9.87 / 9.63 ms | 1-3% slower, accepted | +| intersects, points and column x column arms | 3.20 / 5.98 ms | 3.21 / 5.92 ms | unchanged | + +The overlapping-intersects arm is the disclosed tradeoff: the hoisted bbox check is an early-out, +so where it rarely fires the row pays for it. The port was an out-of-sample test of the API and +passed it: **zero framework changes were needed**, matching the element vocabulary's earlier record +(`TensorRow`, `GeometryRow`, `TensorSink`, each added in its own crate). + +**Deleting the hand-written path made its shape faster.** With `Extension`-over-constant visible to +the stride-0 decode, cosine's `reduce_encoded` constant routing (manufacture an `L2Denorm` from a +constant operand, answer through the denorm paths) became deletable. Its shape then sped up: + +| width | through the deleted rewrite | through the row loop + prepare | +| --- | --- | --- | +| 2 | 118.8 us | **63.08 us** | +| 32 | 554.0 us | **377.9 us** | +| 256 | 5.159 ms | **3.007 ms** | + +Both constant spellings now measure identically (63.08 vs 62.72 us at width 2). The hand-written +fast path was 1.5-1.9x slower than the framework path that replaced it, on top of having missed the +literal shape entirely. That is the dedup argument in its strongest form: not fewer lines, but +fewer wrong ones. + +### The one unenforceable thing + +The design's benefit rests on LLVM treating the per-row branch on the prepared `Option` as +loop-invariant. Three outcomes exist per call site: unswitched (intended), if-converted (both arms +computed, the hoist silently evaporates while staying correct), or retained (a branch in a cheap +scalar kernel can block vectorization). For every real adopter the hoisted work is a loop or a +parse, which cannot be speculated, so the worst case degrades to one predicted branch per row, the +same cost class as the bounds check kept over `unsafe`. It is still a hope rather than a contract, +and the convention that polices it is stated in the trait-choice guide: every adopter lands with a +constant/non-constant benchmark pair, and the non-constant arm must not move. + +### Rejected alongside + +- **`Const` wrapper elements**: needs value-aware dispatch; splits plan/run; 2^arity dispatch + arms. Dead on the purity invariant. +- **Closure-internal `OnceCell` memoization**: wrong without constness plumbing, redundant with it. + Distinct from the `OnceCell` *inside `P`* that contains uses, which is constness-aware and only + defers an expensive build. +- **Plan-time currying through `reduce`** (folding a Literal into Options as a compiled variant): + the only design that amortizes across batches, deferred because `PersistableOptions` admits only + the source value, it misses every run-time-only constant, and re-currying bifurcates function + identity, silently detaching encoding kernels keyed on the original function. Revisit only if + per-batch prepare cost ever measures as material. +- **`visit_prepared_into`** (sink plus prepare): no user. `l2_denorm`'s constant case is a bulk + answer in `reduce_encoded`, not a prepared loop. The asymmetry is deliberate and cheap to fix + when a user appears. + +--- + +## Is there anything left to port? + +Asked directly: could the remaining hand-written vtables move onto `RowFn` if the element vocabulary +covered more types? Classifying all ~30 of them says no, and says the vocabulary is not what is +stopping them. + +| blocker | count | members | +| --- | --- | --- | +| **Not strict.** `RowFn` implies strict, so these cannot reach it at all. | 12 | `between`, `case_when`, `cast`, `dynamic`, `fill_null`, `is_null`, `is_not_null`, `list_contains`, `pack`, `stat`, `row_size`, `zip` | +| **The answer already exists in bulk.** Zero-copy child projection, a metadata field, or a vectorized slice kernel. A row loop would be strictly slower. | 12 | `not`, `list_length`, `binary`, `mask`, `ext_storage`, `get_item`, `select`, `merge`, `variant_get`, `spatial.envelope`, `json_to_variant`, `row_encode` | +| **No element rows to read.** Zero-arity, or a type-erasure adapter. | 5 | `literal`, `root`, `row_idx`, `row_count`, `ForeignScalarFnVTable` | +| **Output side.** Nullable output, or an output dtype that depends on runtime data. | 2 | `list_sum`, `spatial.envelope` | +| **Value-dependent per-batch setup.** | 1 | `like` | + +`spatial.envelope` is the one function counted twice: its output is a struct-of-four extension type *and* +its fast paths hand back existing child arrays untouched. + +`binary` deserves a note, since on strictness alone it looks portable: only its Kleene `And`/`Or` are +non-strict, and `is_strict` already varies by operator, so comparison and arithmetic go through the +strict lifting today. What keeps it columnar is the kernel. `collect_zip_bits` and `LaneZip` run over +`as_slice()` pairs as tight vectorizable loops, with a separate constant-operand path +(`collect_bits(lhs, |a| a.is_eq(rhs))`). Routing that through a per-row closure and `ArgColumn::get` +would give up the slice-level vectorization for nothing. + +Three things follow. + +**The porting well is dry.** The eight functions on `RowFn` (`byte_length`, the four tensor kernels, the +three geo kernels) are the complete set in this repository that wants a row loop. Every remaining one is +blocked, and forcing any of them onto `RowFn` would cost performance rather than save lines. `l2_denorm` +was the last one the vocabulary was actually keeping out, and the sink let it in. + +**Missing elements are not the constraint.** Only `list_contains` would need new input vocabulary, and +it is independently blocked by non-strictness, so a list element would not unblock a single function +today. A list *input* element is nonetheless easy (`Bytes` already proves the shape: `Elem<'a>` is a +GAT, so `&'a [T]` works), and `list_length` could even be a `RowFn` given a `ListLen` element in the +style of `BytesLen`. It should not be, because its answer is a child array or one constant. + +**`like` is a new gap, and the sharpest one.** It is strict, infallible, `(Utf8, Utf8) -> Bool`: on +signature alone it is the ideal `RowFn`. Two things block it, and measuring both is what settled where +it belongs. + +Its constant-pattern path is fine. `reduce_encoded` already sees the argument arrays before the row +loop, so compiling the pattern once and evaluating in bulk has a home, and a constant operand stays +constant even through a filtered batch. No new hook needed for that case. + +Its *per-row* pattern path is what blocks it. That path memoizes the compiled pattern across +consecutive rows carrying the same one, and a `RowFn` closure is `impl Fn`, so it can hold no such +state. Defeating the cache costs **5.7x** (`like_per_row_distinct_patterns` 249.1 µs against +`like_per_row_patterns` 44.03 µs, 2048 rows, same matching work in both), which is the same shape of +regression the constant-operand stride fixed for geo. + +Relaxing the closure to `impl FnMut` would restore the cache, and it compiles as a one-word change. +It is not free. Measured on `byte_length_element`, `fastest` column, both configurations run twice: + +| case | `Fn` | `FnMut` | delta | +| --- | --- | --- | --- | +| `long_strings_bytes_len` 4096 | 11.15 µs | 12.08 µs | +8.3% | +| `long_strings_bytes_len` 65536 | 166.4 µs | 181.7 µs | +9.2% | +| `long_strings_bytes_slice` 4096 | 14.75 µs | 15.97 µs | +8.3% | +| `short_strings_bytes_len` 65536 | 166.2 µs | 180.4 µs | +8.5% | +| `short_strings_bytes_slice` 65536 | 180.9 µs | 200.3 µs | +10.7% | + +Capturing the closure by `&mut` inhibits the vectorization the shared capture allows, so `FnMut` +taxes every row function 8 to 11% to enable state that one function wants. Keep `visit` on `Fn`. + +The conclusion is that `like` does not want a row loop at all: its general path needs cross-row state, +and its fast path is bulk. What it wants is to declare `(Utf8, Utf8) -> Bool` through the element +vocabulary and keep its own kernel, which is the missing cell below. A per-batch setup hook would not +have been enough on its own, since the state `like` needs is mutable *across* rows rather than fixed +before them. + +A second, smaller thing blocks `like` too: it renders custom SQL through `fmt_sql`, and neither +`StrictScalarFnVTable` nor `RowFn` forwards that, so today porting any function with bespoke SQL +rendering would silently lose it. + +--- + +## Known gaps and future work + +Found by the porting probes, left unfixed here because each is a larger change with its own review +surface. Recorded so they are decisions rather than surprises. + +- **~~No constant-operand affordance.~~ Fixed twice over.** A partially-constant call used to decode + the constant column in full, so a broadcast operand cost one decode per row (measured: a broadcast + query vector cost the same as a genuine column, 234 ms vs 226 ms at 50k x 256). That was what kept + the geo functions off `RowFn`. Each decoded column now carries a stride, 0 for a constant, and the + geo functions are row functions. Constant *compute* was the remaining half, closed by + `visit_prepared` (see [Constant compute](#constant-compute-the-last-quadrant-of-the-lifting)). +- **`NullHandling::Dense` is chosen on safety alone, with no cost input.** For a fixed-width element + (`TensorRow`) dense is unambiguously cheaper. For an unbounded-width row (a nested list) the garbage + behind a null row need only be *in bounds*, so it can span the whole elements array, which is + pathologically O(nulls x elements). No current function hits this, but the choice should consider + width. +- **`OutputElement::build(Vec)` forces materialization.** A row function's output is always a + freshly built `Vec` turned into a `PrimitiveArray`, so it cannot return a `ConstantArray` or a lazy + child. This is why `list_length` is a columnar `StrictScalarFnVTable` rather than a `RowFn`, since a + row port would materialize one `u64` per row and lose the `FixedSizeList` constant. A columnar output + escape that stays inside the framework ("given the decoded columns, can you produce the whole output + at once?") would let `list_length`, `byte_length` and `not` share one abstraction. +- **The missing cell.** The two authoring traits cover *declare-signature-once + row-loop* (`RowFn`) + and *hand-write-signature + own-kernel* (`StrictScalarFnVTable`). The cell for + *declare-signature-once + own-kernel* is empty, so a columnar function hand-writes five signature + methods (`arity`, `child_name`, `return_element_dtype`, `null_handling`, `is_fallible`) that are all + mechanically derivable from an element tuple. + + **It is buildable.** The obvious worry is coherence, since `RowFn` already blanket-impls + `StrictScalarFnVTable` and a second blanket impl of the same trait is a hard E0119 conflict. The way + through is to layer rather than branch, putting the new trait *between* the two: + + ```text + StrictScalarFnVTable <-blanket- StrictSignature <-blanket- RowFn + ``` + + One blanket impl per edge, so nothing overlaps, and a columnar function hand-writes `StrictSignature` + while a row function reaches it through `RowFn`. Compiling the shape confirms a hand-written impl + coexists with the blanket one, including from a *downstream* crate, because within the crate that owns + the type rustc can see the blanket impl's bound does not hold. This is not a new trick here: + `impl ScalarFnVTable for V` already coexists with `Like`'s and `Between`'s + hand-written `ScalarFnVTable` impls the same way. + + **The user count is 3, not 12, and 2 of those need an element first.** Being in the columnar category + is not enough: the function's *signature* has to be expressible in the vocabulary, and + `element_dtype()` taking no arguments rules out every function whose return dtype is derived from its + input at runtime. That is most of them: `mask` returns `arg_dtypes[0].as_nullable()`, `ext_storage` + returns `ext_dtype.storage_dtype()`, `get_item` and `select` a projection of the input struct, + `variant_get` an options-derived dtype, `binary` a width negotiated between operands. What is left is + `not` (`(bool,) -> bool`, usable today), `like` (`(Bytes, Bytes) -> bool`, usable today once `fmt_sql` + forwards), and `list_length` (needs a `ListLen` element in the style of `BytesLen`). + + So this is worth building *after* the elements that give it a third user, not before. Against ~140 + lines of new trait and blanket impl it would save roughly 20 lines per function, which at one usable + caller is a wrapper with one impl. The cheap interim is to make `validate_row_args`, + `row_null_handling` and `row_is_fallible` public, which turns each hand-written signature method into + a one-liner and removes the *logic* duplication (each function currently rolling its own dtype check + and asserting rather than deriving its null handling) without adding a layer. +- **No nullable output element, so no non-total `RowFn`.** `OutputElement::build` always produces an + all-valid column, so a row kernel cannot return a null from a valid row. `impl OutputElement for + Option` is the whole fix. Left out because nothing needs it *yet*: `list_sum` would need it, but + is columnar for independent reasons too (the grouped-accumulator path and the `FixedSizeList` + constant). +- **No borrowed output element, so no zero-copy row function.** A row closure returns an + `ApplyResult`, which is `'static`, so its result cannot borrow from the input columns. Note the + asymmetry with the input side, where `InputElement::Elem<'a>` is a GAT and borrows freely. Every + `str -> str` function therefore copies: `OutputElement for String` allocates one `String` per row + and then rebuilds views from them. A string library would hit this on its first `upper`. Two + distinct fixes, of increasing scope: + - `upper`, `lower` and `replace` genuinely allocate, and want a `Cow<'a, str>` output element. That + needs `OutputElement` to grow its own lifetime GAT and `build` to take an iterator rather than a + `Vec`, so a borrowed row passes through without a copy and an owned one is built in place. + - `trim`, `substring`, `left` and `right` want more than a `Cow` can give. Their result is a + *slice* of the input, so the right kernel keeps the input's data buffer entirely and rewrites + only the views, copying no bytes. That stays columnar whatever the output element can express. + + Predicates and measurements (`starts_with`, `contains`, `byte_length`) have none of this problem + and are already the best case for `RowFn`, so the split for a string library falls along the return + type rather than the argument type. + + **A plain higher-ranked bound does not get there,** which is worth recording because it looks like + it should. Writing the visit as `impl for<'a> Fn(A::Elems<'a>) -> R::Elem<'a>` fails with + [E0582]: the `Fn` sugar puts `R::Elem<'a>` in an `Output` binding, and rustc requires the bound + lifetime to appear *structurally* in the trait's input types before a binding may reference it. An + opaque projection `A::Elems<'a>` does not count, even though it plainly mentions `'a`. Three routes + around it, measured by compiling each: + + | route | works | cost | + | --- | --- | --- | + | concrete input type instead of `A::Elems<'a>` | yes | gives up the element abstraction | + | custom callable trait with a generic `apply` method | yes | callers write a struct per kernel, not a closure, and the impl must spell `::Elem<'a>` rather than `&'a str`, or hit [E0195] | + | pass a zero-sized `Row<'a>(PhantomData<&'a ()>)` token beside the row | yes | closures survive, but every row closure grows an ignored parameter | + + The third is the one to build on: the token makes `'a` appear structurally in the `Fn`'s inputs, + which satisfies E0582 and lets the `Output` binding reference it, and plain closures still infer. + The ignored parameter is a tax on *every* row function though, so the shape to prefer is a second + visit method for lending kernels, leaving today's `visit` untouched for the `'static` majority. + + **Still open, and not what `visit_into` is.** The sink method added since is a second visit method, but + for a closure that *writes* rather than one that *lends*: its output is owned by the sink, not borrowed + from the row. A lending visit would still need the `Row<'a>` token. The precedent it sets is that + adding a third visit method costs the existing ones nothing, which is the same additive shape. + + [E0582]: https://doc.rust-lang.org/error_codes/E0582.html + [E0195]: https://doc.rust-lang.org/error_codes/E0195.html +- **~~`OutputElement::element_dtype()` takes no arguments,~~ Resolved, and not the way this predicted.** + An element's output dtype is a property of its Rust type and cannot depend on runtime data, which is + what kept `l2_denorm` columnar: it returns whole tensor rows, and a tensor's dtype carries its shape. + + Calling that a law was wrong, and the fix was recorded here as "widen `element_dtype` to take `args`". + That is *not* what shipped, and the shipped version is better. `OutputSink::sink_dtype(args)` puts the + argument-dependence on the sink, so all three `OutputElement` impls keep their no-argument + `element_dtype()` and only the thing that needs the arguments asks for them. + + This gap also named the real blocker correctly: `build(values: Vec)` with `Self = Vec` means + one heap allocation per row and then a flatten, against a columnar kernel that scales the flat storage + buffer in a single pass. At 16k rows that is 16k allocations versus zero, and no amount of dtype + plumbing fixes it. The prescription it drew, "an output element that writes into a preallocated flat + buffer (`fn apply(row, out: &mut [T])`)", is exactly what `OutputSink` is, generalized past `&mut [T]` + so a byte buffer works too. See + [the audit](#audit-can-the-four-strictscalarfnvtable-impls-really-not-be-rowfn) for what it cost and + bought. + + Note also what *not* to do on the input side: replacing the generic `TensorRow` with a + non-generic element whose `Elem<'a>` is an enum over `f16`/`f32`/`f64` would move the width choice + from monomorphization into a branch inside the row loop. That is precisely what + `match_each_float_ptype!` plus a generic element exists to avoid, so it would cost every tensor + kernel its inner-loop specialization. +- **~~The witness carries four scalars through two associated types.~~ Not a gap.** This looked like + the framework's weakest joint, since `ArgsWitness` and `RetWitness` are read *only* for `ARITY`, + `DENSE_SAFE`, `DECODE_FALLIBLE` and `FALLIBLE`, and for a multi-dispatch function the witness names + an arbitrary representative (`L2Norm` says `f64` for no reason a reader can see). The plan was to + collapse them into three consts. + + Checking the signatures says no. `arity`, `null_handling` and `is_fallible` on + `StrictScalarFnVTable` all take *only* the options, with no input dtypes, while `dispatch` needs + dtypes to choose. So those three answers **must** be dtype-independent, which means they cannot be + read off whatever element types a batch picks, which is exactly why a separate declaration has to + exist. The witness is not redundant bookkeeping; it is the only place those facts can live. + + Given that, types beat consts. With types, dense-safety and fallibility are *derived* from the + element types, so the only available mistake is a witness that disagrees with the dispatch, and that + is a build error. With three hand-written consts an implementor could state a fact wrongly *and* + visit consistently with their mistake. Converting would be a notation change that removes a + derivation, not a fragility fix. Left alone, with the reason now recorded on `ArgsWitness` so the + next reader does not re-open it. + + What is left of the original complaint is presentational: the arbitrary representative reads oddly. + A doc line on each multi-dispatch implementor saying why the width shown is arbitrary is the whole + fix. +- **`InputElement` is an open trait with required consts.** Adding `DECODE_FALLIBLE` broke every + out-of-crate element (`TensorRow`) until updated. If elements are a real extension point for other + crates, `DENSE_SAFE` / `DECODE_FALLIBLE` should carry conservative defaults. +- **`DENSE_SAFE`'s doc guidance is subtly wrong for lists.** It says `false` for "any element that + follows an offset," but a list element *is* dense-safe, because list arrays validate + `offsets[i] + sizes[i] <= elements.len()` for every row including nulls. Following the doc literally + would put `list_length` on `Filter` and lose its encoding fast paths. + +--- + +## What the ports bought + +**Not line count.** That was the first justification I reached for and it does not hold up: `row/` is +514 code lines and `strict/` is 269, against roughly 470 lines saved across six kernels. Near +break-even. Nor is it bug fixes, since none of the three extracted problems is a live miscompute on +`develop`. + +**It is `unsafe`.** Every hand-written kernel in `vortex-tensor` ended the same way: + +```rust +// SAFETY: The buffer length equals `len`, which matches the source validity length. +Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) +``` + +A kernel that computes its own values *and* carries its input's validity has to assert that the two +lengths agree, and the only tool for that is `new_unchecked`. The framework never pairs them: +[`OutputElement::build`] returns a non-nullable column, and the strict lifting applies validity +afterwards by masking. The invariant stops being asserted and becomes unrepresentable. + +Counting production `unsafe` blocks, test modules excluded: + +| function | layer it moved to | `unsafe` on `develop` | `unsafe` now | +| --- | --- | --- | --- | +| `l2_norm` | `RowFn` | 1 | 0 | +| `inner_product` | `RowFn` | 3 | 0 | +| `cosine_similarity` | `RowFn` | 3 | 0 | +| `l2_denorm` | `RowFn` (was `StrictScalarFnVTable`) | 8 | 6 | + +**This started as a controlled experiment and the control has since been ported, so read it in two +stages.** For most of this branch's life `l2_denorm` stayed on `StrictScalarFnVTable` and held all 8 of +its blocks while the three functions that moved onto the row layer lost all of theirs. Same crate, same +reviewers, same standards, so the row layer was what removed them rather than the strict lifting or the +port itself. That is the inference the control bought, and it is still the argument. + +`l2_denorm` then moved onto the row layer too, via `OutputSink`, and dropped to 6. The two it lost are +exactly the memory-safety ones on its kernel path, which is the pattern the other three showed. Of those +two, one (`FixedSizeListArray::new_unchecked` in the constant-norms path) is attributable to the port and +one (`PrimitiveArray::new_unchecked` in `build_tensor_array`) is an independent cleanup noticed along the +way. Its 6 remaining blocks are a different kind and are not the row layer's business: four call +`L2Denorm::new_array_unchecked`, an `unsafe fn` guarding the *semantic* unit-norm invariant rather than +memory safety, and two are buffer pushes in `normalize_as_l2_denorm`, a helper that is not a scalar +function. + +`develop`'s `l2_norm` also hand-rolled a 25-line constant-array fast path that the strict lifting now +does generically for every function, and computed its output nullability by hand. + +This is the justification to carry onto a clean branch. It also bounds the claim: a `vortex-tensor` +local helper owning the same invariant would remove the same `unsafe`, so what earns the *generic* +placement in `vortex-array` is that `vortex-spatial`'s three predicates and `byte_length` use it too, +over three different element types. Two downstream crates plus core is the second-caller test met, not +anticipated. + +### What it costs + +Removing that `unsafe` is not free, because `new_unchecked` was buying something: the old kernel paired +its freshly built buffer with the input's validity in one step, so a nullable input cost it nothing +extra. The framework builds a non-nullable column and the lifting applies validity afterwards, which +for `Validity::Array` means materializing a mask and running a separate pass. + +That pass is `O(rows)` while the kernel is `O(rows * width)`, so width amortizes it. Measured on +`vortex-tensor/benches/l2_norm.rs`, 16384 rows, `fastest` column: + +| width | non-nullable | nullable | cost of the extra pass | +| --- | --- | --- | --- | +| 2 | 68.87 µs | 70.44 µs | +2.3% | +| 32 | 241.4 µs | 243.9 µs | +1.0% | +| 256 | 2.513 ms | 2.529 ms | +0.6% | + +So 1 to 2% on nullable input, worst at the narrowest vector anyone would store, and nothing at all on +non-nullable input where no mask is applied. Trading that for eight memory-safety `unsafe` blocks is the +right side of the deal. + +These figures are near this machine's noise floor and should be re-confirmed on quieter hardware before +being quoted. The larger measurements in these notes (the 5.7x `like` cache loss, the 8 to 11% `FnMut` +tax, the 2x width-2 per-row cost and its removal, the 2x `l2_denorm` sink win) are well clear of it. + +### The like-for-like comparison, and the per-row cost that was hiding in it + +The table above compares the framework against itself, so it isolates the masking pass but says nothing +about the rest of the machinery. `PrePortL2Norm` in the same benchmark closes that: a bench-local +`ScalarFnVTable` running the identical arithmetic, indexing the flat slice directly into a `Buffer` and +attaching validity in one step. + +This measurement found a real defect in the tensor element, and the diagnosis recorded here first was +wrong in a way worth keeping visible. + +**What was measured, and the wrong inference.** `fastest` column, non-nullable, 16384 rows: + +| width | framework | pre-port | delta | +| --- | --- | --- | --- | +| 2 | 68.85 µs | 32.85 µs | **2.10x slower** | +| 32 | 266.6 µs | 255.5 µs | +4% | +| 256 | 2.564 ms | 2.512 ms | +2% | + +The gap in absolute terms is 36 µs at width 2 and 11 µs at 32, and the conclusion drawn was "a cost that +shrinks as total work grows is a constant being amortized, so the framework carries tens of microseconds +of fixed per-batch setup." That reasoning does not hold. 36 µs over 16384 rows is 2.2 ns/row, which is a +*per-row* cost; it stops showing at width 32 because the kernel there is memory-bound and absorbs extra +CPU work in its stalls. Reading "shrinks with width" as "fixed per batch" skipped dividing by the row +count. + +**The actual cause was one per-row accessor, in the tensor element.** `TensorRow::get` called +`FlatElements::row::(i)`, which per row re-derived its typed slice: a ptype comparison against the +stored `PType`, a host-buffer downcast out of the buffer handle, a length division, and then two range +indexings with a bounds check each. All of it loop-invariant except the offset. This is exactly the +hidden-cost-accessor pattern the repository guidelines warn about, and it was written into the element +rather than found in the framework. + +The fix types the column at decode time instead of per row. `TensorRow` is already generic over `T`, +so its `Column` can be a `Buffer` plus a stride, and `get` becomes one multiply and one range index +into a typed slice. `FlatElements` keeps its untyped `row` for the callers that read a handful of rows. + +**After, same bench, same run:** + +| width | framework | pre-port | delta | +| --- | --- | --- | --- | +| 2 | **33.32 µs** | 32.83 µs | **parity, 1.01x** | +| 32 | **227.4 µs** | 258.9 µs | framework **1.14x faster** | +| 256 | **2.422 ms** | 2.522 ms | framework **1.04x faster** | + +The pre-port column is stable across both runs (32.85 then 32.83 µs at width 2), which is what makes +this comparison trustworthy; only the framework side moved. `l2_denorm` gained the same way, from +88.0 µs to 48.9 µs at width 2, since it reads its tensor argument through the same element. + +Three things follow. + +**The row layer was never the cost.** The 2x was one accessor in one element implementation, and the +generic machinery around it (the visitor, the witness, the strict lifting's bookkeeping, `reduce_encoded`'s +probe, the dispatch width match) does not measurably show up at 16384 rows. The planned decomposition +into "strict lifting versus row layer" is moot: neither was it. + +**An element is a performance-critical surface, and nothing in the framework says so.** `InputElement::get` +is documented as needing to be `O(1)`, which `FlatElements::row` technically was. `O(1)` is the wrong +contract; the right one is that `get` must not repeat work that is constant across the batch, because it +is the one function called once per row. `decode` exists precisely to hold that work, and the element +vocabulary's whole promise (anyone can add an element in their own crate) means this trap is now +available to every future implementor. + +**The framework being generic is what let one fix pay out twice.** `l2_norm`, `inner_product`, +`cosine_similarity` and `l2_denorm` all read tensor rows through this element, so a single change moved +all four. That is the case for the shared layer stated in performance terms rather than in line counts. + +### What the harness actually costs, from the optimized IR + +The measurements above say the harness is free at 16384 rows. Reading the post-optimization LLVM IR says +*why*, and settles whether more `#[inline]` would buy anything. Emitted with +`cargo rustc --release -p vortex-tensor --lib -- --emit=llvm-ir -Cdebuginfo=0`, reading the `l2_norm` f64 +arm. + +**The whole stack is already one function.** `execute_row_loop`, `ElementTuple::get` and the row closure +have no `define` of their own anywhere in the module. They survive only as basic-block *labels* carrying +`.exit.i.i.i…` suffixes about sixteen `.i` deep, which is inline-depth notation: the engine's +`ScalarFnVTable::execute`, `execute_dense`, `execute_strict`, `dispatch`, `RowVisitor::visit`, +`execute_row_loop`, `A::get` and the closure are all inlined into a single body. Adding `#[inline]` +anywhere on that path cannot help, because nothing on it is still a call. + +**Per batch the harness leaves five calls**, each correctly placed outside the loop: one +`ArgColumn::decode` per argument, one `tensor_element_ptype` for the width match, one `reduce_encoded`, +one `OutputElement::build` after the loop exits, and the output allocation. + +**Per row it leaves this, and nothing else:** + +```llvm +%row = phi i64 [ 0, %preheader ], [ %next, %loop_latch ] +%next = add nuw i64 %row, 1 +%start = mul i64 %row, %stride ; ArgColumn's stride, fused with list_size +%end = add i64 %start, %list_size +%ovf = icmp ult i64 %end, %start ; the two halves of one slice range check +%oob = icmp ugt i64 %end, %len +br i1 (or %ovf, %oob), label %slice_index_fail, label %body ; cold side out of line +%rowp = getelementptr inbounds nuw double, ptr %elements, i64 %start +%endp = getelementptr inbounds nuw i8, ptr %rowp, i64 %list_size_bytes +... ; element loop, 8x unrolled +%out = getelementptr inbounds nuw double, ptr %values, i64 %row +store double %result, ptr %out +``` + +About ten integer ops and one always-taken branch. The element loop underneath is 8x unrolled with a +serial `fadd` chain (LLVM correctly refuses to reassociate the float sum) terminating on `icmp eq ptr` +against `%endp`, which is what a hand-written `iter().map(|x| x * x).sum().sqrt()` compiles to: the +`Elem<'a> = &'a [T]` GAT is fully scalar-replaced, and the slice iterator becomes pointer bumping at +fixed byte offsets. + +**The one removable cost is not worth removing.** The surviving per-row branch is the range check on +`&elements.as_slice()[start..start + list_size]`. LLVM cannot hoist it because nothing tells it +`len == rows * list_size`. Eliminating it means `get_unchecked`, and this framework's stated value is +removing `unsafe` from kernels, so buying back a perfectly-predicted branch with an unchecked index is +the wrong direction. It is also already hidden: at width 2 the row's `sqrt` alone has longer latency than +the whole index computation. + +LLVM also unswitched the row loop on `list_size == 0` and emitted a zero-width specialization that stores +`0.0` per row. Harmless, and a sign the loop was simple enough to reason about completely. + + + +[`OutputElement::build`]: vortex-array/src/scalar_fn/row/element/mod.rs + +Production lines, before and after: + +| function | layer | before | after | +| --- | --- | --- | --- | +| `byte_length` | `RowFn` (fixed) | n/a | 23 (impl) | +| `list_length` | `StrictScalarFnVTable` | 189 | 143 | +| `not` | `StrictScalarFnVTable` | 76 (impl) | 53 (impl) | +| `list_sum` | `StrictScalarFnVTable` | 78 (impl) | 56 (impl) | +| `l2_norm` | `RowFn` (width) | 254 | 96 | +| `inner_product` | `RowFn` (width) | 277 | 112 | +| `cosine_similarity` | `RowFn` (width) | 309 | 203 | +| `l2_denorm` | `RowFn` (width, sink) | 731 | 618 | +| geo x 3 | `RowFn` (fixed) | 51 each (impl) | 15 each (impl), plus one shared element | + +Nothing outside the functions' own crates changed: the `L2DenormScheme` compressor and every +`ExactScalarFn` matcher are untouched, because the encoding-aware push-downs key off the function +*type* rather than its vtable layer. + +The line-count case does not close on its own. The framework is ~1670 production lines (up from ~1510 +before the sink, which added `result.rs`, `sink.rs` and a second visit path) and removes ~870 across the +ported functions, so **net this branch adds lines**, amortizing around the fourteenth function against +~20 strict candidates in the tree. To be honest, the case for merging is the marginal +cost of the *next* function (~15 lines, and the invariants above enforced rather than reviewed), plus +the correctness the type-derived properties buy, rather than the diff. + +--- + +## Measurements + +`vortex-array/benches/byte_length_element.rs`, element choice for `byte_length`, whole-execution +medians: + +| input | `BytesLen` | `Bytes` | | +| --- | --- | --- | --- | +| 64Ki non-inlined rows | **206 µs** | 256 µs | 24% faster | +| 64Ki inlined rows | **207 µs** | 215 µs | 4% faster | + +`vortex-array/benches/strict_validity.rs`, how the `Dense` path applies validity, same kernel in both +arms: + +| | `lazy` | `eager` | | +| --- | --- | --- | --- | +| 64Ki, one call | **9.0 µs** | 75.3 µs | 8.3x faster | +| 1Mi, one call | 1.357 ms | 1.357 ms | parity | +| 64Ki, chain of 3 | **28.3 µs** | 30.6 µs | 7% faster | + +`Validity::and` is already lazy, so the conjunction is never materialized to be applied. Only +`NullHandling::Filter` needs positions, and only it pays for them. + +`not`, word-wise kernel against the row loop it would have if it were a `RowFn` (release, identical +outputs asserted): + +| len | word-wise `!` | row loop + `bool::build` | +| --- | --- | --- | +| 64Ki | 927 ns | 376 µs (**406x**) | +| 1Mi | 10.3 µs | 5.83 ms (**569x**) | + +This is why `not` is a columnar `StrictScalarFnVTable` rather than a row function. + +--- + +## Rejected alternatives + +- **A wrapper type instead of a blanket impl** (`Strict`): forces churn at every call site, + meaning matchers, kernel registrations, and expression constructors. The blanket impl means a port + edits only the function's own impl block. +- **A `row_family!` macro, a per-crate GAT family, or a framework GAT family**: three encodings of + "element types as a function of the width," all paying for the same limit (the width bound has to + appear literally in a GAT), so each width class needed its own trait *and* adapter. The rank-2 + visitor replaces the whole lineage with one non-generic trait method and no generated code. +- **`ElementwiseFn` as a third trait**: subsumed by `RowFn` with a constant dispatch, see above. +- **One `RowFn` with defaulted `dispatch` and `apply`**: converts "define nothing" from a compile + error into a runtime panic. +- **Renaming `StrictScalarFnVTable` to `TotalFnVTable`**: the trait admits non-total members on + purpose, so the name would be wrong. +- **An `is_total` method feeding a derived `validity`**: a new concept to compute what a function can + state directly. Mirroring `validity` with a `None` default makes the unsound answer the one that + takes work. +- **Macro-generated per-type constructors**: a bespoke API per function, where the general + `ScalarFnFactoryExt::try_new_array` is what every other scalar function already uses. +- **A separate `FallibleElementwiseFn`**: an associated return type (`ApplyResult`) costs one line per + function instead of a whole trait and a spent coherence slot. + +## Null strategies and the non-strict frontier + +The question that opened this chapter: with the strict trait retiring into a private lifting under +`RowFn`, could the row framework also serve non-strict functions, where the kernel sees each input +as an `Option` and owns null semantics itself? The prior expectation was "probably not useful or +performant, but worth establishing why." The answer splits into three verdicts, one per axis, and +the investigation surfaced a fourth result nobody asked for that is worth more than the question. + +Method: a survey of every non-strict `ScalarFnVTable` impl in the workspace plus every consumer of +`is_strict` and `validity()`, and a working prototype (worktree branch `proto/null-strategies`, +2,034-line diff, not for merging) that implemented both a branch-and-skip execution strategy and a +`Nullable` input element, benchmarked on 65,536-row batches at null densities from 0% to 90%. +All 435 vortex-array scalar_fn tests and 223 vortex-spatial tests pass with the prototype strategy both +off and on, including new hostile tests (out-of-bounds views and poison divisors behind null rows) +proving the kernel never runs behind a null. + +### Verdict 1: null-visible inputs have no customer, and now we know the price + +The survey found 15 non-strict functions. Thirteen are cheap columnar mask algebra or pure +structure. The canonical case is Kleene `AND`: a fused kernel computing values and validity +together at roughly six bitwise ops per 64 rows, with validity `(lv & rv) | (lv & !l) | (rv & !r)`. +The prototype measured a row-function Kleene `AND` over `(Nullable, Nullable)` against +it: **250x to 1,030x slower** depending on density. That is the honest price of spelling bitwise +logic one row at a time, and no framework design recovers it. + +The remaining two, `RowEncode` and `RowSize` in vortex-row, are the only genuinely expensive +null-visible per-row kernels in the tree, and they are excluded by something the Option tier does +not touch: they are variadic over heterogeneous column types with a shared per-row write cursor, +which the fixed-arity tuple witness cannot express. Null-visible inputs alone unlock nothing. + +Four functions (Kleene `AND`/`OR`, `zip`, `case_when`, `list_contains`) have **value-dependent +output validity**: `false AND null` is a *valid* `false`. For these no validity expression over +child validities exists even in principle, so the lifting's derivations (validity expression, mask +motion, dictionary push-down eligibility) are unavailable by definition rather than by +implementation gap. Any future Option-input tier must let the kernel author value and validity +together, which is to say it must be a different trait, not a mode of this one. + +What `is_strict = false` forfeits is exactly enumerable: the dictionary values push-down +(`arrays/dict/compute/rules.rs`), the dict-layout below-decode push-down +(`vortex-layout/src/layouts/dict/reader.rs`), and, when `validity()` is also `None`, lazy validity +on an unexecuted `ScalarFnArray` degrades to executing the kernel to read its nulls. Nothing in +vortex-scan, vortex-file, or the engine integrations consumes strictness. + +Mechanically, `Nullable` works exactly as sketched: `Elem<'a> = Option>`, decode +materializes the validity mask once, `get(i)` consults it, `DENSE_SAFE = true` by construction. +Niche packing is free for every by-reference element (`Option<&[u8]>`, `Option<&str>`, +`Option<&[T]>`, `Option<&Geometry>`, `Option` all compile-time asserted same-size) and +doubles every by-value primitive, which are precisely the elements that were already dense-safe +and never needed a strategy. The prototype's geo `contains` over `(Nullable, const)` +tracked branch-and-skip within 2-8%, so the shape is viable for a kernel that wants null +visibility for semantic reasons. Nothing in the tree does. **Do not build it; keep the survey's +constraint list for whenever a real variadic or null-visible demand shows up.** + +### Verdict 2: Option outputs inside the strict tier are the real demand + +Strictness is a subset bound, `valid(out) ⊆ valid(in)`, so a kernel that turns a valid row into a +null is still strict, and the strict lifting already keeps kernel-produced nulls, unioned with the +lifted ones. What excludes such functions from `RowFn` today is only the all-valid-output rule on +`OutputElement`. Two in-tree functions are shaped exactly like this: `list_sum` (a valid empty +list sums to null; the module doc names it as the canonical exclusion) and `variant_get` +(expensive per-row path traversal where a missing path yields null). The extension is small and +local: an `Option` output form whose element dtype is nullable and whose build sets validity, +`RetWitness` gaining a nullability bit alongside `FALLIBLE`, and the derived `validity()` moving +from `union_child_validities` to `None` for such functions, which costs them lazy validity but is +already the status quo for both named candidates. `is_strict` stays `true`. **This is the piece +worth building.** + +### Verdict 3: branch-and-skip, the result nobody asked for + +Today the derived null handling is binary: `Dense` (run over garbage, mask after) when every +element is dense-safe and the kernel infallible, else `Filter` (filter every input to the +conjoined-valid rows, run, scatter back). The prototype added the missing third strategy: +materialize the conjoined mask once, run over the *unfiltered* inputs visiting only set rows +word-at-a-time (`BitBuffer::for_each_set_index`), pre-fill the output with garbage, mask exactly +as Dense does. Fallible kernels stay sound because apply never runs behind a null. + +Measured against Filter at 65,536 rows (divan fastest, two runs): + +| workload | 1% nulls | 10% | 25% | 50% | 90% | +| --- | --- | --- | --- | --- | --- | +| `byte_length` at `Bytes` (cheap kernel) | branch 1.8x | 2.6x | 3.8x | 4.7x | **5.9x** | +| geo `contains`, one nullable operand | branch 1.07x | 1.11x | 1.18x | 1.11x | filter 1.38x | +| geo `contains`, two nullable operands | branch 1.06x | even | filter 1.2x | filter 1.9x | filter 11.3x | + +For the cheap kernel Filter never wins: at even 1% nulls, filtering the input plus scattering the +output costs more than the entire branch-side loop. For the expensive kernel the governing +quantity is the **surviving-row fraction**: branch pays O(n) decode regardless, Filter pays +O(survivors) decode plus filter and scatter. Geo's ablation makes the mechanism explicit: filter +plus scatter are under 4% of `contains`' total, so Filter's entire advantage at sparse validity is +the shrunken arrow-export-and-parse, while for `byte_length` those same two steps are 20-40% of +Filter's total and pure waste. Crossover lands near 50-75% surviving rows for one nullable operand +and lower with two (the conjoined fraction shrinks quadratically). + +The strategy is invisible to function authors: it slots under the existing derived null handling, +selectable per batch from `Mask::true_count`, with Filter kept for the sparse tail. **This is now +implemented on this branch** (see "Adaptive null strategy, as shipped" below); the rest of this +section records the prototype evidence that justified it. The prototype +also validated the two supporting pieces: a null-tolerant `decode_branch` on `InputElement` +(defaulting to plain decode, correct for bulk canonicalization) and `OutputElement::garbage()` +for pre-fill. Production caveats recorded in the prototype report: `reduce_encoded` is not +consulted on the branch path, sinks fall back to Filter, the toggle must become per-execution and +cost-based, and geo's null-tolerant decode covered Point and Polygon only, still paying a +full-length arrow export that a run-slicing decode would shrink. The prototype's conclusion, since borne out: `Bytes`-element functions were paying the +Filter tax on every nullable batch, and most of it is recoverable. + +### Adjacent findings, recorded so they are not relearned + +- `Between::validity` declares the strict three-way conjunction while its fallback execute path + joins two comparisons with Kleene `AND`; with per-row nullable bounds the lazy validity and the + executed result disagree (a valid `false` reported as null). Pre-existing on develop, + independent of this work, slated-for-removal expression; deserves an issue. +- `not` is already at the optimum reachable through the current ownership model: `to_bit_buffer()` + is a handle clone, the source array keeps the buffer shared, so in-place negation (a real 19% on + uniquely owned buffers) is unreachable without redesigning `ExecutionArgs` ownership. Encoded + NOT flows through `NotReduce` (Constant, Sparse) and generic per-encoding push-down (Dictionary, + RunEnd) at 13-24x below canonical cost; `NotKernel` has no implementations and looks like dead + code. The three columnar ports of the retired strict trait revert entirely. +- The strict lifting's small-batch overhead is generic prelude bookkeeping (collect inputs, + compute the declared dtype, conjoin validity), not any single avoidable allocation; ablations + including SmallVec found nothing independently beneficial, and the earlier -10%-at-100-rows + reading did not reproduce uniformly. The row layer can eventually monomorphize the prelude over + its compile-time arity (`[ArrayRef; N]` via the tuple witness), which is the only structural + answer if small batches ever matter. + +## Adaptive null strategy, as shipped + +Branch-and-skip is implemented as a third null strategy, chosen per batch by the lifting. Nothing +about a function's definition changes: the row layer already derived `Dense` or `Filter` from the +element types, and `Filter` now names a *contract* (the kernel never sees a row null in any input) +rather than a mechanism. Two mechanisms satisfy that contract, and the lifting picks between them +where the conjoined mask is materialized. + +The selection rule needs one fact the framework cannot infer, so elements state it: +`InputElement::DECODE_SHRINKS_WHEN_FILTERED`, defaulted `false`, is `true` for an element whose +decode parses every row (geometry from coordinate storage) and `false` for a bulk canonicalization +(bytes, bools, primitives). Getting it wrong is a performance bug, never a correctness bug. +`ElementTuple` ORs it across arguments, the witness check pins it like dense-safety and +fallibility, and the rule is: + +```text +branch-and-skip, UNLESS some argument's decode shrinks when filtered + AND fewer than BRANCH_MIN_SURVIVING_FRACTION (0.75) of rows survive +``` + +Two supporting hooks: `InputElement::decode_null_tolerant` (defaults to the ordinary decode, sound +because the branch loop never resolves an unset row, so hostile bytes behind a null are never +touched) and `OutputElement::placeholder` (the pre-fill written behind nulls, masked before anyone +observes it). Geo overrides the decode for Point and Polygon; other geometry types report +unsupported and the selection falls back to Filter, which is tested rather than asserted in a +comment. Sinks stay on Dense/Filter, documented at the visitor. `reduce_encoded` runs on the +branch path over the *original* encodings, which is strictly better for encoding fast paths than +Filter's canonical copies, and its contract doc now states the row count differs per strategy. + +The original forced-filter, forced-branch and auto measurements used 65,536 rows on a shared 4-vCPU +VM: + +| workload | 1% | 5% | 10% | 25% | 50% | 90% | +| --- | --- | --- | --- | --- | --- | --- | +| `byte_length` at `Bytes`, auto over filter | 5.0x | 5.3x | 5.8x | 4.0x | 4.5x | 6.3x | +| geo `contains` x const, auto picks | branch | branch | branch | branch | filter | filter | +| geo `contains` x column, auto picks | branch | branch | branch | filter | filter | filter | + +Those historical rows justified shipping branch-and-skip, but they no longer calibrate the global +threshold. The controlled x86 AVX-512 rerun used a Ryzen 9 7950X pinned to CPU 4, TSC timing, a +performance governor, 60 samples for 2-4 seconds per arm, and two runs. Its representative medians +were: + +| workload | auto | branch | filter | verdict | +| --- | ---: | ---: | ---: | --- | +| one nullable, 50% nulls | 5.999-6.050 ms | 5.560-5.642 ms | 6.026-6.049 ms | auto filters, branch is 6-8% lower latency | +| two nullable, 10% nulls | 10.40-10.48 ms | 10.49-10.60 ms | 10.20-10.34 ms | auto branches, filter is 2.5-2.8% lower latency | +| two nullable, 25% nulls | 7.502-7.678 ms | 9.156-9.285 ms | 7.588-7.749 ms | auto correctly filters; filter is 1.21-1.22x faster than branch | +| two nullable, 90% nulls | 277.1-277.4 us | 3.232-3.253 ms | 277.7-278.5 us | auto matches filter; filter is about 11.6x faster than branch | + +The two misses point in opposite directions. A 50% surviving one-element decode still favors +branch, while an approximately 81% surviving two-element decode already favors filter. A single +threshold against the conjoined survivor fraction therefore cannot represent both decode cost and +arity. Replace it with per-element/arity inputs or a small estimated-cost comparison when this work +moves onto production branches. Batch size remains an unmeasured input to that model. + +Verified independently of the implementing agent: 3,441 tests pass across vortex-array and +vortex-spatial (17 new: hostile out-of-bounds views behind nulls, a fallible kernel with poison +divisors behind nulls in one and both operands, conjoined-mask honoring, constant operands, real +errors still propagating, geo filter-versus-branch agreement, the unsupported-geometry fallback, +and six selection-rule cases), vortex-tensor's 164 pass unchanged, clippy `--all-targets +--all-features` is silent on both crates, and fmt and whitespace are clean. + +Open items, none blocking: the branch fallback probes `reduce_encoded` twice when the dispatch +turns out unsupported (cheap encoding check, no in-tree function affected since every +`reduce_encoded` implementor is a dense-path tensor function); geo's null-tolerant decode still +arrow-exports the full column, and slicing runs of valid rows would blunt Filter's sparse-validity +advantage enough to retire the threshold for geo; the fallible branch loop pays one `is_none` +check per set row after the first error because `for_each_set_index` cannot early-return. + +## The strict trait, deleted + +`StrictScalarFnVTable` is gone. Not made private: deleted, with its lifting kept as private +machinery under `vortex-array/src/scalar_fn/row/lift.rs`. The chain is now `RowFn` -> +`ScalarFnVTable`, one blanket impl, no intermediate trait. + +Three things converged on that. First, reverting the columnar ports left the trait with exactly one +implementor, the blanket impl over `RowFn`, and a trait with one impl is indirection rather than +abstraction. Second, the mirroring tax existed only because that blanket impl occupied the +`ScalarFnVTable` slot: `reduce` and `validity` were forwarded so a strict function could override +them despite being unable to implement `ScalarFnVTable` itself. `RowFn` keeps `validity`, because +all-valid outputs make it the child conjunction, and the `reduce` mirror went with the trait since +no adopter ever used it. Third, the naming objection a local review raised was real and is now +moot: `is_strict` names the semantic property `valid(f(x)) ⊆ valid(x)` that pushdown consumes, +while the trait demanded the *operational* property that a kernel may run over the garbage behind a +null row or over a filtered copy. Those are independent, `Bytes` being strict and not dense-safe, +so the trait was named for the wrong one of the two. + +What replaced each member: `execute_strict` and `execute_strict_branch` are the two closures +`Batch::execute` takes, `decode_shrinks_when_filtered` is a `Batch` field read off +`ElementTuple::DECODE_SHRINKS_WHEN_FILTERED`, `return_element_dtype` is what a visit returns before +`ScalarFnVTable::return_dtype` widens it, `null_handling` is `row_null_handling` over the witnesses, +and options serde is `RowFn::Options: PersistableOptions` delegated from the blanket impl. +`Batch` carries one batch's facts (id, arguments, collected inputs, conjoined validity, declared +return dtype, null handling, and the decode-shrinks flag) and takes the kernel as closures rather +than through a trait, which is the point: there is no second implementor to name. + +The one behaviour deliberately dropped is the runtime rejection of `Dense` paired with a fallible +kernel. `row_null_handling` derives the pairing from the same witnesses `is_fallible` reads, so the +combination cannot be constructed, and the requirement now lives in `NullHandling::Dense`'s doc +pointing at the derivation. Four tests went with the trait: three described a strict kernel that +returns nulls of its own (`list_sum`'s shape), which no `RowFn` can be until the `Option` output +form of open item 3 exists, and one pinned the `reduce` mirror. + +`PersistableOptions` survives with `EmptyOptions` as its only implementor, since every row function +in tree uses it. That is a bound on `RowFn::Options` rather than a speculative trait, and the +reverted `list_sum` port is what removed its second implementor. + +If a non-row columnar kernel ever wants the lifting, extract the trait then, named for the lifting +contract rather than for strictness, with that kernel as its first user. + +## Sink-only execution, the final prototype + +The last executor revision collapses every row function onto one primitive: + +```rust +visitor.visit_prepared_into::( + |constant_args| prepare(constant_args), + |state, args, output| write_one_row(state, args, output), +) +``` + +The ordinary case uses unit preparation and `ElementSink`. A tensor uses `TensorSink` so the +input dtype can determine the runtime row width. A future string transform can own one batch-wide +builder. These are not different executor modes, so the API no longer gives them different visit +methods. + +### Why the return witness disappeared + +A returning row closure needed a return witness before dispatch so `return_dtype` and fallibility +could be derived without knowing which dtype arm dispatch would select. Once every closure writes +through a sink, the sink already answers the output question: + +- `sink_dtype(args)` supplies the non-nullable element or runtime-shaped dtype. +- `with_capacity` allocates once for the batch. +- `rows` borrows the loop-local storage once. +- `row_count_matches` proves the output bound once. +- `row` hands one slot into the closure. +- `finish` builds the column and interprets any deferred error. + +`RowFn::ArgsWitness` remains load-bearing because arity and input decode properties are needed +before dispatch. `RowFn::FALLIBLE` remains because `ScalarFnVTable::is_fallible` is queried without +input dtypes. There is no analogous need for a return witness. + +The closure stays `Fn`, not `FnMut`. An earlier sink design captured `&mut Sink` in the closure and +measured 8 to 11% slower because the mutable capture blocked loop vectorization. The executor now +owns the sink, borrows its rows once, and passes a row slot as an ordinary argument. + +### Errors without a per-row result branch + +`SinkResult` has three implementations: + +- `()` for an infallible write. +- `VortexResult<()>` for an error that must exit immediately. +- `DeferredError` for a row that can write a legal provisional value and report failure after the + loop. + +Checked integer addition is the motivating deferred case. Its sink writes the wrapping sum, each +row returns a word whose sign bit means overflow, and the executor OR-reduces those words. `finish` +returns the overflow error only when the final word has its sign bit set. No `Result` discriminant +or conditional error branch is required per row. + +Nullable dense execution needs one extra rule. Garbage behind a null may overflow even when every +valid row succeeds. When dense execution finishes with a deferred error, the lifting materializes +the conjoined validity and retries only valid rows. A successful retry proves the first error came +only from discarded rows; a second deferred error is real. This preserves strict null propagation +without giving up the dense vector loop on the common path. + +This is deliberately narrow. Parsing, allocation, and any computation that cannot produce a legal +provisional row still returns `VortexResult<()>` and receives valid-row-only execution. + +### Skipped rows are a sink property + +`OutputSink::SUPPORTS_SKIPPED_ROWS` replaces the earlier blanket statement that sinks cannot use +branch-and-skip. `ElementSink` pre-fills `OutputElement::placeholder` and supports skipped rows. +A custom sink may do the same, or decline and let the lifting filter and scatter. The semantic +contract remains that skipped values are legal but arbitrary and are masked before the result +escapes. + +### Final executor measurements and IR + +The authoritative `row_fn_executor` run used 65,536 `i64` rows, 100 samples, a one-second minimum +per arm, TSC timing, CPU 4, and a performance governor on the Ryzen 9 7950X. Each cell is the range +across two runs as fastest / median: + +| workload | specialized | sink-only `RowFn` | specialized / `RowFn` | +| --- | ---: | ---: | ---: | +| checked add, two columns | 131.5-132.3 / 132.3-133.3 us | 128.4-129.6 / 129.5-130.9 us | 1.021-1.024x / 1.018-1.022x | +| checked add, column and constant | 16.90-16.93 / 17.10-17.21 us | 13.82-13.85 / 14.04 us | 1.222x / 1.218-1.226x | +| checked add, nullable columns | 133.8-134.8 / 136.1 us | 128.4-128.7 / 130.6-131.8 us | 1.042-1.047x / 1.033-1.042x | + +The native release IR has `<8 x i64>` vector error-word accumulators and +`llvm.vector.reduce.or.v8i64`. The two-column assembly is four-way unrolled over AVX-512 `zmm` +registers, producing 32 `i64` rows per iteration with four `vpaddq` instructions. Overflow bits +accumulate through vector xor/ternary-OR operations and reduce after the loop; there is no per-row +result discriminant or error branch. The specialized arm remains benchmark-local, and no production +deferred-error user exists yet. + +Other final diagnostic medians: + +- `strict_validity` lazy versus eager stayed within 2% across 65,536 and 1,048,576 rows, including + a chain of three calls. +- `byte_length_element` found `BytesLen` 1.410-1.411x faster by median than resolving a byte slice + for long strings and 1.097x for short/inlined strings at 65,536 rows. This justifies the element + choice but is not a production benchmark. +- `null_strategy_bytes` auto matched branch-and-skip; at 90% nulls it took 24.95 us against + 175.4 us for filter-and-scatter. +- Geo auto broadly tracks branch at dense validity and filter at sparse validity, but the controlled + x86 run found the two threshold misses recorded above. The full forced-strategy matrix remains an + implementation diagnostic, not permanent CodSpeed coverage. +- Distinct per-row LIKE patterns took 126.4 us against 26.87 us for a repeated pattern, 4.7x + slower. That is the measured reason LIKE remains a stateful columnar implementation. + +### Durable benchmark boundary + +Draft PR [#9136](https://github.com/vortex-data/vortex/pull/9136) now owns the stable production +benchmark names. At `bf814bbe02cb` it covers public-path byte length; signed and unsigned add, +including constant and nullable inputs; repeated and distinct LIKE patterns; tensor functions and +the `Normalized` encoding; and geo contains, intersects, and distance with constant and nullable +shapes. It also reduces the expensive overlapping-contains simulation to 1,024 rows and uses +vendored `mimalloc` in allocating binaries. + +Do not merge the research harnesses above into that permanent suite. They compare internal +strategies or frozen controls that do not exist on develop. Land #9136 first, then use its identical +benchmark names to gate each production implementation PR through CodSpeed's compiled amd64/AVX2 +simulation. Keep local Divan for real wall-clock diagnosis and generated IR for explaining a +regression. + +### Final API consequence + +Issue 9129's current sketch is obsolete: it still has `RetWitness`, `visit`, `visit_prepared`, and +`visit_into`. Issue 9130 still says sink-backed execution cannot branch-and-skip. Update both before +using their checklists to cut the implementation stack. The prototype to carry forward is: + +```text +RowFn + -> dispatches Args + OutputSink through visit_prepared_into + -> private Batch lifting chooses dense, branch-and-skip, or filter-and-scatter + -> ElementSink covers ordinary output + -> custom sinks cover runtime shape and deferred errors + -> ScalarFnVTable blanket impl exposes the function +``` + +Nullable outputs remain separate. A sink can build values plus validity, but doing so invalidates +the unconditional `validity() = union_child_validities` derivation. That semantic change should +land with its first strict non-total user, not inside the initial sink executor. + +--- + +## Final API simplification review + +This section supersedes every earlier API sketch in this document. In particular, do not carry +forward `ArgsWitness`, `RetWitness`, `PersistableOptions`, public `NullHandling`, +`DECODE_SHRINKS_WHEN_FILTERED`, or `TensorSink`. + +The review started from two constraints. The public API should expose only decisions a function +author can meaningfully make, and the executor should not trust facts fabricated by downstream +implementations. Applying both constraints removed more framework surface without preventing a +function from defining domain-specific rows. + +### The final extension boundary + +The framework is selectively sealed: + +- `RowFn` remains open. It names the function, options, argument names, fallibility, persistence, + and dtype-based dispatch. +- `InputElement` remains open. This is how a crate adds a new decoder for a geometry, tensor view, + byte view, or another domain scalar. +- `OutputElement` remains open for ordinary one-value-per-row outputs. +- `OutputSink` remains open for output representations that need their own builder or row state. +- `RowVisitor`, `ElementTuple`, and `SinkResult` are sealed because their implementations assert + executor facts used by the blanket vtable. + +Sealing `ElementTuple` does not seal decoding. The framework supplies tuple recursion for arities 0 +through 12, and a function places any open `InputElement` implementation inside those tuples. +Sealing `SinkResult` likewise does not seal output representation. A custom `OutputSink` selects one +of the supplied result behaviors. + +This keeps the author vocabulary extensible while avoiding public implementations that can lie +about arity, dense safety, result fallibility, deferred errors, or skipped-row support. + +### Dispatch contains its own evidence + +`RowFn` no longer has argument or return witnesses. `ARG_NAMES.len()` is the exact arity. The types +selected by `dispatch` carry the remaining evidence: + +```text +(InputElement, ...) + OutputSink + SinkResult + -> arity and decode properties + -> output representation and dtype + -> row fallibility and deferred-error word +``` + +The visitor asserts at compile time that the dispatched tuple arity matches `ARG_NAMES`, a +fallible decoder or result implies `RowFn::FALLIBLE`, and deferred evidence is accepted by the +selected sink. These are implications rather than equalities. A function may conservatively +declare `FALLIBLE = true` while selecting an infallible arm for some dtypes. + +This is enough for planning because dispatch is pure in `(options, args)`. It is also simpler than +duplicating the same tuple in a witness and every dispatch arm, then proving that the declarations +agree. + +### Persistence follows the function ID + +`Options: PersistableOptions` assigned one wire contract to a Rust type. That was the wrong owner. +Two functions may reuse an options type while choosing different encodings or serializability, and +an unregistered function should not invent persistence merely because its options type supports it. + +The final `RowFn` therefore owns `serialize` and `deserialize` hooks. Serialization defaults to +`Ok(None)`, and deserialization defaults to an error. Registered tensor and geo functions preserve +their explicit existing formats. The unregistered `NumericBinary` needs no otherwise-unused +serialization implementation for `NumericOperator`. + +### One custom sink is enough + +`OutputSink` already permits arbitrary internal state. A function that needs two builders defines +one sink with two fields rather than asking the executor to understand pairs of sinks. The same +rule applies to other composite or runtime-shaped results: express the shape inside one sink and +add framework abstraction only after two real users expose shared mechanics. + +The public `TensorSink` had no user after `l2_denorm` became the `Normalized` encoding. `l2_norm`, +inner product, and cosine similarity all return scalar rows through `ElementSink`. Removing +`TensorSink` avoids stabilizing roughly 90 lines of runtime-shaped row behavior without preventing +a future tensor-valued function from defining a private sink. + +`ElementSink` also no longer needs an `ElementRow` wrapper. Its row is `&mut T`, and a closure +writes with `*output = value`. The sink still pre-fills legal placeholders so branch-and-skip may +leave masked rows untouched. + +### Per-argument filtered-decode cost + +The aggregate `DECODE_SHRINKS_WHEN_FILTERED` flag was measurably lossy. OR-ing the flag made one +expensive decode indistinguishable from two, even though the x86 data selected opposite mechanisms: + +- one nullable geometry argument at 50% nulls favored branch-and-skip; and +- two independently nullable geometry arguments at 10% nulls, about 81% surviving rows, favored + filter-and-scatter. + +`InputElement::FILTERED_DECODE_COST` now defaults to zero, and each tuple adds the costs of all its +arguments. The batch selector uses the following coarse policy: + +- cost 0 always branches; +- cost 1 branches at 50% or more survivors; and +- cost 2 or greater branches at 85% or more survivors. + +The exact values come from the measured cases rather than a general cost model. There is not yet +enough evidence to distinguish two costly arguments from three, or to make the crossover depend on +batch size. Keep the value additive so a later selector can use that information without another +author-facing API change. + +The old public `NullHandling` enum is gone. The executor privately derives `Dense`, +`DenseWithRetry`, or `ValidOnly { filtered_decode_cost }`. Authors declare local safety and cost on +their input/result types, not a global mechanism. `NullStrategy` survives only in the test harness +to force branch-and-skip or filter-and-scatter. + +### Deferred errors stay in a loop-local word + +The numeric migration confirmed two constraints on deferred error evidence: + +- the accumulated word must be no wider than the element type; and +- the accumulator must live in the generated loop, not behind a mutable sink reference. + +The sealed `SinkResult` implementations for `bool`, `u8`, `u16`, `u32`, and `u64` preserve both. +Checked multiplication can report discarded high bits directly, LLVM can accumulate those words in +vectors, and `finish` turns the final evidence into the function error. `VortexResult<()>` remains +the separate early-exit form for a row that cannot write a legal provisional value. + +### Code generation after the simplification + +The final cleanup at `4becc863ae` was compared with parent `53c51d803c` using rustc 1.91.0 and LLVM +21.1.2. Both revisions were cross-compiled with: + +```bash +cargo rustc -p vortex-array --bench row_fn_executor --profile bench \ + --target x86_64-apple-darwin -- \ + --emit=llvm-ir -C codegen-units=1 -C target-cpu=x86-64-v3 +``` + +The optimized executor monomorphs were normalized to remove revision-specific symbol names and +metadata. Their vector/reduction block hashes matched exactly for wrapping add through +`ElementSink`, checked add with deferred evidence, and wrapping add through the custom `I64Sink`. + +The two wrapping paths retain 256-bit `<4 x i64>` loads, adds, and stores across six vector loop +bodies covering constant and varying inputs. Checked add retains `<4 x i64>` arithmetic, derives +overflow with vector xor/and/compare operations, ORs `<4 x i1>` evidence in the vector loop, and +reduces after the loop. The vector bodies have no calls or panic references. Scalar tails are +present in both revisions. + +The production tensor benchmark IR was checked separately for `l2_norm`, inner product, and cosine +similarity. After normalizing SSA and metadata, arithmetic sequences and instruction counts matched +between revisions for both `f32` and `f64`. Their ordered floating-point reductions remain +eightfold scalar-unrolled in both revisions. They were not vectorized before the cleanup, so the +API change did not cause that property. + +Native Apple M4 Max `row_fn_executor` timings used 65,536 rows, two alternating revisions, 100 +samples, and a 0.5-second minimum per arm. RowFn median deltas ranged from 1.11% faster to 0.94% +slower. Fastest deltas stayed within approximately 0.17%, while specialized controls had median +drift as high as 3.7%. That is no measurable native regression. + +This evidence is deliberately bounded. Cross-target optimized IR shows that the API cleanup did +not change the x86_64-v3 hot loops. It cannot establish the runtime effect of the new null selector +on an x86 branch predictor. Re-run the measured null shapes on x86 before changing or declaring the +50% and 85% thresholds settled. + +### Required x86 rerun + +The next session will run on an x86 machine. It must rerun the production comparison before this +performance record is considered complete. The #9136 benchmark baseline is now on `develop` at +`9a482c0230`, including the public binary, tensor, and geo benchmark binaries used by this work. +Fetch the latest `origin/develop`, record both exact revisions, and compare the branch against +`develop` with the same benchmark names. + +Run `binary_ops` and `like` from `vortex-array`. Run `l2_norm`, `inner_product`, +`cosine_similarity`, and `normalized` from `vortex-tensor`. Run `binary_predicates`, `distance`, +`envelope`, and `predicate_bbox` from `vortex-spatial`. Use at least two alternating runs per +revision. +If the host permits it, pin one core. Report both fastest and median values with the CPU, timer, and +governor configuration. + +The stable production binaries are the cross-revision gate because they now exist on `develop`. +The branch-only `vortex-spatial` `null_strategies` benchmark remains the forced-policy diagnostic. Run +it on the same x86 host to verify both measured selector decisions: one costly decode at 50% +survivors must select the faster mechanism, and two costly decodes at about 81% survivors must do +the same. Inspect optimized LLVM IR again for any stable regression before changing the API or the +selector. + +### Final verification state + +The final API state recorded 67 focused RowFn tests, 179 tensor tests, and 230 geo tests. Nightly +formatting passed. Full workspace clippy passed with +`PYO3_NO_PYTHON=1 PYO3_BUILD_EXTENSION_MODULE=1`, required because the host `/usr/bin/python3` is +3.9 while the workspace targets the Python 3.11 stable ABI. + +Issues #9128, #9129, and #9130 were updated to this API. The durable public-path benchmark baseline +from #9136 is now in the repository. Earlier statements in this document that those issues or the +baseline still need updating are historical only. diff --git a/docs/strictness-and-validity-pushdown.typ b/docs/strictness-and-validity-pushdown.typ new file mode 100644 index 00000000000..d45d87a37c9 --- /dev/null +++ b/docs/strictness-and-validity-pushdown.typ @@ -0,0 +1,243 @@ +#set page(paper: "a4", margin: 2.2cm, numbering: "1 / 1") +#set text(font: "Libertinus Serif", size: 10.5pt) +#set par(justify: true, leading: 0.62em) +#set heading(numbering: "1.") +#show heading: it => block(above: 1.4em, below: 0.8em, it) +#show raw: it => text(font: "Noto Sans Mono", size: 0.88em, it) +#set table(stroke: 0.4pt + luma(65%), inset: 5pt) + +#let mask = math.op("mask") +#let valid = math.op("valid") +#let N = text(fill: rgb("#b03a2e"), weight: "bold", [NULL]) + +#let node(body, fill: luma(96%)) = box( + inset: (x: 7pt, y: 5pt), radius: 3pt, stroke: 0.5pt + luma(55%), fill: fill, body, +) + +#let lead(body) = block( + inset: (x: 10pt, y: 8pt), radius: 3pt, fill: luma(97%), + stroke: (left: 2pt + rgb("#2c3e50")), width: 100%, body, +) + +#align(center)[ + #text(size: 17pt, weight: "bold")[Strictness and validity push-down] + #v(-0.4em) + #text(size: 12pt)[the same value law, once partiality is accounted for] +] + +#v(1em) + +#lead[ + *Summary.* A row-local function may be pushed through an input's validity exactly when it is strict + in that argument *and* remains defined after validity masks that argument. The first condition is the + usual null-propagation meaning of `is_strict`; the second matters only for partial functions. It is + automatic for an infallible function. Return-dtype representability, totality, speculative errors, + and `Dense` safety remain separate concerns. +] + += Model + +Scalar functions are *row-local*: output row $i$ depends only on input rows $i$. They are also assumed +deterministic and insensitive to the bytes behind nulls. Equality below is therefore *logical equality* +$eq.triple$: equal length, equal validity, and equal values at valid rows. + +A mask is a non-nullable boolean column. It applies validity without changing valid values: + +$ mask(a, m)[i] = cases(#N &"if" not m[i], a[i] &"otherwise") $ + +For example, masking does not distinguish a newly nulled row from one that was already null: + +#figure( + table( + columns: 4, + align: center, + table.header([$i$], [$a$], [$m$], [$mask(a, m)$]), + [0], [10], [`true`], [10], + [1], [20], [`false`], N, + [2], N, [true], N, + ), + caption: [Rows 1 and 2 are both null after masking, for different reasons.], +) + +The function $f$ may be partial: an evaluation can error instead of returning a column. Statements +about its result are quantified only where that evaluation succeeds. + += The law and its missing premise + +Fix an argument position $j$. + +#lead[ + *$(S_j)$ Strictness.* If $f(a_1, ..., a_k)$ succeeds and $a_j[i] = #N$, its output at $i$ is #N. + + *$(C_j)$ Mask closure.* If $f(a_1, ..., a_k)$ succeeds, then + $f(a_1, ..., mask(a_j, m), ..., a_k)$ succeeds for every mask $m$. + + *$(M_j)$ Validity equivariance.* Whenever $f(a_1, ..., a_k)$ succeeds, the masked evaluation also + succeeds and + $ f(a_1, ..., mask(a_j, m), ..., a_k) eq.triple mask(f(a_1, ..., a_k), m). $ +] + +$(M_j)$ is the law used by a validity push-down: compute after masking one argument, or compute first +and mask the result. It includes definedness of both sides, rather than treating an error as a value. + +#pagebreak() + +For an ordinary addition, $(M_1)$ says the following two columns agree. The evaluation after masking +is defined, and strictness makes its second row null. + +#figure( + table( + columns: 6, + align: center, + table.header( + [$i$], [$a_1$], [$a_2$], [$m$], + [mask first, then add], [add first, then mask], + ), + [0], [1], [10], [`true`], [11], [11], + [1], [2], [20], [`false`], N, N, + [2], [3], [30], [`false`], N, N, + ), + caption: [The two orders differ only in the unobserved bytes behind null rows.], +) + +#lead[ + *Theorem.* For a row-local deterministic function, + $ (S_j) " and " (C_j) quad arrow.l.r quad (M_j). $ + Consequently, full strictness plus mask closure in every argument is exactly what licenses every + per-argument validity push-down. +] + +== Forward: strictness and closure imply the law + +Assume $(S_j)$ and $(C_j)$, and start with any successful evaluation +$f(a_1, ..., a_k)$. By closure, the left side below also succeeds. Fix a row $i$; row-locality means +there are only two cases to check: + +#figure( + table( + columns: (auto, 1fr, 1fr), + align: (center, left, left), + table.header([mask bit], [left: compute after masking], [right: mask after computing]), + [$m[i] = $ `true`], + [the input at row $i$ is unchanged, so this is $f(a_1, ..., a_k)[i]$], + [masking preserves $f(a_1, ..., a_k)[i]$], + [$m[i] = $ `false`], + [argument $j$ is #N; the successful left evaluation is #N by $(S_j)$], + [the mask makes the result #N by definition], + ), + caption: [Each row agrees, so the columns are logically equal.], +) + +This proves $(M_j)$. Notice the distinct jobs of the two premises: closure establishes that the left +evaluation exists; strictness establishes its value at masked rows. + +== Reverse (by contrapositive): the law implies strictness and closure + +$(M_j)$ explicitly includes $(C_j)$. To obtain $(S_j)$, use its contrapositive: suppose a successful +input $b$ has a null in argument $j$ at row $i$, but gives a non-null result $v$ there. This is exactly +the negation of $(S_j)$, and we will derive a contradiction with $(M_j)$. + +Choose a mask $m$ that is false only at $i$, and write +$b'_j = mask(b_j, m)$. At row $i$, $b_j[i]$ was already #N; at every other row, $m$ is true. Thus +$b'_j eq.triple b_j$. Replacing $b_j$ by $b'_j$ changes no logical input value, including at the one +row we care about. + +Now apply $(M_j)$ to the successful input $b$. Its left-hand side is precisely the evaluation with +$b'_j = mask(b_j, m)$, and it guarantees that evaluation succeeds. At row $i$, the common left-hand +side has these two incompatible values: + +$ f(b_1, ..., mask(b_j, m), ..., b_k)[i] + = f(b_1, ..., b'_j, ..., b_k)[i] + = f(b_1, ..., b_j, ..., b_k)[i] = v != #N. $ + +But $(M_j)$ also says + +$ f(b_1, ..., mask(b_j, m), ..., b_k)[i] + = mask(f(b_1, ..., b_j, ..., b_k), m)[i] = #N. $ + +The first line uses the definition of $b'_j$, then row-locality and $b'_j eq.triple b_j$; the second is +$(M_j)$ and $m[i] = $ `false`. We do not use $(S_j)$ here --- it is the fact being proved. One successful +evaluation cannot be both $v$ and #N, so the assumed counterexample cannot exist. Therefore $(M_j)$ +implies $(S_j)$. $square.stroked$ + +#pagebreak() + +The closure premise is necessary. A binary function that succeeds on $(0, 1)$, errors on $(#N, 1)$, +and otherwise returns null whenever it does evaluate with a null first argument satisfies $(S_1)$ under +the partiality convention, but not $(M_1)$: masking the first input turns a successful evaluation into +an error. Defining strictness to require a *successful* null result on every null input is an equivalent +way to build this premise into $(S_j)$. + +#figure( + table( + columns: 4, + align: center, + table.header([input], [$f$], [after masking argument 1], [$f$ after masking]), + [$(0, 1)$], [0], [$(#N, 1)$], [*error*], + ), + caption: [The function is vacuously strict at $(#N, 1)$ because it does not return a non-null value; + nevertheless, it cannot satisfy the masked-evaluation law.], +) + += What the optimizer uses + +The dictionary rule has the shape + +#align(center)[ + #grid( + columns: 3, column-gutter: 1.2em, align: horizon, + node[`f(dict(codes, values), c)`], + text(size: 13pt)[$arrow.r.long$], + node(fill: rgb("#eafaf1"))[`dict(codes, f(values, c))`], + ) +] + +A null code masks only the dictionary argument while $c$ stays live, so this requires $(M_j)$ for that +argument, not a weaker law that masks all arguments together. Kleene `AND` illustrates the difference: +`false AND NULL` is `false`, so masking only its second argument is not equivariant. + +#table( + columns: 6, + align: center, + table.header( + [$a_1$], [$a_2$], [$m$], [mask $a_2$, then `AND`], [`AND`, then mask], [result], + ), + [`false`], [`true`], [`false`], [`false`], N, [not $(M_2)$], +) + +Value equivalence is not enough for this rewrite when $f$ is fallible. It evaluates *every* dictionary +value, including values with no live code; `div(100, 0)` can then error on the rewritten side although +the original never evaluated it. Thus the dictionary rule also needs its existing no-speculative-error +condition (normally `!is_fallible`). Mask closure addresses masked input rows; it does not make dead +dictionary values safe to evaluate. + += Independent obligations + +#table( + columns: (auto, 1fr, 1fr), + align: (left, left, left), + table.header([property], [statement], [what it enables]), + [strict + mask-closed], [null inputs produce null outputs and remain evaluable], + [validity push-down], + [representable], [the declared return dtype admits required nulls], + [advertising `is_strict`], + [total], [valid inputs never produce null], + [precomputing output validity], + [infallible], [no legal evaluation errors], + [speculative evaluation], + [dense-safe], [bytes behind nulls may be read safely], + [`NullHandling::Dense`], +) + +Representability is a type-level obligation: a strict `cast` with a pinned non-nullable return type +cannot represent the null its value semantics demand. Totality is different again. A strict `list_sum` +may return null for a valid empty list, so strictness only gives + +$ valid(f(a_1, ..., a_k)) subset.eq valid(a_1) " and " dots " and " valid(a_k). $ + +Equality, and hence a precomputed output-validity mask, additionally needs totality. + +`RowFn` supplies strictness structurally. Its `Filter` path evaluates only rows valid in every input and +scatters nulls back; its `Dense` path evaluates all rows then applies that combined validity. The latter +still needs `InputElement::DENSE_SAFE`, because an invalid string view may hold unsafe bytes. That is an +operational property of an element representation, not a consequence of strictness. diff --git a/research/rowfn-x86-2026-08-07/README.md b/research/rowfn-x86-2026-08-07/README.md new file mode 100644 index 00000000000..5a657f47fa0 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/README.md @@ -0,0 +1,276 @@ + + + +# RowFn owned-output and x86 numeric research + +This is the durable record for the investigation that produced the owned-output RowFn path. The +result is not that RowFn is inherently difficult to optimize. The declaration must distinguish an +independent returned value from a stateful output sink, and dense primitive inputs must cross a +validated indexed-source boundary that shared execution can lower directly. + +The selected implementation restores `i64` and `u64` varying multiplication to within about 1% of +the actual merge-base throughput. It does so without a numeric array downcast, `reduce_encoded` +override, numeric-owned allocation, or numeric-specific null and constant policy. + +## Revisions and environment + +- Merge-base baseline: `19f771f2a426103aa7d1bf7153a258bb1bab1e19`. +- Untouched sink-only RowFn: `35098c72118f1b555a24bd2f9b58b0400fa46dc5`. +- Selected implementation: `1a0a055c752b54448c8e1d54af032fe43acf8517`. +- Selected diff fingerprint: + `928e7a0baa2895609d102c98d110c21fb7a12e079b04195b85903277c71537a2`. + +The research branch has older tensor and spatial RowFn users. The result was ported rather than +rebased so that history remains intact. Its port also backports `map_checked_into`, which already +exists at the mergeable branch's base. + +```text +AMD Ryzen 9 7950X +1 socket, 16 physical cores, 32 threads +benchmark logical CPU: 8; SMT sibling: 24 +Linux CTCachyDesktop 7.1.6-1-cachyos, x86_64 +rustc 1.91.0, LLVM 21.1.2 +cargo 1.91.0 +``` + +The CPU reports AVX2 and AVX-512F/DQ/BW/VL. Builds used the default repository target and bench +profile without LTO, `target-cpu=native`, profile changes, or forced inlining. The scaling governor +was `performance`. Timed executions were pinned to CPU 8 and never overlapped compilation. + +```bash +taskset -c 8 "$BENCH" --bench --sample-count 100 --max-time 0.5 --color never \ + mul_i8_nonnull mul_u8_nonnull mul_i16_nonnull mul_u16_nonnull \ + mul_i32_nonnull mul_u32_nonnull mul_i64_nonnull mul_u64_nonnull \ + add_i64_nonnull add_i64_constant sub_i64_constant \ + mul_i32_constant mul_i32_nullable div_i64_nonnull +``` + +Every file in [`benchmarks`](benchmarks) is unedited Divan output wrapped in Markdown. It includes +fastest, slowest, median, mean, samples, and iterations rather than only the selected medians. + +## Stage 0: reproduction + +Order: baseline, candidate, baseline, candidate. Values are median microseconds. + +| Benchmark | Baseline 1 / 2 | Candidate 1 / 2 | Candidate/baseline | +| --- | ---: | ---: | ---: | +| `add_i64_constant` | 8.449 / 8.399 | 9.269 / 9.290 | 1.097 / 1.106 | +| `add_i64_nonnull` | 9.205 / 9.149 | 9.455 / 9.490 | 1.027 / 1.037 | +| `div_i64_nonnull` | 44.850 / 44.800 | 45.090 / 45.160 | 1.005 / 1.008 | +| `mul_i8_nonnull` | 6.184 / 6.199 | 4.694 / 4.699 | 0.759 / 0.758 | +| `mul_i16_nonnull` | 4.099 / 4.119 | 4.269 / 4.269 | 1.041 / 1.036 | +| `mul_i32_constant` | 26.420 / 26.430 | 18.880 / 18.870 | 0.715 / 0.714 | +| `mul_i32_nonnull` | 26.410 / 26.420 | 28.390 / 28.350 | 1.075 / 1.073 | +| `mul_i32_nullable` | 27.350 / 27.400 | 29.180 / 29.150 | 1.067 / 1.064 | +| `mul_i64_nonnull` | 23.220 / 23.200 | 30.020 / 30.080 | **1.293 / 1.297** | +| `mul_u8_nonnull` | 3.319 / 3.329 | 3.539 / 3.529 | 1.066 / 1.060 | +| `mul_u16_nonnull` | 2.599 / 2.599 | 2.429 / 2.429 | 0.935 / 0.935 | +| `mul_u32_nonnull` | 6.939 / 6.949 | 7.069 / 7.059 | 1.019 / 1.016 | +| `mul_u64_nonnull` | 19.210 / 19.190 | 30.430 / 30.490 | **1.584 / 1.589** | +| `sub_i64_constant` | 8.255 / 8.239 | 9.099 / 9.099 | 1.102 / 1.104 | + +The x86 regression reproduced. Raw runs are the four `stage0-*` files. + +## Stage 1: owned output without indexed input + +The closure returned `(output, failure)`, shared execution owned the store, and failure remained a +loop-local OR. This removed the numeric checked sink and materially improved 64-bit cases, but did +not solve the general problem. + +| Benchmark | Baseline 1 / 2 | Owned 1 / 2 | Owned/baseline | +| --- | ---: | ---: | ---: | +| `mul_i64_nonnull` | 23.20 / 23.26 | 25.65 / 25.59 | 1.106 / 1.100 | +| `mul_u64_nonnull` | 19.18 / 19.21 | 19.41 / 19.41 | 1.012 / 1.010 | +| `mul_i32_constant` | 26.43 / 26.44 | 32.36 / 32.38 | 1.224 / 1.225 | +| `mul_i32_nonnull` | 26.42 / 26.41 | 31.24 / 31.23 | 1.182 / 1.183 | +| `mul_i32_nullable` | 27.36 / 27.35 | 32.04 / 32.04 | 1.171 / 1.171 | + +The six `stage1-*` files contain the full matrix. This falsifies output ownership as a complete +explanation: it matters, but does not give LLVM the specialized kernel's input representation. + +## Stage 2: indexed dense input + +`IndexedElementTuple` lets a primitive pair expose `LaneZip<&[Left], &[Right]>` after shared +execution validates both varying lengths once. The generic owned executor calls +`map_checked_into`; numeric code still declares only row types, operation, failure, and error. + +| Benchmark | Baseline 1 / 2 | Indexed 1 / 2 | Candidate 1 / 2 | +| --- | ---: | ---: | ---: | +| `mul_i32_nonnull` | 26.39 / 26.41 | 26.58 / 26.60 | 28.34 / 28.36 | +| `mul_i32_nullable` | 27.37 / 27.38 | 27.41 / 27.43 | 29.20 / 29.17 | +| `mul_i64_nonnull` | 23.22 / 23.24 | 23.43 / 23.44 | 30.02 / 30.10 | +| `mul_u64_nonnull` | 19.22 / 19.21 | 19.41 / 19.42 | 30.41 / 30.43 | +| `div_i64_nonnull` | 44.84 / 44.87 | 45.07 / 45.03 | 45.07 / 45.12 | +| `mul_i32_constant` | 26.42 / 26.43 | 32.38 / 32.39 | 18.88 / 18.88 | + +The indexed source closed the varying and nullable gap. It did not affect mixed constants, which +exposed the next compiler-sensitive detail. + +## Compiler ablations: `Copy`, source order, and whole-function sensitivity + +The completed ablation matrix isolates the public `Output: Copy` bound as a reliable trigger, while +falsifying the simpler explanations considered during the initial investigation: + +| Variant | `mul_i32_constant` run 1 / 2 | +| --- | ---: | +| No `Copy` bound | 18.77 / 18.72 us | +| Inert private marker bound | 18.77 / 18.72 us | +| `Output: Copy` | 29.94 / 29.93 us | +| `Output: Copy`, `codegen-units=1` | 29.87 / 29.89 us | + +The `i64` and `u64` controls did not move. The inert private marker is important: an arbitrary +where-clause or source perturbation is insufficient to trigger the loss. The result is specific to +the optimizer-visible `Copy` constraint, though the mechanism is not yet known. + +The default-CGU DWARF ranges show large whole-function differences for the exact `i32 CheckedMul` +monomorph. The `Copy` function spans `0xe58c90..0xe59adc` (`0xe4c` bytes); no-Copy spans +`0xe7a1c0..0xe7b6d0` (`0x1510` bytes). The `Copy` hot loop at `0xe58f90` is only 16-byte aligned and +computes the low multiply before the widened chain. The no-Copy loop at `0xe7b260` is 32-byte +aligned, computes the widened chain first, and delays the low multiply. LLVM-MCA nevertheless +predicts the smaller `Copy` loop slightly better, 2.5 versus 2.7 cycles. Alignment and final loop +scheduling therefore do not explain the measured direction. + +A fresh `Copy` plus `codegen-units=1` build makes this conclusion stronger: its optimized IR already +has store-before-OR, yet the linked benchmark remains at about 29.9 microseconds. Store-before-OR is +neither sufficient nor established as causal. The earlier source-order edit changed production +performance, but it must be described only as another trigger for a whole-function compiler +interaction. An isolated exact-loop hardware ablation also found OR-before-store slightly faster +(0.75-0.77 ns/row) than store-before-OR (0.823-0.825 ns/row), while LLVM-MCA rated both at 2.7 +cycles. The loop's local instruction order cannot explain the production result. + +A standalone generic `MaybeUninit` loop emits identical optimized IR and assembly with and without +`Copy`. The sensitivity therefore needs the real trait, closure, `Vec`, and monomorphization context. +This is currently evidence of compiler phase-order or code-quality sensitivity, not enough to claim +a rustc correctness bug or a specific LLVM bug. The next upstream step is to reduce the real +monomorph while retaining both the timing and whole-function delta, then bisect MIR/LLVM passes and +compiler versions. The executor needs only the no-drop property, so the selected API continues to +enforce `!needs_drop::()` without exposing the harmful, unnecessary `Copy` bound. See the +compact [Copy-ablation evidence](codegen/copy-ablation.md). + +## Final results + +Order: baseline, final, candidate, repeated twice. Values are median microseconds. + +| Benchmark | Baseline 1 / 2 | Candidate 1 / 2 | Final 1 / 2 | Final/baseline | +| --- | ---: | ---: | ---: | ---: | +| `add_i64_constant` | 8.399 / 8.449 | 9.310 / 9.289 | 9.269 / 9.279 | 1.104 / 1.098 | +| `add_i64_nonnull` | 9.159 / 9.239 | 9.374 / 9.449 | 9.379 / 9.389 | 1.024 / 1.016 | +| `div_i64_nonnull` | 44.820 / 44.860 | 45.040 / 45.080 | 45.020 / 45.060 | 1.004 / 1.004 | +| `mul_i8_nonnull` | 6.209 / 6.199 | 4.719 / 4.699 | 6.389 / 6.409 | 1.029 / 1.034 | +| `mul_i16_nonnull` | 4.099 / 4.109 | 4.269 / 4.269 | 4.265 / 4.299 | 1.040 / 1.046 | +| `mul_i32_constant` | 26.440 / 26.440 | 18.890 / 18.840 | 18.690 / 18.700 | **0.707 / 0.707** | +| `mul_i32_nonnull` | 26.410 / 26.420 | 28.350 / 28.390 | 26.590 / 26.640 | 1.007 / 1.008 | +| `mul_i32_nullable` | 27.380 / 27.360 | 29.170 / 29.170 | 27.400 / 27.440 | 1.001 / 1.003 | +| `mul_i64_nonnull` | 23.200 / 23.350 | 30.010 / 30.050 | 23.460 / 23.430 | **1.011 / 1.003** | +| `mul_u8_nonnull` | 3.319 / 3.319 | 3.545 / 3.519 | 3.514 / 3.549 | 1.059 / 1.069 | +| `mul_u16_nonnull` | 2.609 / 2.609 | 2.429 / 2.429 | 2.789 / 2.810 | 1.069 / 1.077 | +| `mul_u32_nonnull` | 6.949 / 6.959 | 7.060 / 7.059 | 7.129 / 7.149 | 1.026 / 1.027 | +| `mul_u64_nonnull` | 19.180 / 19.210 | 30.370 / 30.400 | 19.370 / 19.380 | **1.010 / 1.009** | +| `sub_i64_constant` | 8.239 / 8.259 | 9.114 / 9.079 | 9.149 / 9.159 | 1.110 / 1.109 | + +The six `land-*` logs preserve every final run. Narrow widths avoid the rejected zipped-iterator +experiment's 3x to 9x losses. Constant add/sub retain the untouched candidate's roughly 10% gap; +constant multiplication is faster than merge base. Division stays at parity. + +## Generated code: confirmed evidence + +```bash +CARGO_TARGET_DIR="$TARGET" cargo rustc -p vortex-array --lib --profile bench -- \ + --emit=llvm-ir,asm -C codegen-units=1 -C remark=loop-vectorize +``` + +Full output was about 1.85 GiB IR plus 1.02 GiB assembly and was deleted after extracting exact +production monomorphs into [`codegen`](codegen). These are not fixture or benchmark control loops. + +Baseline, candidate, owned, and final signed `i64` use a scalar one-lane loop: one high/low `imulq`, +one store, `sarq`/`xorq` overflow evidence, register OR, and one backedge. Unsigned `u64` uses two +independent scalar `mulq` groups per backedge plus an odd remainder. Neither final loop has a hot +call, panic edge, bounds check, runtime alias check, or vector body. The second input length check is +an `llvm.assume`; loads and stores carry disjoint alias metadata; failure is a register `phi`. + +Therefore host SIMD did not hide a deficient loop. The default build did not enable optional native +AVX features, and LLVM selected the same essential scalar high-half strategy as merge base. See +[`base summary`](codegen/base-codegen-summary.md), +[`final i64 assembly`](codegen/final-i64-mul-dense-s.md), and +[`final u64 assembly`](codegen/final-u64-mul-dense-s.md). + +A separate minimal `target-cpu=native` experiment did form `<8 x i128>` operations in LLVM IR for +the widened `u64` product. The x86 backend still scalarized them into eight `mulq`/`imulq` +instructions, then used ZMM registers only to pack and reduce the scalar results. x86 has no true +wide 64-by-64-to-128 integer multiply here. Seeing a vector IR type or ZMM instruction is therefore +not evidence that the expensive multiply itself executed as SIMD. + +`-C remark=loop-vectorize` emitted no remark attributable to the exact dense production loop. The +constant fallback source line had successes for other monomorphs and duplicated cost-model misses, +but diagnostics lacked function identity. Exact IR proves the measured specialization is scalar; +it cannot assign those remarks to it. The merge-base focused remark rebuild was cancelled, so no +merge-base missed-vectorization reason is claimed. + +## Findings + +Confirmed: + +- Bounds checks are not the all-varying blocker; candidate dense multiply had no hot bounds edge. +- `SinkResult` already reduced to a register OR and did not impose a per-row `Result`. +- Output ownership materially helped but was insufficient alone. +- A typed indexed source restored stable parity for varying primitive tuples. +- An `Output: Copy` bound reliably triggers slower LLVM 21.1.2 production codegen; an inert marker + does not. Source store/OR order is neither sufficient nor established as causal. The mechanism is + an unresolved whole-function compiler interaction. +- The default x86 target prefers scalar high-half 64-bit multiply; SIMD is not the recovered speed. + +Still inference: + +- No single alias defect explains the original gap. Baseline and candidate had useful metadata too. +- The nearly identical dense inner loops do not explain all end-to-end timing. Surrounding control + flow, placement, and instruction-cache effects remain candidates. +- Unattributed source-line remarks do not prove a missed-vectorization reason for one monomorph. + +Rejected controls: checked unchecked access only partially helped and regressed some `u8` runs; +direct failure accumulation matched existing IR; safe zipped iterators caused 3x to 9x narrow +losses; a numeric `reduce_encoded` fast path recovered speed by duplicating shared policy; and a +primitive-binary visitor seam moved that specialization into generic execution. Earlier Apple work, +including the non-affine `index & mask` failure, remains in +[`NUMERIC_ROWFN_PLAN.md`](../../NUMERIC_ROWFN_PLAN.md). + +## Why both visitor methods exist + +`visit_prepared_deferred` represents an independent owned value and OR-reducible failure per row. +The executor allocates contiguous output, owns the store, and can use a typed indexed source. It is +intentionally limited to indexed inputs, fixed no-drop output, and a batch-deferred row error. + +`visit_prepared_into` represents stateful construction: shared buffers, runtime-shaped layouts, +multiple coordinated builders, skip-capable output, drop-requiring values, non-indexed tuples, and +ordinary immediate or deferred `SinkResult` forms. Encoding those through the owned method would +either hide a mutable builder reference inside a supposed value, allocate a temporary per row, +forbid legitimate output, or duplicate lifting. Encoding numeric output only through the sink loses +the fact that each value and store are independent. These are distinct capabilities. + +## Indexed source, specialization, and safety + +`InputElement` is open and many elements are not contiguous. Sealed `ElementTuple` is the safe +composition point for unchecked reads after one length validation. Stable Rust cannot overlap a +blanket fallback for every tuple with a more specific associated dense source without +specialization. Runtime erasure would obscure the source type LLVM needs. The indexed capability is +therefore explicit and opt-in; only the proven primitive pair implements it today. + +The executor reserves `row_count` slots and exposes exactly that many `MaybeUninit` values. It +validates varying lengths before `LaneZip`; `map_checked_into` validates output length. Either loop +writes every slot exactly once before `set_len`. On panic the vector length remains zero, and the +compile-time no-drop assertion makes abandoning initialized slots safe. Deferred errors are examined +only after initialization. Nullable lifting retries a deferred error over valid rows, so a failure +shaped value behind null cannot surface. + +## Open improvements + +- Investigate infallible owned output only with a measured caller; avoid a speculative result tree. +- Revisit constant add/sub only with exact production IR and a stable regression. +- Add indexed tuple/element families only for real consumers with a safe source. +- Re-run the store-order and `Copy` ablations after LLVM upgrades. +- Produce an upstream LLVM reproducer for those compiler sensitivities. +- Preserve assembly checks because throughput can hide compensating target-specific instructions. + +The selected branch passed focused checks, 87 numeric tests, 3,385 nextest tests with one skipped, +73 doctests with 13 ignored, nightly formatting, all-target/all-feature clippy, and `diff --check`. +One intermediate 1.85 GiB IR copy hit `ENOSPC`; exact-final codegen later completed. The requested +`ROWFN_FIRST_PR_PROMPT.md` was absent from the repository, fetched refs, home tree, and worktrees. diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md new file mode 100644 index 00000000000..2759e0fa9e5 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-base-1.md @@ -0,0 +1,39 @@ + + + +# `land-base-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.059 µs │ 781.5 µs │ 8.399 µs │ 16.16 µs │ 100 │ 100 +│ 4.065 Gitem/s │ 41.92 Mitem/s │ 3.901 Gitem/s │ 2.027 Gitem/s │ │ +├─ add_i64_nonnull 9.079 µs │ 29.55 µs │ 9.159 µs │ 9.466 µs │ 100 │ 100 +│ 3.608 Gitem/s │ 1.108 Gitem/s │ 3.577 Gitem/s │ 3.461 Gitem/s │ │ +├─ div_i64_nonnull 44.73 µs │ 78.44 µs │ 44.82 µs │ 45.35 µs │ 100 │ 100 +│ 732.4 Mitem/s │ 417.6 Mitem/s │ 731 Mitem/s │ 722.5 Mitem/s │ │ +├─ mul_i8_nonnull 5.819 µs │ 72.73 µs │ 6.209 µs │ 6.939 µs │ 100 │ 100 +│ 5.63 Gitem/s │ 450.5 Mitem/s │ 5.276 Gitem/s │ 4.721 Gitem/s │ │ +├─ mul_i16_nonnull 4.039 µs │ 66.44 µs │ 4.099 µs │ 4.731 µs │ 100 │ 100 +│ 8.111 Gitem/s │ 493.1 Mitem/s │ 7.992 Gitem/s │ 6.926 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 56.27 µs │ 26.44 µs │ 26.97 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 582.2 Mitem/s │ 1.238 Gitem/s │ 1.214 Gitem/s │ │ +├─ mul_i32_nonnull 26.35 µs │ 38.26 µs │ 26.41 µs │ 26.64 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 856.4 Mitem/s │ 1.24 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.28 µs │ 340.1 µs │ 27.38 µs │ 30.62 µs │ 100 │ 100 +│ 1.2 Gitem/s │ 96.34 Mitem/s │ 1.196 Gitem/s │ 1.069 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 45.96 µs │ 23.2 µs │ 23.55 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 712.9 Mitem/s │ 1.411 Gitem/s │ 1.391 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 52.07 µs │ 3.319 µs │ 3.838 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 629.1 Mitem/s │ 9.87 Gitem/s │ 8.535 Gitem/s │ │ +├─ mul_u16_nonnull 2.539 µs │ 30.81 µs │ 2.609 µs │ 2.888 µs │ 100 │ 100 +│ 12.9 Gitem/s │ 1.063 Gitem/s │ 12.55 Gitem/s │ 11.34 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 26.44 µs │ 6.949 µs │ 7.183 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 1.238 Gitem/s │ 4.714 Gitem/s │ 4.561 Gitem/s │ │ +├─ mul_u64_nonnull 19.11 µs │ 41.4 µs │ 19.18 µs │ 19.53 µs │ 100 │ 100 +│ 1.713 Gitem/s │ 791.4 Mitem/s │ 1.707 Gitem/s │ 1.677 Gitem/s │ │ +╰─ sub_i64_constant 8.109 µs │ 40.38 µs │ 8.239 µs │ 8.604 µs │ 100 │ 100 + 4.04 Gitem/s │ 811.2 Mitem/s │ 3.976 Gitem/s │ 3.808 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md new file mode 100644 index 00000000000..ec40d57a6e2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-base-2.md @@ -0,0 +1,39 @@ + + + +# `land-base-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.319 µs │ 39.44 µs │ 8.449 µs │ 8.813 µs │ 100 │ 100 +│ 3.938 Gitem/s │ 830.8 Mitem/s │ 3.877 Gitem/s │ 3.718 Gitem/s │ │ +├─ add_i64_nonnull 9.169 µs │ 12.56 µs │ 9.239 µs │ 9.304 µs │ 100 │ 100 +│ 3.573 Gitem/s │ 2.608 Gitem/s │ 3.546 Gitem/s │ 3.521 Gitem/s │ │ +├─ div_i64_nonnull 44.77 µs │ 50.37 µs │ 44.86 µs │ 45.15 µs │ 100 │ 100 +│ 731.7 Mitem/s │ 650.4 Mitem/s │ 730.2 Mitem/s │ 725.7 Mitem/s │ │ +├─ mul_i8_nonnull 5.829 µs │ 8.179 µs │ 6.199 µs │ 6.265 µs │ 100 │ 100 +│ 5.62 Gitem/s │ 4.005 Gitem/s │ 5.285 Gitem/s │ 5.23 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 8.029 µs │ 4.109 µs │ 4.15 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 4.08 Gitem/s │ 7.973 Gitem/s │ 7.895 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 35.43 µs │ 26.44 µs │ 26.77 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 924.6 Mitem/s │ 1.239 Gitem/s │ 1.223 Gitem/s │ │ +├─ mul_i32_nonnull 26.37 µs │ 35.11 µs │ 26.42 µs │ 26.84 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 933 Mitem/s │ 1.239 Gitem/s │ 1.22 Gitem/s │ │ +├─ mul_i32_nullable 27.26 µs │ 40.16 µs │ 27.36 µs │ 27.78 µs │ 100 │ 100 +│ 1.201 Gitem/s │ 815.9 Mitem/s │ 1.197 Gitem/s │ 1.179 Gitem/s │ │ +├─ mul_i64_nonnull 23.24 µs │ 32.01 µs │ 23.35 µs │ 23.63 µs │ 100 │ 100 +│ 1.409 Gitem/s │ 1.023 Gitem/s │ 1.402 Gitem/s │ 1.386 Gitem/s │ │ +├─ mul_u8_nonnull 3.26 µs │ 4.759 µs │ 3.319 µs │ 3.349 µs │ 100 │ 100 +│ 10.04 Gitem/s │ 6.884 Gitem/s │ 9.87 Gitem/s │ 9.783 Gitem/s │ │ +├─ mul_u16_nonnull 2.53 µs │ 7.289 µs │ 2.609 µs │ 2.664 µs │ 100 │ 100 +│ 12.94 Gitem/s │ 4.495 Gitem/s │ 12.55 Gitem/s │ 12.29 Gitem/s │ │ +├─ mul_u32_nonnull 6.889 µs │ 12.89 µs │ 6.959 µs │ 7.027 µs │ 100 │ 100 +│ 4.756 Gitem/s │ 2.54 Gitem/s │ 4.708 Gitem/s │ 4.663 Gitem/s │ │ +├─ mul_u64_nonnull 19.13 µs │ 22.83 µs │ 19.21 µs │ 19.29 µs │ 100 │ 100 +│ 1.712 Gitem/s │ 1.435 Gitem/s │ 1.704 Gitem/s │ 1.698 Gitem/s │ │ +╰─ sub_i64_constant 8.139 µs │ 11.06 µs │ 8.259 µs │ 8.297 µs │ 100 │ 100 + 4.025 Gitem/s │ 2.96 Gitem/s │ 3.967 Gitem/s │ 3.949 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md new file mode 100644 index 00000000000..8ec200b7df1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-1.md @@ -0,0 +1,39 @@ + + + +# `land-candidate-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.989 µs │ 1.016 ms │ 9.31 µs │ 19.44 µs │ 100 │ 100 +│ 3.645 Gitem/s │ 32.25 Mitem/s │ 3.519 Gitem/s │ 1.684 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 13.06 µs │ 9.374 µs │ 9.443 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 2.508 Gitem/s │ 3.495 Gitem/s │ 3.469 Gitem/s │ │ +├─ div_i64_nonnull 44.97 µs │ 63.03 µs │ 45.04 µs │ 45.41 µs │ 100 │ 100 +│ 728.5 Mitem/s │ 519.7 Mitem/s │ 727.3 Mitem/s │ 721.5 Mitem/s │ │ +├─ mul_i8_nonnull 4.649 µs │ 60.73 µs │ 4.719 µs │ 5.455 µs │ 100 │ 100 +│ 7.047 Gitem/s │ 539.4 Mitem/s │ 6.942 Gitem/s │ 6.006 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 71.24 µs │ 4.269 µs │ 4.959 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 459.9 Mitem/s │ 7.674 Gitem/s │ 6.607 Gitem/s │ │ +├─ mul_i32_constant 18.79 µs │ 72.37 µs │ 18.89 µs │ 19.53 µs │ 100 │ 100 +│ 1.742 Gitem/s │ 452.7 Mitem/s │ 1.734 Gitem/s │ 1.677 Gitem/s │ │ +├─ mul_i32_nonnull 28.24 µs │ 33.43 µs │ 28.35 µs │ 28.48 µs │ 100 │ 100 +│ 1.159 Gitem/s │ 980.1 Mitem/s │ 1.155 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.01 µs │ 234.9 µs │ 29.17 µs │ 31.37 µs │ 100 │ 100 +│ 1.129 Gitem/s │ 139.4 Mitem/s │ 1.122 Gitem/s │ 1.044 Gitem/s │ │ +├─ mul_i64_nonnull 29.63 µs │ 52.67 µs │ 30.01 µs │ 30.5 µs │ 100 │ 100 +│ 1.105 Gitem/s │ 622 Mitem/s │ 1.091 Gitem/s │ 1.074 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 14.69 µs │ 3.545 µs │ 3.659 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 2.229 Gitem/s │ 9.242 Gitem/s │ 8.953 Gitem/s │ │ +├─ mul_u16_nonnull 2.369 µs │ 13.01 µs │ 2.429 µs │ 2.615 µs │ 100 │ 100 +│ 13.82 Gitem/s │ 2.516 Gitem/s │ 13.48 Gitem/s │ 12.52 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 18.45 µs │ 7.06 µs │ 7.181 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 1.775 Gitem/s │ 4.641 Gitem/s │ 4.562 Gitem/s │ │ +├─ mul_u64_nonnull 30.27 µs │ 43.99 µs │ 30.37 µs │ 30.65 µs │ 100 │ 100 +│ 1.082 Gitem/s │ 744.8 Mitem/s │ 1.078 Gitem/s │ 1.068 Gitem/s │ │ +╰─ sub_i64_constant 8.959 µs │ 31.53 µs │ 9.114 µs │ 9.385 µs │ 100 │ 100 + 3.657 Gitem/s │ 1.038 Gitem/s │ 3.595 Gitem/s │ 3.491 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md new file mode 100644 index 00000000000..84f969b8646 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-candidate-2.md @@ -0,0 +1,39 @@ + + + +# `land-candidate-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.129 µs │ 48.92 µs │ 9.289 µs │ 9.744 µs │ 100 │ 100 +│ 3.589 Gitem/s │ 669.8 Mitem/s │ 3.527 Gitem/s │ 3.362 Gitem/s │ │ +├─ add_i64_nonnull 9.349 µs │ 10.43 µs │ 9.449 µs │ 9.46 µs │ 100 │ 100 +│ 3.504 Gitem/s │ 3.141 Gitem/s │ 3.467 Gitem/s │ 3.463 Gitem/s │ │ +├─ div_i64_nonnull 44.99 µs │ 50.41 µs │ 45.08 µs │ 45.29 µs │ 100 │ 100 +│ 728.1 Mitem/s │ 650 Mitem/s │ 726.8 Mitem/s │ 723.5 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 7.969 µs │ 4.699 µs │ 4.761 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 4.111 Gitem/s │ 6.972 Gitem/s │ 6.881 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 7.669 µs │ 4.269 µs │ 4.308 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 4.272 Gitem/s │ 7.674 Gitem/s │ 7.605 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.77 µs │ 18.84 µs │ 18.95 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.439 Gitem/s │ 1.738 Gitem/s │ 1.728 Gitem/s │ │ +├─ mul_i32_nonnull 28.27 µs │ 45.57 µs │ 28.39 µs │ 28.65 µs │ 100 │ 100 +│ 1.158 Gitem/s │ 718.9 Mitem/s │ 1.154 Gitem/s │ 1.143 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 43.06 µs │ 29.17 µs │ 29.41 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 760.8 Mitem/s │ 1.122 Gitem/s │ 1.113 Gitem/s │ │ +├─ mul_i64_nonnull 29.7 µs │ 35.65 µs │ 30.05 µs │ 30.18 µs │ 100 │ 100 +│ 1.103 Gitem/s │ 918.9 Mitem/s │ 1.09 Gitem/s │ 1.085 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 4.579 µs │ 3.519 µs │ 3.532 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 7.154 Gitem/s │ 9.309 Gitem/s │ 9.275 Gitem/s │ │ +├─ mul_u16_nonnull 2.359 µs │ 3.269 µs │ 2.429 µs │ 2.441 µs │ 100 │ 100 +│ 13.88 Gitem/s │ 10.02 Gitem/s │ 13.48 Gitem/s │ 13.42 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 11.21 µs │ 7.059 µs │ 7.105 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 2.92 Gitem/s │ 4.641 Gitem/s │ 4.611 Gitem/s │ │ +├─ mul_u64_nonnull 30.33 µs │ 34.65 µs │ 30.4 µs │ 30.53 µs │ 100 │ 100 +│ 1.08 Gitem/s │ 945.4 Mitem/s │ 1.077 Gitem/s │ 1.073 Gitem/s │ │ +╰─ sub_i64_constant 8.949 µs │ 12.59 µs │ 9.079 µs │ 9.155 µs │ 100 │ 100 + 3.661 Gitem/s │ 2.6 Gitem/s │ 3.608 Gitem/s │ 3.578 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md b/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md new file mode 100644 index 00000000000..fc941735f9c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-final-1.md @@ -0,0 +1,39 @@ + + + +# `land-final-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.979 µs │ 88.65 µs │ 9.269 µs │ 10.12 µs │ 100 │ 100 +│ 3.649 Gitem/s │ 369.6 Mitem/s │ 3.534 Gitem/s │ 3.237 Gitem/s │ │ +├─ add_i64_nonnull 9.299 µs │ 13.4 µs │ 9.379 µs │ 9.444 µs │ 100 │ 100 +│ 3.523 Gitem/s │ 2.443 Gitem/s │ 3.493 Gitem/s │ 3.469 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 55 µs │ 45.02 µs │ 45.48 µs │ 100 │ 100 +│ 728.8 Mitem/s │ 595.6 Mitem/s │ 727.6 Mitem/s │ 720.4 Mitem/s │ │ +├─ mul_i8_nonnull 5.959 µs │ 44.56 µs │ 6.389 µs │ 6.855 µs │ 100 │ 100 +│ 5.498 Gitem/s │ 735.3 Mitem/s │ 5.128 Gitem/s │ 4.78 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 7.599 µs │ 4.265 µs │ 4.321 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 4.311 Gitem/s │ 7.682 Gitem/s │ 7.581 Gitem/s │ │ +├─ mul_i32_constant 18.6 µs │ 22.22 µs │ 18.69 µs │ 18.81 µs │ 100 │ 100 +│ 1.76 Gitem/s │ 1.474 Gitem/s │ 1.753 Gitem/s │ 1.741 Gitem/s │ │ +├─ mul_i32_nonnull 26.52 µs │ 34.76 µs │ 26.59 µs │ 26.77 µs │ 100 │ 100 +│ 1.235 Gitem/s │ 942.6 Mitem/s │ 1.231 Gitem/s │ 1.223 Gitem/s │ │ +├─ mul_i32_nullable 27.3 µs │ 51.11 µs │ 27.4 µs │ 27.77 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 641 Mitem/s │ 1.195 Gitem/s │ 1.179 Gitem/s │ │ +├─ mul_i64_nonnull 23.37 µs │ 27.17 µs │ 23.46 µs │ 23.56 µs │ 100 │ 100 +│ 1.401 Gitem/s │ 1.206 Gitem/s │ 1.396 Gitem/s │ 1.39 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 59.52 µs │ 3.514 µs │ 4.079 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 550.5 Mitem/s │ 9.322 Gitem/s │ 8.033 Gitem/s │ │ +├─ mul_u16_nonnull 2.709 µs │ 3.799 µs │ 2.789 µs │ 2.798 µs │ 100 │ 100 +│ 12.09 Gitem/s │ 8.623 Gitem/s │ 11.74 Gitem/s │ 11.71 Gitem/s │ │ +├─ mul_u32_nonnull 7.049 µs │ 10.9 µs │ 7.129 µs │ 7.19 µs │ 100 │ 100 +│ 4.648 Gitem/s │ 3.003 Gitem/s │ 4.595 Gitem/s │ 4.556 Gitem/s │ │ +├─ mul_u64_nonnull 19.26 µs │ 22.46 µs │ 19.37 µs │ 19.45 µs │ 100 │ 100 +│ 1.7 Gitem/s │ 1.458 Gitem/s │ 1.691 Gitem/s │ 1.683 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 13.94 µs │ 9.149 µs │ 9.215 µs │ 100 │ 100 + 3.636 Gitem/s │ 2.348 Gitem/s │ 3.581 Gitem/s │ 3.555 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md b/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md new file mode 100644 index 00000000000..6588d08f1e1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/land-final-2.md @@ -0,0 +1,39 @@ + + + +# `land-final-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 73.57 µs │ 9.279 µs │ 9.987 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 445.3 Mitem/s │ 3.531 Gitem/s │ 3.28 Gitem/s │ │ +├─ add_i64_nonnull 9.319 µs │ 14.28 µs │ 9.389 µs │ 9.475 µs │ 100 │ 100 +│ 3.515 Gitem/s │ 2.293 Gitem/s │ 3.489 Gitem/s │ 3.458 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 62.15 µs │ 45.06 µs │ 45.46 µs │ 100 │ 100 +│ 728.6 Mitem/s │ 527.2 Mitem/s │ 727 Mitem/s │ 720.7 Mitem/s │ │ +├─ mul_i8_nonnull 5.979 µs │ 41.01 µs │ 6.409 µs │ 6.853 µs │ 100 │ 100 +│ 5.479 Gitem/s │ 798.8 Mitem/s │ 5.112 Gitem/s │ 4.78 Gitem/s │ │ +├─ mul_i16_nonnull 4.229 µs │ 5.739 µs │ 4.299 µs │ 4.314 µs │ 100 │ 100 +│ 7.746 Gitem/s │ 5.708 Gitem/s │ 7.62 Gitem/s │ 7.595 Gitem/s │ │ +├─ mul_i32_constant 18.6 µs │ 23.91 µs │ 18.7 µs │ 18.79 µs │ 100 │ 100 +│ 1.76 Gitem/s │ 1.369 Gitem/s │ 1.751 Gitem/s │ 1.743 Gitem/s │ │ +├─ mul_i32_nonnull 26.56 µs │ 30.13 µs │ 26.64 µs │ 26.74 µs │ 100 │ 100 +│ 1.233 Gitem/s │ 1.087 Gitem/s │ 1.229 Gitem/s │ 1.225 Gitem/s │ │ +├─ mul_i32_nullable 27.35 µs │ 42.74 µs │ 27.44 µs │ 27.69 µs │ 100 │ 100 +│ 1.197 Gitem/s │ 766.5 Mitem/s │ 1.193 Gitem/s │ 1.183 Gitem/s │ │ +├─ mul_i64_nonnull 23.31 µs │ 27.31 µs │ 23.43 µs │ 23.56 µs │ 100 │ 100 +│ 1.405 Gitem/s │ 1.199 Gitem/s │ 1.398 Gitem/s │ 1.39 Gitem/s │ │ +├─ mul_u8_nonnull 3.479 µs │ 52.76 µs │ 3.549 µs │ 4.103 µs │ 100 │ 100 +│ 9.416 Gitem/s │ 620.9 Mitem/s │ 9.231 Gitem/s │ 7.984 Gitem/s │ │ +├─ mul_u16_nonnull 2.739 µs │ 3.799 µs │ 2.81 µs │ 2.824 µs │ 100 │ 100 +│ 11.96 Gitem/s │ 8.623 Gitem/s │ 11.66 Gitem/s │ 11.6 Gitem/s │ │ +├─ mul_u32_nonnull 7.089 µs │ 10.28 µs │ 7.149 µs │ 7.207 µs │ 100 │ 100 +│ 4.621 Gitem/s │ 3.184 Gitem/s │ 4.583 Gitem/s │ 4.546 Gitem/s │ │ +├─ mul_u64_nonnull 19.32 µs │ 23.55 µs │ 19.38 µs │ 19.47 µs │ 100 │ 100 +│ 1.695 Gitem/s │ 1.391 Gitem/s │ 1.689 Gitem/s │ 1.682 Gitem/s │ │ +╰─ sub_i64_constant 8.939 µs │ 30.07 µs │ 9.159 µs │ 9.386 µs │ 100 │ 100 + 3.665 Gitem/s │ 1.089 Gitem/s │ 3.577 Gitem/s │ 3.49 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md new file mode 100644 index 00000000000..b12390b54c2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-1.md @@ -0,0 +1,38 @@ + + + +# `stage0-base-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.319 µs │ 62.83 µs │ 8.449 µs │ 9.036 µs │ 100 │ 100 +│ 3.938 Gitem/s │ 521.5 Mitem/s │ 3.877 Gitem/s │ 3.626 Gitem/s │ │ +├─ add_i64_nonnull 9.139 µs │ 13.11 µs │ 9.205 µs │ 9.316 µs │ 100 │ 100 +│ 3.585 Gitem/s │ 2.497 Gitem/s │ 3.559 Gitem/s │ 3.517 Gitem/s │ │ +├─ div_i64_nonnull 44.78 µs │ 66.09 µs │ 44.85 µs │ 45.23 µs │ 100 │ 100 +│ 731.5 Mitem/s │ 495.8 Mitem/s │ 730.4 Mitem/s │ 724.4 Mitem/s │ │ +├─ mul_i8_nonnull 5.799 µs │ 17.72 µs │ 6.184 µs │ 6.39 µs │ 100 │ 100 +│ 5.649 Gitem/s │ 1.848 Gitem/s │ 5.298 Gitem/s │ 5.127 Gitem/s │ │ +├─ mul_i16_nonnull 4.009 µs │ 10.13 µs │ 4.099 µs │ 4.214 µs │ 100 │ 100 +│ 8.172 Gitem/s │ 3.234 Gitem/s │ 7.992 Gitem/s │ 7.774 Gitem/s │ │ +├─ mul_i32_constant 26.35 µs │ 30.02 µs │ 26.42 µs │ 26.53 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 1.091 Gitem/s │ 1.239 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 30.26 µs │ 26.41 µs │ 26.51 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.082 Gitem/s │ 1.24 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nullable 27.26 µs │ 48.92 µs │ 27.35 µs │ 27.7 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 669.8 Mitem/s │ 1.197 Gitem/s │ 1.182 Gitem/s │ │ +├─ mul_i64_nonnull 23.13 µs │ 28.03 µs │ 23.22 µs │ 23.37 µs │ 100 │ 100 +│ 1.416 Gitem/s │ 1.168 Gitem/s │ 1.41 Gitem/s │ 1.402 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 6.989 µs │ 3.319 µs │ 3.365 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 4.687 Gitem/s │ 9.87 Gitem/s │ 9.735 Gitem/s │ │ +├─ mul_u16_nonnull 2.539 µs │ 3.489 µs │ 2.599 µs │ 2.613 µs │ 100 │ 100 +│ 12.9 Gitem/s │ 9.389 Gitem/s │ 12.6 Gitem/s │ 12.53 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 12.13 µs │ 6.939 µs │ 7.009 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 2.699 Gitem/s │ 4.721 Gitem/s │ 4.674 Gitem/s │ │ +├─ mul_u64_nonnull 19.14 µs │ 23.82 µs │ 19.21 µs │ 19.32 µs │ 100 │ 100 +│ 1.711 Gitem/s │ 1.375 Gitem/s │ 1.704 Gitem/s │ 1.695 Gitem/s │ │ +╰─ sub_i64_constant 8.129 µs │ 12.18 µs │ 8.255 µs │ 8.34 µs │ 100 │ 100 + 4.03 Gitem/s │ 2.688 Gitem/s │ 3.969 Gitem/s │ 3.928 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md new file mode 100644 index 00000000000..c2254e0238f --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-base-2.md @@ -0,0 +1,38 @@ + + + +# `stage0-base-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.089 µs │ 63.25 µs │ 8.399 µs │ 8.966 µs │ 100 │ 100 +│ 4.05 Gitem/s │ 518 Mitem/s │ 3.901 Gitem/s │ 3.654 Gitem/s │ │ +├─ add_i64_nonnull 9.069 µs │ 13.16 µs │ 9.149 µs │ 9.232 µs │ 100 │ 100 +│ 3.612 Gitem/s │ 2.488 Gitem/s │ 3.581 Gitem/s │ 3.549 Gitem/s │ │ +├─ div_i64_nonnull 44.73 µs │ 51.02 µs │ 44.8 µs │ 45.03 µs │ 100 │ 100 +│ 732.4 Mitem/s │ 642.1 Mitem/s │ 731.2 Mitem/s │ 727.6 Mitem/s │ │ +├─ mul_i8_nonnull 5.979 µs │ 11.05 µs │ 6.199 µs │ 6.323 µs │ 100 │ 100 +│ 5.479 Gitem/s │ 2.962 Gitem/s │ 5.285 Gitem/s │ 5.182 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 8.709 µs │ 4.119 µs │ 4.205 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 3.762 Gitem/s │ 7.953 Gitem/s │ 7.791 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 29.65 µs │ 26.43 µs │ 26.51 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.104 Gitem/s │ 1.239 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nonnull 26.37 µs │ 36.95 µs │ 26.42 µs │ 26.61 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 886.8 Mitem/s │ 1.239 Gitem/s │ 1.231 Gitem/s │ │ +├─ mul_i32_nullable 27.3 µs │ 49.11 µs │ 27.4 µs │ 27.76 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 667.1 Mitem/s │ 1.195 Gitem/s │ 1.18 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 27.98 µs │ 23.2 µs │ 23.32 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 1.171 Gitem/s │ 1.411 Gitem/s │ 1.404 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 4.809 µs │ 3.329 µs │ 3.345 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 6.812 Gitem/s │ 9.84 Gitem/s │ 9.794 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.649 µs │ 2.599 µs │ 2.611 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 8.978 Gitem/s │ 12.6 Gitem/s │ 12.54 Gitem/s │ │ +├─ mul_u32_nonnull 6.889 µs │ 10.15 µs │ 6.949 µs │ 6.999 µs │ 100 │ 100 +│ 4.756 Gitem/s │ 3.228 Gitem/s │ 4.714 Gitem/s │ 4.681 Gitem/s │ │ +├─ mul_u64_nonnull 19.12 µs │ 24.11 µs │ 19.19 µs │ 19.3 µs │ 100 │ 100 +│ 1.712 Gitem/s │ 1.358 Gitem/s │ 1.706 Gitem/s │ 1.697 Gitem/s │ │ +╰─ sub_i64_constant 8.119 µs │ 12.58 µs │ 8.239 µs │ 8.323 µs │ 100 │ 100 + 4.035 Gitem/s │ 2.602 Gitem/s │ 3.976 Gitem/s │ 3.936 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md new file mode 100644 index 00000000000..c8539b0b1c6 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-1.md @@ -0,0 +1,38 @@ + + + +# `stage0-candidate-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.13 µs │ 93.77 µs │ 9.269 µs │ 10.17 µs │ 100 │ 100 +│ 3.588 Gitem/s │ 349.4 Mitem/s │ 3.534 Gitem/s │ 3.22 Gitem/s │ │ +├─ add_i64_nonnull 9.369 µs │ 12.45 µs │ 9.455 µs │ 9.51 µs │ 100 │ 100 +│ 3.497 Gitem/s │ 2.629 Gitem/s │ 3.465 Gitem/s │ 3.445 Gitem/s │ │ +├─ div_i64_nonnull 45.01 µs │ 54.4 µs │ 45.09 µs │ 45.42 µs │ 100 │ 100 +│ 727.8 Mitem/s │ 602.2 Mitem/s │ 726.5 Mitem/s │ 721.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 12.25 µs │ 4.694 µs │ 4.777 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 2.672 Gitem/s │ 6.979 Gitem/s │ 6.858 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 6.999 µs │ 4.269 µs │ 4.33 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 4.681 Gitem/s │ 7.674 Gitem/s │ 7.567 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 21.99 µs │ 18.88 µs │ 19 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 1.489 Gitem/s │ 1.734 Gitem/s │ 1.724 Gitem/s │ │ +├─ mul_i32_nonnull 28.23 µs │ 31.75 µs │ 28.39 µs │ 28.48 µs │ 100 │ 100 +│ 1.16 Gitem/s │ 1.031 Gitem/s │ 1.153 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 44.74 µs │ 29.18 µs │ 29.42 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 732.2 Mitem/s │ 1.122 Gitem/s │ 1.113 Gitem/s │ │ +├─ mul_i64_nonnull 29.7 µs │ 34.12 µs │ 30.02 µs │ 30.14 µs │ 100 │ 100 +│ 1.102 Gitem/s │ 960.1 Mitem/s │ 1.091 Gitem/s │ 1.087 Gitem/s │ │ +├─ mul_u8_nonnull 3.489 µs │ 7.519 µs │ 3.539 µs │ 3.602 µs │ 100 │ 100 +│ 9.389 Gitem/s │ 4.357 Gitem/s │ 9.257 Gitem/s │ 9.095 Gitem/s │ │ +├─ mul_u16_nonnull 2.349 µs │ 3.849 µs │ 2.429 µs │ 2.446 µs │ 100 │ 100 +│ 13.94 Gitem/s │ 8.511 Gitem/s │ 13.48 Gitem/s │ 13.39 Gitem/s │ │ +├─ mul_u32_nonnull 6.999 µs │ 9.889 µs │ 7.069 µs │ 7.11 µs │ 100 │ 100 +│ 4.681 Gitem/s │ 3.313 Gitem/s │ 4.634 Gitem/s │ 4.608 Gitem/s │ │ +├─ mul_u64_nonnull 30.36 µs │ 33.89 µs │ 30.43 µs │ 30.55 µs │ 100 │ 100 +│ 1.078 Gitem/s │ 966.6 Mitem/s │ 1.076 Gitem/s │ 1.072 Gitem/s │ │ +╰─ sub_i64_constant 8.979 µs │ 12.15 µs │ 9.099 µs │ 9.159 µs │ 100 │ 100 + 3.649 Gitem/s │ 2.696 Gitem/s │ 3.6 Gitem/s │ 3.577 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md new file mode 100644 index 00000000000..401b1ae9bc8 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage0-candidate-2.md @@ -0,0 +1,38 @@ + + + +# `stage0-candidate-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.159 µs │ 88.36 µs │ 9.29 µs │ 10.41 µs │ 100 │ 100 +│ 3.577 Gitem/s │ 370.8 Mitem/s │ 3.527 Gitem/s │ 3.147 Gitem/s │ │ +├─ add_i64_nonnull 9.389 µs │ 12.37 µs │ 9.49 µs │ 9.622 µs │ 100 │ 100 +│ 3.489 Gitem/s │ 2.646 Gitem/s │ 3.452 Gitem/s │ 3.405 Gitem/s │ │ +├─ div_i64_nonnull 45.1 µs │ 48.73 µs │ 45.16 µs │ 45.34 µs │ 100 │ 100 +│ 726.4 Mitem/s │ 672.3 Mitem/s │ 725.5 Mitem/s │ 722.6 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 7.529 µs │ 4.699 µs │ 4.751 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 4.351 Gitem/s │ 6.972 Gitem/s │ 6.897 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 6.379 µs │ 4.269 µs │ 4.295 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 5.136 Gitem/s │ 7.674 Gitem/s │ 7.629 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 24.05 µs │ 18.87 µs │ 19 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 1.361 Gitem/s │ 1.736 Gitem/s │ 1.724 Gitem/s │ │ +├─ mul_i32_nonnull 28.19 µs │ 31.61 µs │ 28.35 µs │ 28.45 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 1.036 Gitem/s │ 1.155 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29 µs │ 50.06 µs │ 29.15 µs │ 29.47 µs │ 100 │ 100 +│ 1.129 Gitem/s │ 654.5 Mitem/s │ 1.123 Gitem/s │ 1.111 Gitem/s │ │ +├─ mul_i64_nonnull 29.82 µs │ 33.7 µs │ 30.08 µs │ 30.21 µs │ 100 │ 100 +│ 1.098 Gitem/s │ 972 Mitem/s │ 1.089 Gitem/s │ 1.084 Gitem/s │ │ +├─ mul_u8_nonnull 3.469 µs │ 9.249 µs │ 3.529 µs │ 3.592 µs │ 100 │ 100 +│ 9.443 Gitem/s │ 3.542 Gitem/s │ 9.283 Gitem/s │ 9.119 Gitem/s │ │ +├─ mul_u16_nonnull 2.369 µs │ 3.699 µs │ 2.429 µs │ 2.448 µs │ 100 │ 100 +│ 13.82 Gitem/s │ 8.856 Gitem/s │ 13.48 Gitem/s │ 13.38 Gitem/s │ │ +├─ mul_u32_nonnull 6.989 µs │ 9.659 µs │ 7.059 µs │ 7.111 µs │ 100 │ 100 +│ 4.687 Gitem/s │ 3.392 Gitem/s │ 4.641 Gitem/s │ 4.607 Gitem/s │ │ +├─ mul_u64_nonnull 30.41 µs │ 33.95 µs │ 30.49 µs │ 30.63 µs │ 100 │ 100 +│ 1.077 Gitem/s │ 964.9 Mitem/s │ 1.074 Gitem/s │ 1.069 Gitem/s │ │ +╰─ sub_i64_constant 8.989 µs │ 11.76 µs │ 9.099 µs │ 9.169 µs │ 100 │ 100 + 3.645 Gitem/s │ 2.784 Gitem/s │ 3.6 Gitem/s │ 3.573 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md new file mode 100644 index 00000000000..4b6e17ff63e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-base-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.089 µs │ 783.5 µs │ 8.419 µs │ 16.21 µs │ 100 │ 100 +│ 4.05 Gitem/s │ 41.81 Mitem/s │ 3.891 Gitem/s │ 2.02 Gitem/s │ │ +├─ add_i64_nonnull 9.089 µs │ 32.99 µs │ 9.189 µs │ 9.53 µs │ 100 │ 100 +│ 3.604 Gitem/s │ 992.9 Mitem/s │ 3.565 Gitem/s │ 3.438 Gitem/s │ │ +├─ div_i64_nonnull 44.76 µs │ 76.23 µs │ 44.84 µs │ 45.51 µs │ 100 │ 100 +│ 731.9 Mitem/s │ 429.8 Mitem/s │ 730.6 Mitem/s │ 719.8 Mitem/s │ │ +├─ mul_i8_nonnull 5.929 µs │ 72.34 µs │ 6.239 µs │ 7.053 µs │ 100 │ 100 +│ 5.526 Gitem/s │ 452.9 Mitem/s │ 5.251 Gitem/s │ 4.645 Gitem/s │ │ +├─ mul_i16_nonnull 4.049 µs │ 61.69 µs │ 4.114 µs │ 4.692 µs │ 100 │ 100 +│ 8.091 Gitem/s │ 531.1 Mitem/s │ 7.963 Gitem/s │ 6.983 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 55.98 µs │ 26.43 µs │ 26.85 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 585.2 Mitem/s │ 1.239 Gitem/s │ 1.219 Gitem/s │ │ +├─ mul_i32_nonnull 26.38 µs │ 37.56 µs │ 26.42 µs │ 26.65 µs │ 100 │ 100 +│ 1.241 Gitem/s │ 872.3 Mitem/s │ 1.239 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.23 µs │ 340.3 µs │ 27.36 µs │ 30.62 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 96.26 Mitem/s │ 1.197 Gitem/s │ 1.07 Gitem/s │ │ +├─ mul_i64_nonnull 23.11 µs │ 44.46 µs │ 23.2 µs │ 23.5 µs │ 100 │ 100 +│ 1.417 Gitem/s │ 736.8 Mitem/s │ 1.411 Gitem/s │ 1.394 Gitem/s │ │ +├─ mul_u8_nonnull 3.269 µs │ 51.96 µs │ 3.329 µs │ 3.829 µs │ 100 │ 100 +│ 10.02 Gitem/s │ 630.6 Mitem/s │ 9.84 Gitem/s │ 8.556 Gitem/s │ │ +├─ mul_u16_nonnull 2.559 µs │ 30.97 µs │ 2.609 µs │ 2.898 µs │ 100 │ 100 +│ 12.8 Gitem/s │ 1.057 Gitem/s │ 12.55 Gitem/s │ 11.3 Gitem/s │ │ +├─ mul_u32_nonnull 6.88 µs │ 26.3 µs │ 6.959 µs │ 7.202 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 1.245 Gitem/s │ 4.708 Gitem/s │ 4.549 Gitem/s │ │ +├─ mul_u64_nonnull 19.11 µs │ 40.79 µs │ 19.18 µs │ 19.48 µs │ 100 │ 100 +│ 1.713 Gitem/s │ 803.3 Mitem/s │ 1.707 Gitem/s │ 1.681 Gitem/s │ │ +╰─ sub_i64_constant 8.109 µs │ 41.21 µs │ 8.219 µs │ 8.589 µs │ 100 │ 100 + 4.04 Gitem/s │ 794.9 Mitem/s │ 3.986 Gitem/s │ 3.814 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md new file mode 100644 index 00000000000..e540fafc12f --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-base-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-base-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.309 µs │ 51.73 µs │ 8.459 µs │ 8.93 µs │ 100 │ 100 +│ 3.943 Gitem/s │ 633.3 Mitem/s │ 3.873 Gitem/s │ 3.669 Gitem/s │ │ +├─ add_i64_nonnull 9.119 µs │ 19.9 µs │ 9.199 µs │ 9.345 µs │ 100 │ 100 +│ 3.593 Gitem/s │ 1.646 Gitem/s │ 3.561 Gitem/s │ 3.506 Gitem/s │ │ +├─ div_i64_nonnull 44.77 µs │ 49.58 µs │ 44.85 µs │ 45.06 µs │ 100 │ 100 +│ 731.7 Mitem/s │ 660.7 Mitem/s │ 730.5 Mitem/s │ 727.1 Mitem/s │ │ +├─ mul_i8_nonnull 5.779 µs │ 10.19 µs │ 6.15 µs │ 6.267 µs │ 100 │ 100 +│ 5.669 Gitem/s │ 3.212 Gitem/s │ 5.327 Gitem/s │ 5.228 Gitem/s │ │ +├─ mul_i16_nonnull 4.059 µs │ 8.189 µs │ 4.109 µs │ 4.198 µs │ 100 │ 100 +│ 8.071 Gitem/s │ 4.001 Gitem/s │ 7.973 Gitem/s │ 7.804 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 30.04 µs │ 26.44 µs │ 26.54 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.09 Gitem/s │ 1.238 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 29.99 µs │ 26.41 µs │ 26.54 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.092 Gitem/s │ 1.24 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nullable 27.23 µs │ 42.3 µs │ 27.35 µs │ 27.61 µs │ 100 │ 100 +│ 1.202 Gitem/s │ 774.4 Mitem/s │ 1.197 Gitem/s │ 1.186 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 27.63 µs │ 23.26 µs │ 23.41 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 1.185 Gitem/s │ 1.408 Gitem/s │ 1.399 Gitem/s │ │ +├─ mul_u8_nonnull 3.269 µs │ 4.809 µs │ 3.319 µs │ 3.339 µs │ 100 │ 100 +│ 10.02 Gitem/s │ 6.812 Gitem/s │ 9.87 Gitem/s │ 9.812 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.519 µs │ 2.609 µs │ 2.618 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 9.309 Gitem/s │ 12.55 Gitem/s │ 12.51 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 11.37 µs │ 6.95 µs │ 7.026 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 2.879 Gitem/s │ 4.714 Gitem/s │ 4.663 Gitem/s │ │ +├─ mul_u64_nonnull 19.16 µs │ 22.33 µs │ 19.21 µs │ 19.3 µs │ 100 │ 100 +│ 1.709 Gitem/s │ 1.466 Gitem/s │ 1.705 Gitem/s │ 1.697 Gitem/s │ │ +╰─ sub_i64_constant 8.139 µs │ 11.6 µs │ 8.259 µs │ 8.319 µs │ 100 │ 100 + 4.025 Gitem/s │ 2.822 Gitem/s │ 3.967 Gitem/s │ 3.938 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md new file mode 100644 index 00000000000..89efbeab66c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-candidate-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 1.038 ms │ 9.279 µs │ 19.64 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 31.54 Mitem/s │ 3.531 Gitem/s │ 1.667 Gitem/s │ │ +├─ add_i64_nonnull 9.399 µs │ 23.57 µs │ 9.459 µs │ 9.66 µs │ 100 │ 100 +│ 3.486 Gitem/s │ 1.389 Gitem/s │ 3.463 Gitem/s │ 3.391 Gitem/s │ │ +├─ div_i64_nonnull 45.03 µs │ 63.07 µs │ 45.1 µs │ 45.48 µs │ 100 │ 100 +│ 727.5 Mitem/s │ 519.4 Mitem/s │ 726.4 Mitem/s │ 720.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.629 µs │ 65.15 µs │ 4.689 µs │ 5.344 µs │ 100 │ 100 +│ 7.077 Gitem/s │ 502.8 Mitem/s │ 6.987 Gitem/s │ 6.131 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 71.32 µs │ 4.259 µs │ 4.934 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 459.3 Mitem/s │ 7.692 Gitem/s │ 6.64 Gitem/s │ │ +├─ mul_i32_constant 18.71 µs │ 73.84 µs │ 18.82 µs │ 19.43 µs │ 100 │ 100 +│ 1.75 Gitem/s │ 443.7 Mitem/s │ 1.74 Gitem/s │ 1.685 Gitem/s │ │ +├─ mul_i32_nonnull 28.22 µs │ 32.07 µs │ 28.34 µs │ 28.45 µs │ 100 │ 100 +│ 1.16 Gitem/s │ 1.021 Gitem/s │ 1.155 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 237.4 µs │ 29.16 µs │ 31.4 µs │ 100 │ 100 +│ 1.127 Gitem/s │ 138 Mitem/s │ 1.123 Gitem/s │ 1.043 Gitem/s │ │ +├─ mul_i64_nonnull 29.72 µs │ 54.86 µs │ 30.07 µs │ 30.53 µs │ 100 │ 100 +│ 1.102 Gitem/s │ 597.1 Mitem/s │ 1.089 Gitem/s │ 1.073 Gitem/s │ │ +├─ mul_u8_nonnull 3.469 µs │ 15.4 µs │ 3.529 µs │ 3.658 µs │ 100 │ 100 +│ 9.443 Gitem/s │ 2.126 Gitem/s │ 9.283 Gitem/s │ 8.956 Gitem/s │ │ +├─ mul_u16_nonnull 2.339 µs │ 13.45 µs │ 2.419 µs │ 2.574 µs │ 100 │ 100 +│ 14 Gitem/s │ 2.434 Gitem/s │ 13.54 Gitem/s │ 12.72 Gitem/s │ │ +├─ mul_u32_nonnull 6.969 µs │ 19.59 µs │ 7.049 µs │ 7.223 µs │ 100 │ 100 +│ 4.701 Gitem/s │ 1.671 Gitem/s │ 4.648 Gitem/s │ 4.536 Gitem/s │ │ +├─ mul_u64_nonnull 30.36 µs │ 42.47 µs │ 30.45 µs │ 30.69 µs │ 100 │ 100 +│ 1.079 Gitem/s │ 771.3 Mitem/s │ 1.075 Gitem/s │ 1.067 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 31.68 µs │ 9.119 µs │ 9.431 µs │ 100 │ 100 + 3.636 Gitem/s │ 1.034 Gitem/s │ 3.593 Gitem/s │ 3.474 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md new file mode 100644 index 00000000000..4bb5ec2261e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-candidate-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-candidate-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.149 µs │ 65.44 µs │ 9.309 µs │ 9.916 µs │ 100 │ 100 +│ 3.581 Gitem/s │ 500.6 Mitem/s │ 3.519 Gitem/s │ 3.304 Gitem/s │ │ +├─ add_i64_nonnull 9.429 µs │ 18.82 µs │ 9.529 µs │ 9.64 µs │ 100 │ 100 +│ 3.474 Gitem/s │ 1.74 Gitem/s │ 3.438 Gitem/s │ 3.399 Gitem/s │ │ +├─ div_i64_nonnull 45.09 µs │ 51.42 µs │ 45.16 µs │ 45.37 µs │ 100 │ 100 +│ 726.5 Mitem/s │ 637.2 Mitem/s │ 725.4 Mitem/s │ 722.1 Mitem/s │ │ +├─ mul_i8_nonnull 4.669 µs │ 7.159 µs │ 4.729 µs │ 4.781 µs │ 100 │ 100 +│ 7.017 Gitem/s │ 4.576 Gitem/s │ 6.928 Gitem/s │ 6.853 Gitem/s │ │ +├─ mul_i16_nonnull 4.249 µs │ 8.269 µs │ 4.319 µs │ 4.391 µs │ 100 │ 100 +│ 7.71 Gitem/s │ 3.962 Gitem/s │ 7.585 Gitem/s │ 7.461 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.35 µs │ 18.85 µs │ 18.93 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.465 Gitem/s │ 1.737 Gitem/s │ 1.73 Gitem/s │ │ +├─ mul_i32_nonnull 28.25 µs │ 32.31 µs │ 28.39 µs │ 28.46 µs │ 100 │ 100 +│ 1.159 Gitem/s │ 1.013 Gitem/s │ 1.153 Gitem/s │ 1.151 Gitem/s │ │ +├─ mul_i32_nullable 29.04 µs │ 38.59 µs │ 29.18 µs │ 29.37 µs │ 100 │ 100 +│ 1.127 Gitem/s │ 848.9 Mitem/s │ 1.122 Gitem/s │ 1.115 Gitem/s │ │ +├─ mul_i64_nonnull 29.84 µs │ 33.8 µs │ 30.16 µs │ 30.24 µs │ 100 │ 100 +│ 1.097 Gitem/s │ 969.1 Mitem/s │ 1.086 Gitem/s │ 1.083 Gitem/s │ │ +├─ mul_u8_nonnull 3.509 µs │ 6.339 µs │ 3.579 µs │ 3.604 µs │ 100 │ 100 +│ 9.336 Gitem/s │ 5.168 Gitem/s │ 9.153 Gitem/s │ 9.091 Gitem/s │ │ +├─ mul_u16_nonnull 2.389 µs │ 38.12 µs │ 2.474 µs │ 2.857 µs │ 100 │ 100 +│ 13.71 Gitem/s │ 859.3 Mitem/s │ 13.24 Gitem/s │ 11.46 Gitem/s │ │ +├─ mul_u32_nonnull 7.019 µs │ 8.19 µs │ 7.109 µs │ 7.121 µs │ 100 │ 100 +│ 4.667 Gitem/s │ 4 Gitem/s │ 4.608 Gitem/s │ 4.601 Gitem/s │ │ +├─ mul_u64_nonnull 30.4 µs │ 33.46 µs │ 30.49 µs │ 30.63 µs │ 100 │ 100 +│ 1.077 Gitem/s │ 979 Mitem/s │ 1.074 Gitem/s │ 1.069 Gitem/s │ │ +╰─ sub_i64_constant 9.009 µs │ 10.35 µs │ 9.119 µs │ 9.142 µs │ 100 │ 100 + 3.636 Gitem/s │ 3.163 Gitem/s │ 3.593 Gitem/s │ 3.584 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md new file mode 100644 index 00000000000..06e474ab7f0 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-1.md @@ -0,0 +1,38 @@ + + + +# `stage1-owned-1` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.869 µs │ 97.56 µs │ 9.224 µs │ 10.16 µs │ 100 │ 100 +│ 3.694 Gitem/s │ 335.8 Mitem/s │ 3.552 Gitem/s │ 3.225 Gitem/s │ │ +├─ add_i64_nonnull 9.299 µs │ 13.77 µs │ 9.389 µs │ 9.495 µs │ 100 │ 100 +│ 3.523 Gitem/s │ 2.377 Gitem/s │ 3.489 Gitem/s │ 3.45 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 54.06 µs │ 45.04 µs │ 45.34 µs │ 100 │ 100 +│ 728.8 Mitem/s │ 606.1 Mitem/s │ 727.5 Mitem/s │ 722.7 Mitem/s │ │ +├─ mul_i8_nonnull 4.559 µs │ 58.61 µs │ 4.619 µs │ 5.202 µs │ 100 │ 100 +│ 7.186 Gitem/s │ 558.9 Mitem/s │ 7.092 Gitem/s │ 6.298 Gitem/s │ │ +├─ mul_i16_nonnull 4.159 µs │ 5.809 µs │ 4.229 µs │ 4.244 µs │ 100 │ 100 +│ 7.877 Gitem/s │ 5.64 Gitem/s │ 7.746 Gitem/s │ 7.72 Gitem/s │ │ +├─ mul_i32_constant 32.23 µs │ 36.15 µs │ 32.36 µs │ 32.49 µs │ 100 │ 100 +│ 1.016 Gitem/s │ 906.2 Mitem/s │ 1.012 Gitem/s │ 1.008 Gitem/s │ │ +├─ mul_i32_nonnull 27.81 µs │ 43.9 µs │ 31.24 µs │ 31.22 µs │ 100 │ 100 +│ 1.177 Gitem/s │ 746.2 Mitem/s │ 1.048 Gitem/s │ 1.049 Gitem/s │ │ +├─ mul_i32_nullable 28.55 µs │ 50.24 µs │ 32.04 µs │ 31.62 µs │ 100 │ 100 +│ 1.147 Gitem/s │ 652.1 Mitem/s │ 1.022 Gitem/s │ 1.036 Gitem/s │ │ +├─ mul_i64_nonnull 25.31 µs │ 29.26 µs │ 25.65 µs │ 25.77 µs │ 100 │ 100 +│ 1.294 Gitem/s │ 1.119 Gitem/s │ 1.277 Gitem/s │ 1.271 Gitem/s │ │ +├─ mul_u8_nonnull 3.399 µs │ 55.89 µs │ 3.469 µs │ 3.999 µs │ 100 │ 100 +│ 9.638 Gitem/s │ 586.1 Mitem/s │ 9.443 Gitem/s │ 8.193 Gitem/s │ │ +├─ mul_u16_nonnull 2.289 µs │ 6.769 µs │ 2.369 µs │ 2.415 µs │ 100 │ 100 +│ 14.31 Gitem/s │ 4.84 Gitem/s │ 13.82 Gitem/s │ 13.56 Gitem/s │ │ +├─ mul_u32_nonnull 6.919 µs │ 9.879 µs │ 7.009 µs │ 7.059 µs │ 100 │ 100 +│ 4.735 Gitem/s │ 3.316 Gitem/s │ 4.674 Gitem/s │ 4.641 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 22.38 µs │ 19.41 µs │ 19.5 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.463 Gitem/s │ 1.687 Gitem/s │ 1.68 Gitem/s │ │ +╰─ sub_i64_constant 9.419 µs │ 12.37 µs │ 9.554 µs │ 9.607 µs │ 100 │ 100 + 3.478 Gitem/s │ 2.646 Gitem/s │ 3.429 Gitem/s │ 3.41 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md new file mode 100644 index 00000000000..a653d31d205 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage1-owned-2.md @@ -0,0 +1,38 @@ + + + +# `stage1-owned-2` + +```text +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.919 µs │ 99.93 µs │ 9.229 µs │ 10.19 µs │ 100 │ 100 +│ 3.673 Gitem/s │ 327.8 Mitem/s │ 3.55 Gitem/s │ 3.214 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 12.53 µs │ 9.339 µs │ 9.417 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 2.613 Gitem/s │ 3.508 Gitem/s │ 3.479 Gitem/s │ │ +├─ div_i64_nonnull 44.91 µs │ 54.36 µs │ 45.01 µs │ 45.32 µs │ 100 │ 100 +│ 729.4 Mitem/s │ 602.6 Mitem/s │ 727.8 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 4.579 µs │ 61.3 µs │ 4.649 µs │ 5.229 µs │ 100 │ 100 +│ 7.154 Gitem/s │ 534.5 Mitem/s │ 7.047 Gitem/s │ 6.265 Gitem/s │ │ +├─ mul_i16_nonnull 4.169 µs │ 7.419 µs │ 4.229 µs │ 4.276 µs │ 100 │ 100 +│ 7.858 Gitem/s │ 4.416 Gitem/s │ 7.746 Gitem/s │ 7.661 Gitem/s │ │ +├─ mul_i32_constant 32.27 µs │ 35.87 µs │ 32.38 µs │ 32.49 µs │ 100 │ 100 +│ 1.015 Gitem/s │ 913.5 Mitem/s │ 1.011 Gitem/s │ 1.008 Gitem/s │ │ +├─ mul_i32_nonnull 27.77 µs │ 32.39 µs │ 31.23 µs │ 30.31 µs │ 100 │ 100 +│ 1.179 Gitem/s │ 1.011 Gitem/s │ 1.048 Gitem/s │ 1.08 Gitem/s │ │ +├─ mul_i32_nullable 28.53 µs │ 49.46 µs │ 32.04 µs │ 31.34 µs │ 100 │ 100 +│ 1.148 Gitem/s │ 662.3 Mitem/s │ 1.022 Gitem/s │ 1.045 Gitem/s │ │ +├─ mul_i64_nonnull 25.25 µs │ 29.06 µs │ 25.59 µs │ 25.68 µs │ 100 │ 100 +│ 1.297 Gitem/s │ 1.127 Gitem/s │ 1.28 Gitem/s │ 1.275 Gitem/s │ │ +├─ mul_u8_nonnull 3.429 µs │ 60.44 µs │ 3.479 µs │ 4.062 µs │ 100 │ 100 +│ 9.553 Gitem/s │ 542 Mitem/s │ 9.416 Gitem/s │ 8.065 Gitem/s │ │ +├─ mul_u16_nonnull 2.319 µs │ 7.879 µs │ 2.389 µs │ 2.446 µs │ 100 │ 100 +│ 14.12 Gitem/s │ 4.158 Gitem/s │ 13.71 Gitem/s │ 13.39 Gitem/s │ │ +├─ mul_u32_nonnull 6.949 µs │ 10.35 µs │ 7.019 µs │ 7.069 µs │ 100 │ 100 +│ 4.714 Gitem/s │ 3.163 Gitem/s │ 4.667 Gitem/s │ 4.635 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 22.75 µs │ 19.41 µs │ 19.49 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.44 Gitem/s │ 1.687 Gitem/s │ 1.68 Gitem/s │ │ +╰─ sub_i64_constant 9.439 µs │ 12.1 µs │ 9.569 µs │ 9.636 µs │ 100 │ 100 + 3.471 Gitem/s │ 2.705 Gitem/s │ 3.424 Gitem/s │ 3.4 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md new file mode 100644 index 00000000000..3053244a92a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-base-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.099 µs │ 766.1 µs │ 8.419 µs │ 16.03 µs │ 100 │ 100 +│ 4.045 Gitem/s │ 42.76 Mitem/s │ 3.891 Gitem/s │ 2.043 Gitem/s │ │ +├─ add_i64_nonnull 9.099 µs │ 30.45 µs │ 9.179 µs │ 9.474 µs │ 100 │ 100 +│ 3.6 Gitem/s │ 1.075 Gitem/s │ 3.569 Gitem/s │ 3.458 Gitem/s │ │ +├─ div_i64_nonnull 44.76 µs │ 75.04 µs │ 44.84 µs │ 45.32 µs │ 100 │ 100 +│ 731.9 Mitem/s │ 436.6 Mitem/s │ 730.6 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 5.809 µs │ 69.77 µs │ 6.209 µs │ 6.952 µs │ 100 │ 100 +│ 5.64 Gitem/s │ 469.5 Mitem/s │ 5.276 Gitem/s │ 4.713 Gitem/s │ │ +├─ mul_i16_nonnull 4.029 µs │ 66.19 µs │ 4.099 µs │ 4.725 µs │ 100 │ 100 +│ 8.131 Gitem/s │ 494.9 Mitem/s │ 7.992 Gitem/s │ 6.934 Gitem/s │ │ +├─ mul_i32_constant 26.36 µs │ 55.21 µs │ 26.42 µs │ 26.88 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 593.4 Mitem/s │ 1.239 Gitem/s │ 1.218 Gitem/s │ │ +├─ mul_i32_nonnull 26.34 µs │ 39.19 µs │ 26.39 µs │ 26.64 µs │ 100 │ 100 +│ 1.243 Gitem/s │ 835.9 Mitem/s │ 1.241 Gitem/s │ 1.229 Gitem/s │ │ +├─ mul_i32_nullable 27.21 µs │ 333.7 µs │ 27.37 µs │ 30.56 µs │ 100 │ 100 +│ 1.203 Gitem/s │ 98.17 Mitem/s │ 1.196 Gitem/s │ 1.072 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 43.91 µs │ 23.22 µs │ 23.6 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 746 Mitem/s │ 1.41 Gitem/s │ 1.388 Gitem/s │ │ +├─ mul_u8_nonnull 3.259 µs │ 52.38 µs │ 3.329 µs │ 3.817 µs │ 100 │ 100 +│ 10.05 Gitem/s │ 625.4 Mitem/s │ 9.84 Gitem/s │ 8.582 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 30.18 µs │ 2.609 µs │ 2.929 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 1.085 Gitem/s │ 12.55 Gitem/s │ 11.18 Gitem/s │ │ +├─ mul_u32_nonnull 6.869 µs │ 27.32 µs │ 6.939 µs │ 7.147 µs │ 100 │ 100 +│ 4.769 Gitem/s │ 1.198 Gitem/s │ 4.721 Gitem/s │ 4.584 Gitem/s │ │ +├─ mul_u64_nonnull 19.14 µs │ 41.82 µs │ 19.22 µs │ 19.57 µs │ 100 │ 100 +│ 1.711 Gitem/s │ 783.3 Mitem/s │ 1.704 Gitem/s │ 1.674 Gitem/s │ │ +╰─ sub_i64_constant 8.159 µs │ 40.98 µs │ 8.249 µs │ 8.63 µs │ 100 │ 100 + 4.015 Gitem/s │ 799.4 Mitem/s │ 3.971 Gitem/s │ 3.796 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md new file mode 100644 index 00000000000..793ece2b6f1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-base-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-base-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.299 µs │ 47.08 µs │ 8.429 µs │ 8.916 µs │ 100 │ 100 +│ 3.948 Gitem/s │ 695.8 Mitem/s │ 3.887 Gitem/s │ 3.675 Gitem/s │ │ +├─ add_i64_nonnull 9.139 µs │ 12.35 µs │ 9.209 µs │ 9.286 µs │ 100 │ 100 +│ 3.585 Gitem/s │ 2.651 Gitem/s │ 3.557 Gitem/s │ 3.528 Gitem/s │ │ +├─ div_i64_nonnull 44.8 µs │ 49.64 µs │ 44.87 µs │ 45.09 µs │ 100 │ 100 +│ 731.2 Mitem/s │ 659.9 Mitem/s │ 730.1 Mitem/s │ 726.6 Mitem/s │ │ +├─ mul_i8_nonnull 5.869 µs │ 9.279 µs │ 6.174 µs │ 6.295 µs │ 100 │ 100 +│ 5.582 Gitem/s │ 3.531 Gitem/s │ 5.306 Gitem/s │ 5.204 Gitem/s │ │ +├─ mul_i16_nonnull 4.039 µs │ 7.909 µs │ 4.104 µs │ 4.153 µs │ 100 │ 100 +│ 8.111 Gitem/s │ 4.142 Gitem/s │ 7.982 Gitem/s │ 7.888 Gitem/s │ │ +├─ mul_i32_constant 26.37 µs │ 29.56 µs │ 26.43 µs │ 26.52 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.108 Gitem/s │ 1.239 Gitem/s │ 1.235 Gitem/s │ │ +├─ mul_i32_nonnull 26.36 µs │ 30.91 µs │ 26.41 µs │ 26.53 µs │ 100 │ 100 +│ 1.242 Gitem/s │ 1.059 Gitem/s │ 1.24 Gitem/s │ 1.234 Gitem/s │ │ +├─ mul_i32_nullable 27.27 µs │ 42.76 µs │ 27.38 µs │ 27.63 µs │ 100 │ 100 +│ 1.201 Gitem/s │ 766.1 Mitem/s │ 1.196 Gitem/s │ 1.185 Gitem/s │ │ +├─ mul_i64_nonnull 23.14 µs │ 26.52 µs │ 23.24 µs │ 23.33 µs │ 100 │ 100 +│ 1.415 Gitem/s │ 1.235 Gitem/s │ 1.409 Gitem/s │ 1.404 Gitem/s │ │ +├─ mul_u8_nonnull 3.279 µs │ 6.329 µs │ 3.329 µs │ 3.378 µs │ 100 │ 100 +│ 9.99 Gitem/s │ 5.176 Gitem/s │ 9.84 Gitem/s │ 9.698 Gitem/s │ │ +├─ mul_u16_nonnull 2.549 µs │ 3.489 µs │ 2.599 µs │ 2.615 µs │ 100 │ 100 +│ 12.85 Gitem/s │ 9.389 Gitem/s │ 12.6 Gitem/s │ 12.52 Gitem/s │ │ +├─ mul_u32_nonnull 6.879 µs │ 9.609 µs │ 6.939 µs │ 7.003 µs │ 100 │ 100 +│ 4.762 Gitem/s │ 3.409 Gitem/s │ 4.721 Gitem/s │ 4.678 Gitem/s │ │ +├─ mul_u64_nonnull 19.15 µs │ 22.82 µs │ 19.21 µs │ 19.33 µs │ 100 │ 100 +│ 1.71 Gitem/s │ 1.435 Gitem/s │ 1.704 Gitem/s │ 1.694 Gitem/s │ │ +╰─ sub_i64_constant 8.119 µs │ 11.08 µs │ 8.249 µs │ 8.293 µs │ 100 │ 100 + 4.035 Gitem/s │ 2.954 Gitem/s │ 3.971 Gitem/s │ 3.951 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md new file mode 100644 index 00000000000..6364ccde44c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-candidate-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.099 µs │ 1.015 ms │ 9.269 µs │ 19.41 µs │ 100 │ 100 +│ 3.6 Gitem/s │ 32.26 Mitem/s │ 3.534 Gitem/s │ 1.687 Gitem/s │ │ +├─ add_i64_nonnull 9.319 µs │ 12.12 µs │ 9.429 µs │ 9.48 µs │ 100 │ 100 +│ 3.515 Gitem/s │ 2.701 Gitem/s │ 3.474 Gitem/s │ 3.456 Gitem/s │ │ +├─ div_i64_nonnull 44.99 µs │ 61.85 µs │ 45.07 µs │ 45.6 µs │ 100 │ 100 +│ 728.1 Mitem/s │ 529.7 Mitem/s │ 726.8 Mitem/s │ 718.4 Mitem/s │ │ +├─ mul_i8_nonnull 4.639 µs │ 61.66 µs │ 4.689 µs │ 5.268 µs │ 100 │ 100 +│ 7.062 Gitem/s │ 531.3 Mitem/s │ 6.987 Gitem/s │ 6.219 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 66.06 µs │ 4.279 µs │ 4.931 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 495.9 Mitem/s │ 7.656 Gitem/s │ 6.644 Gitem/s │ │ +├─ mul_i32_constant 18.77 µs │ 71.76 µs │ 18.88 µs │ 19.64 µs │ 100 │ 100 +│ 1.744 Gitem/s │ 456.5 Mitem/s │ 1.734 Gitem/s │ 1.667 Gitem/s │ │ +├─ mul_i32_nonnull 28.21 µs │ 33.28 µs │ 28.34 µs │ 28.48 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 984.3 Mitem/s │ 1.155 Gitem/s │ 1.15 Gitem/s │ │ +├─ mul_i32_nullable 29.02 µs │ 233.9 µs │ 29.2 µs │ 31.37 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 140 Mitem/s │ 1.121 Gitem/s │ 1.044 Gitem/s │ │ +├─ mul_i64_nonnull 29.73 µs │ 52.85 µs │ 30.02 µs │ 30.37 µs │ 100 │ 100 +│ 1.101 Gitem/s │ 619.9 Mitem/s │ 1.091 Gitem/s │ 1.078 Gitem/s │ │ +├─ mul_u8_nonnull 3.449 µs │ 14.5 µs │ 3.529 µs │ 3.679 µs │ 100 │ 100 +│ 9.498 Gitem/s │ 2.258 Gitem/s │ 9.283 Gitem/s │ 8.906 Gitem/s │ │ +├─ mul_u16_nonnull 2.349 µs │ 13.44 µs │ 2.419 µs │ 2.529 µs │ 100 │ 100 +│ 13.94 Gitem/s │ 2.436 Gitem/s │ 13.54 Gitem/s │ 12.95 Gitem/s │ │ +├─ mul_u32_nonnull 6.979 µs │ 18.54 µs │ 7.059 µs │ 7.228 µs │ 100 │ 100 +│ 4.694 Gitem/s │ 1.766 Gitem/s │ 4.641 Gitem/s │ 4.532 Gitem/s │ │ +├─ mul_u64_nonnull 30.31 µs │ 42.57 µs │ 30.41 µs │ 30.68 µs │ 100 │ 100 +│ 1.08 Gitem/s │ 769.5 Mitem/s │ 1.077 Gitem/s │ 1.067 Gitem/s │ │ +╰─ sub_i64_constant 8.979 µs │ 32.68 µs │ 9.064 µs │ 9.358 µs │ 100 │ 100 + 3.649 Gitem/s │ 1.002 Gitem/s │ 3.614 Gitem/s │ 3.501 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md new file mode 100644 index 00000000000..6093fe215ee --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-candidate-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-candidate-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.159 µs │ 57.71 µs │ 9.269 µs │ 9.826 µs │ 100 │ 100 +│ 3.577 Gitem/s │ 567.7 Mitem/s │ 3.534 Gitem/s │ 3.334 Gitem/s │ │ +├─ add_i64_nonnull 9.359 µs │ 18.58 µs │ 9.454 µs │ 9.765 µs │ 100 │ 100 +│ 3.5 Gitem/s │ 1.762 Gitem/s │ 3.465 Gitem/s │ 3.355 Gitem/s │ │ +├─ div_i64_nonnull 45.03 µs │ 49.11 µs │ 45.12 µs │ 45.32 µs │ 100 │ 100 +│ 727.5 Mitem/s │ 667.1 Mitem/s │ 726 Mitem/s │ 722.9 Mitem/s │ │ +├─ mul_i8_nonnull 4.649 µs │ 8.699 µs │ 4.729 µs │ 4.798 µs │ 100 │ 100 +│ 7.047 Gitem/s │ 3.766 Gitem/s │ 6.928 Gitem/s │ 6.828 Gitem/s │ │ +├─ mul_i16_nonnull 4.219 µs │ 7.469 µs │ 4.309 µs │ 4.374 µs │ 100 │ 100 +│ 7.765 Gitem/s │ 4.386 Gitem/s │ 7.603 Gitem/s │ 7.491 Gitem/s │ │ +├─ mul_i32_constant 18.75 µs │ 22.67 µs │ 18.88 µs │ 18.95 µs │ 100 │ 100 +│ 1.746 Gitem/s │ 1.444 Gitem/s │ 1.734 Gitem/s │ 1.728 Gitem/s │ │ +├─ mul_i32_nonnull 28.2 µs │ 40.32 µs │ 28.36 µs │ 28.69 µs │ 100 │ 100 +│ 1.161 Gitem/s │ 812.5 Mitem/s │ 1.155 Gitem/s │ 1.141 Gitem/s │ │ +├─ mul_i32_nullable 29.02 µs │ 40.92 µs │ 29.17 µs │ 29.4 µs │ 100 │ 100 +│ 1.128 Gitem/s │ 800.5 Mitem/s │ 1.123 Gitem/s │ 1.114 Gitem/s │ │ +├─ mul_i64_nonnull 29.63 µs │ 33.89 µs │ 30.1 µs │ 30.22 µs │ 100 │ 100 +│ 1.105 Gitem/s │ 966.6 Mitem/s │ 1.088 Gitem/s │ 1.084 Gitem/s │ │ +├─ mul_u8_nonnull 3.489 µs │ 4.839 µs │ 3.559 µs │ 3.576 µs │ 100 │ 100 +│ 9.389 Gitem/s │ 6.77 Gitem/s │ 9.205 Gitem/s │ 9.163 Gitem/s │ │ +├─ mul_u16_nonnull 2.379 µs │ 5.529 µs │ 2.439 µs │ 2.489 µs │ 100 │ 100 +│ 13.76 Gitem/s │ 5.925 Gitem/s │ 13.43 Gitem/s │ 13.16 Gitem/s │ │ +├─ mul_u32_nonnull 6.989 µs │ 8.019 µs │ 7.079 µs │ 7.089 µs │ 100 │ 100 +│ 4.687 Gitem/s │ 4.085 Gitem/s │ 4.628 Gitem/s │ 4.621 Gitem/s │ │ +├─ mul_u64_nonnull 30.35 µs │ 34.77 µs │ 30.43 µs │ 30.57 µs │ 100 │ 100 +│ 1.079 Gitem/s │ 942.1 Mitem/s │ 1.076 Gitem/s │ 1.071 Gitem/s │ │ +╰─ sub_i64_constant 8.949 µs │ 10.5 µs │ 9.089 µs │ 9.109 µs │ 100 │ 100 + 3.661 Gitem/s │ 3.117 Gitem/s │ 3.604 Gitem/s │ 3.597 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md new file mode 100644 index 00000000000..440304b08fe --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-1.md @@ -0,0 +1,39 @@ + + + +# `stage2-indexed-1` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 8.889 µs │ 89.9 µs │ 9.244 µs │ 10.11 µs │ 100 │ 100 +│ 3.686 Gitem/s │ 364.4 Mitem/s │ 3.544 Gitem/s │ 3.24 Gitem/s │ │ +├─ add_i64_nonnull 9.279 µs │ 18.28 µs │ 9.399 µs │ 9.605 µs │ 100 │ 100 +│ 3.531 Gitem/s │ 1.791 Gitem/s │ 3.486 Gitem/s │ 3.411 Gitem/s │ │ +├─ div_i64_nonnull 44.92 µs │ 52.66 µs │ 45.07 µs │ 45.39 µs │ 100 │ 100 +│ 729.3 Mitem/s │ 622.1 Mitem/s │ 726.8 Mitem/s │ 721.9 Mitem/s │ │ +├─ mul_i8_nonnull 6.069 µs │ 69.59 µs │ 6.339 µs │ 7.085 µs │ 100 │ 100 +│ 5.398 Gitem/s │ 470.8 Mitem/s │ 5.168 Gitem/s │ 4.624 Gitem/s │ │ +├─ mul_i16_nonnull 4.209 µs │ 5.439 µs │ 4.259 µs │ 4.274 µs │ 100 │ 100 +│ 7.783 Gitem/s │ 6.023 Gitem/s │ 7.692 Gitem/s │ 7.665 Gitem/s │ │ +├─ mul_i32_constant 32.23 µs │ 36.63 µs │ 32.38 µs │ 32.54 µs │ 100 │ 100 +│ 1.016 Gitem/s │ 894.3 Mitem/s │ 1.011 Gitem/s │ 1.006 Gitem/s │ │ +├─ mul_i32_nonnull 26.52 µs │ 31.34 µs │ 26.58 µs │ 26.69 µs │ 100 │ 100 +│ 1.235 Gitem/s │ 1.045 Gitem/s │ 1.232 Gitem/s │ 1.227 Gitem/s │ │ +├─ mul_i32_nullable 27.31 µs │ 47.02 µs │ 27.41 µs │ 27.83 µs │ 100 │ 100 +│ 1.199 Gitem/s │ 696.7 Mitem/s │ 1.195 Gitem/s │ 1.177 Gitem/s │ │ +├─ mul_i64_nonnull 23.33 µs │ 32.13 µs │ 23.43 µs │ 23.74 µs │ 100 │ 100 +│ 1.403 Gitem/s │ 1.019 Gitem/s │ 1.397 Gitem/s │ 1.379 Gitem/s │ │ +├─ mul_u8_nonnull 3.439 µs │ 61.51 µs │ 3.509 µs │ 4.121 µs │ 100 │ 100 +│ 9.526 Gitem/s │ 532.6 Mitem/s │ 9.336 Gitem/s │ 7.951 Gitem/s │ │ +├─ mul_u16_nonnull 2.699 µs │ 6.979 µs │ 2.769 µs │ 2.813 µs │ 100 │ 100 +│ 12.13 Gitem/s │ 4.694 Gitem/s │ 11.83 Gitem/s │ 11.64 Gitem/s │ │ +├─ mul_u32_nonnull 7.029 µs │ 9.939 µs │ 7.109 µs │ 7.16 µs │ 100 │ 100 +│ 4.661 Gitem/s │ 3.296 Gitem/s │ 4.608 Gitem/s │ 4.576 Gitem/s │ │ +├─ mul_u64_nonnull 19.35 µs │ 22.92 µs │ 19.41 µs │ 19.51 µs │ 100 │ 100 +│ 1.692 Gitem/s │ 1.429 Gitem/s │ 1.687 Gitem/s │ 1.679 Gitem/s │ │ +╰─ sub_i64_constant 9.499 µs │ 12.48 µs │ 9.609 µs │ 9.668 µs │ 100 │ 100 + 3.449 Gitem/s │ 2.623 Gitem/s │ 3.409 Gitem/s │ 3.389 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md new file mode 100644 index 00000000000..aa7ed2c846a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/benchmarks/stage2-indexed-2.md @@ -0,0 +1,39 @@ + + + +# `stage2-indexed-2` + +```text +Timer precision: 10 ns +binary_ops fastest │ slowest │ median │ mean │ samples │ iters +├─ add_i64_constant 9.009 µs │ 93.1 µs │ 9.259 µs │ 10.14 µs │ 100 │ 100 +│ 3.636 Gitem/s │ 351.9 Mitem/s │ 3.538 Gitem/s │ 3.23 Gitem/s │ │ +├─ add_i64_nonnull 9.309 µs │ 12.56 µs │ 9.379 µs │ 9.426 µs │ 100 │ 100 +│ 3.519 Gitem/s │ 2.606 Gitem/s │ 3.493 Gitem/s │ 3.476 Gitem/s │ │ +├─ div_i64_nonnull 44.96 µs │ 48.24 µs │ 45.03 µs │ 45.22 µs │ 100 │ 100 +│ 728.6 Mitem/s │ 679.1 Mitem/s │ 727.5 Mitem/s │ 724.5 Mitem/s │ │ +├─ mul_i8_nonnull 5.969 µs │ 50.06 µs │ 6.359 µs │ 6.902 µs │ 100 │ 100 +│ 5.488 Gitem/s │ 654.4 Mitem/s │ 5.152 Gitem/s │ 4.747 Gitem/s │ │ +├─ mul_i16_nonnull 4.199 µs │ 15.43 µs │ 4.284 µs │ 4.418 µs │ 100 │ 100 +│ 7.802 Gitem/s │ 2.122 Gitem/s │ 7.647 Gitem/s │ 7.415 Gitem/s │ │ +├─ mul_i32_constant 32.25 µs │ 35.51 µs │ 32.39 µs │ 32.51 µs │ 100 │ 100 +│ 1.015 Gitem/s │ 922.5 Mitem/s │ 1.011 Gitem/s │ 1.007 Gitem/s │ │ +├─ mul_i32_nonnull 26.54 µs │ 29.82 µs │ 26.6 µs │ 26.7 µs │ 100 │ 100 +│ 1.234 Gitem/s │ 1.098 Gitem/s │ 1.231 Gitem/s │ 1.226 Gitem/s │ │ +├─ mul_i32_nullable 27.32 µs │ 40.06 µs │ 27.43 µs │ 27.68 µs │ 100 │ 100 +│ 1.198 Gitem/s │ 817.7 Mitem/s │ 1.194 Gitem/s │ 1.183 Gitem/s │ │ +├─ mul_i64_nonnull 23.33 µs │ 26.7 µs │ 23.44 µs │ 23.53 µs │ 100 │ 100 +│ 1.403 Gitem/s │ 1.226 Gitem/s │ 1.397 Gitem/s │ 1.392 Gitem/s │ │ +├─ mul_u8_nonnull 3.459 µs │ 61.29 µs │ 3.519 µs │ 4.131 µs │ 100 │ 100 +│ 9.471 Gitem/s │ 534.5 Mitem/s │ 9.309 Gitem/s │ 7.931 Gitem/s │ │ +├─ mul_u16_nonnull 2.709 µs │ 6.939 µs │ 2.789 µs │ 2.828 µs │ 100 │ 100 +│ 12.09 Gitem/s │ 4.721 Gitem/s │ 11.74 Gitem/s │ 11.58 Gitem/s │ │ +├─ mul_u32_nonnull 7.039 µs │ 10.56 µs │ 7.119 µs │ 7.18 µs │ 100 │ 100 +│ 4.654 Gitem/s │ 3.1 Gitem/s │ 4.602 Gitem/s │ 4.563 Gitem/s │ │ +├─ mul_u64_nonnull 19.34 µs │ 23.2 µs │ 19.42 µs │ 19.48 µs │ 100 │ 100 +│ 1.693 Gitem/s │ 1.411 Gitem/s │ 1.686 Gitem/s │ 1.681 Gitem/s │ │ +╰─ sub_i64_constant 9.509 µs │ 12.69 µs │ 9.619 µs │ 9.668 µs │ 100 │ 100 + 3.445 Gitem/s │ 2.58 Gitem/s │ 3.406 Gitem/s │ 3.389 Gitem/s │ │ + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md b/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md new file mode 100644 index 00000000000..1296a6b3bb1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/base-codegen-summary.md @@ -0,0 +1,139 @@ + + + +# Merge-base production numeric multiply code generation + +Revision: `19f771f2a426103aa7d1bf7153a258bb1bab1e19` + +Command: + +```text +CARGO_TARGET_DIR=/tmp/rowfn-x86.ccCdz5/target-base-codegen \ + cargo rustc -p vortex-array --lib --profile bench -- \ + --emit=llvm-ir,asm -C codegen-units=1 +``` + +Artifacts: + +```text +/tmp/rowfn-x86.ccCdz5/target-base-codegen/release/deps/vortex_array-4e5fe3dd7af89793.ll +/tmp/rowfn-x86.ccCdz5/target-base-codegen/release/deps/vortex_array-4e5fe3dd7af89793.s +``` + +## Production symbols + +```text +i64 execute_checked_typed: 648ec4b22808a2d4 +i64 checked_op_lanes (varying x varying): df33f84e66e75a91 +u64 execute_checked_typed: 8da1eac40a9b0934 +u64 checked_op_lanes (varying x varying): 84edf83ddcd3fe05 +``` + +## i64 hot loop + +Assembly source begins at line 6,698,561 in the `.s` artifact. The loop is +`.LBB6227_8`: + +```asm +movq (%rdi,%rsi,8), %rax +imulq (%r15,%rsi,8) +movq %rax, (%r13,%rsi,8) +incq %rsi +sarq $63, %rax +xorq %rdx, %rax +orq %rax, %rcx +cmpq %rsi, %rbx +jne .LBB6227_8 +``` + +This is one lane per backedge. The one-operand `imulq` produces the signed +128-bit product in `RDX:RAX`; the low half is stored and the high half is +compared with the low-half sign extension through `sarq`/`xorq`. Failure stays +in register `%rcx`. There is no `vector.body`, unroll, or separate remainder. + +## u64 hot loop + +Assembly source begins at line 6,646,461 in the `.s` artifact. The loop is +`.LBB6175_10`: + +```asm +movq (%rbx,%rdi,8), %rax +mulq (%r11,%rdi,8) +movq %rdx, %rsi +movq %rax, -8(%r9,%rdi,8) +movq 8(%rbx,%rdi,8), %rax +mulq 8(%r11,%rdi,8) +orq %rcx, %rsi +movq %rax, (%r9,%rdi,8) +addq $2, %rdi +movq %rdx, %rcx +orq %rsi, %rcx +cmpq %r10, %rdi +jne .LBB6175_10 +``` + +This is scalar unsigned high-half multiplication unrolled by two, followed by +a one-lane remainder when the row count is odd. The two loads, multiplies, and +stores are independent except for the register OR reduction. There is no +`vector.body` in this fast value loop. + +## IR facts + +The all-varying functions are internal and take the source structure through a +`noalias readonly` pointer and return storage through a `noalias writeonly` +pointer. The allocated output stores carry a distinct `!alias.scope` and +`!noalias`; both input loads carry input-side `!noalias`. The second input +length check has become `llvm.assume`, so no panic branch remains in either hot +loop. A slice/assert failure edge exists before the loop at the output-length +validation boundary. + +The u64 IR loop is unrolled by two and reduces two i128 high halves through +scalar `or i64`; it has a one-lane epilogue. The i64 IR loop is scalar and uses +an i128 signed multiply, truncation, arithmetic sign extraction, XOR, and a +loop-carried register OR. Neither fast loop contains a call. + +Both monomorphs have this parameter-level ownership shape (metadata IDs differ +between them): + +```llvm +define internal fastcc void @checked_op_lanes( + ptr noalias writable writeonly %output, + ptr noalias readonly %source, + i64 %valid_rows_tag, + ptr readonly %valid_rows_data) +``` + +The relevant u64 body is structurally: + +```llvm +%failed = phi i64 [ 0, %preheader ], [ %failed_2, %loop ] +%lhs_0 = load i64, ptr %lhs_ptr_0, !noalias !input_scope +%rhs_0 = load i64, ptr %rhs_ptr_0, !noalias !input_scope +%low_0 = mul i64 %rhs_0, %lhs_0 +%wide_0 = mul nuw i128 (zext i64 %rhs_0), (zext i64 %lhs_0) +%high_0 = trunc i128 (lshr i128 %wide_0, 64) to i64 +%failed_1 = or i64 %failed, %high_0 +store i64 %low_0, ptr %output_0, !alias.scope !output_scope, !noalias !output_noalias + +%lhs_1 = load i64, ptr %lhs_ptr_1, !noalias !input_scope +%rhs_1 = load i64, ptr %rhs_ptr_1, !noalias !input_scope +%low_1 = mul i64 %rhs_1, %lhs_1 +%wide_1 = mul nuw i128 (zext i64 %rhs_1), (zext i64 %lhs_1) +%high_1 = trunc i128 (lshr i128 %wide_1, 64) to i64 +%failed_2 = or i64 %failed_1, %high_1 +store i64 %low_1, ptr %output_1, !alias.scope !output_scope, !noalias !output_noalias +``` + +The relevant i64 body is structurally: + +```llvm +%failed = phi i64 [ 0, %preheader ], [ %failed_next, %loop ] +%lhs = load i64, ptr %lhs_ptr, !noalias !input_scope +%rhs = load i64, ptr %rhs_ptr, !noalias !input_scope +%wide = mul nsw i128 (sext i64 %rhs), (sext i64 %lhs) +%low = trunc i128 %wide to i64 +%high = trunc i128 (lshr i128 %wide, 64) to i64 +%discarded_mismatch = xor i64 (ashr i64 %low, 63), %high +%failed_next = or i64 %discarded_mismatch, %failed +store i64 %low, ptr %output, !alias.scope !output_scope, !noalias !output_noalias +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md new file mode 100644 index 00000000000..b93c15fd04e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-ll.md @@ -0,0 +1,84 @@ + + + +# `candidate-i64-mul-dense.ll` + +```ll + %_16456.i = phi i64 [ 1, %bb28.lr.ph.i ], [ %_164.i, %bb28.i ] + %iter.sroa.0.055.i = phi i64 [ 0, %bb28.lr.ph.i ], [ %_16456.i, %bb28.i ] + %accumulated.sroa.0.054.i = phi i64 [ 0, %bb28.lr.ph.i ], [ %77, %bb28.i ] + #dbg_value(i64 %iter.sroa.0.055.i, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + #dbg_value(i64 %accumulated.sroa.0.054.i, !561355, !DIExpression(), !561631) + #dbg_value(i64 %iter.sroa.0.055.i, !561378, !DIExpression(), !562028) + #dbg_value(ptr undef, !558643, !DIExpression(), !561458) + #dbg_value(i64 %iter.sroa.0.055.i, !558649, !DIExpression(), !561458) + #dbg_value(ptr poison, !559346, !DIExpression(), !562029) + #dbg_value(i64 %iter.sroa.0.055.i, !559351, !DIExpression(), !562029) + #dbg_value(ptr poison, !559346, !DIExpression(), !562031) + #dbg_value(i64 %iter.sroa.0.055.i, !559351, !DIExpression(), !562031) + #dbg_value(ptr poison, !561841, !DIExpression(), !562033) + #dbg_value(i64 %iter.sroa.0.055.i, !561851, !DIExpression(), !562033) + %75 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.055.i, !dbg !562035 + %_0.i5.i.i = load i64, ptr %75, align 8, !dbg !562035, !noalias !562036, !noundef !23 + %76 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.055.i, !dbg !562039 + %_0.i.i123.i = load i64, ptr %76, align 8, !dbg !562039, !noalias !562036, !noundef !23 + %_3.i126.i = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %iter.sroa.0.055.i, !dbg !562040 + #dbg_value(ptr poison, !561857, !DIExpression(), !562041) + #dbg_value(ptr poison, !561867, !DIExpression(), !562041) + #dbg_value(i64 %_0.i.i123.i, !561868, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !562041) + #dbg_value(i64 %_0.i5.i.i, !561868, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !562041) + #dbg_value(ptr %_3.i126.i, !561863, !DIExpression(), !562041) + #dbg_value(i64 %_0.i.i123.i, !561864, !DIExpression(), !562043) + #dbg_value(i64 %_0.i.i123.i, !561873, !DIExpression(), !562044) + #dbg_value(i64 %_0.i5.i.i, !561866, !DIExpression(), !562043) + #dbg_value(i64 %_0.i5.i.i, !561880, !DIExpression(), !562044) + #dbg_value(ptr %_3.i126.i, !561879, !DIExpression(), !562044) + #dbg_value(ptr %_3.i126.i, !561886, !DIExpression(), !562046) + #dbg_value(i64 %_0.i.i123.i, !561892, !DIExpression(), !562048) + #dbg_value(i64 %_0.i5.i.i, !561901, !DIExpression(), !562048) + #dbg_value(i64 %_0.i.i123.i, !561904, !DIExpression(), !562050) + #dbg_value(i64 %_0.i.i123.i, !561910, !DIExpression(), !562052) + #dbg_value(i64 %_0.i5.i.i, !561907, !DIExpression(), !562050) + #dbg_value(i64 %_0.i5.i.i, !561913, !DIExpression(), !562052) + %_0.i.i128.i = mul i64 %_0.i.i123.i, %_0.i5.i.i, !dbg !562054 + #dbg_value(i64 %_0.i.i123.i, !561917, !DIExpression(), !562055) + #dbg_value(i64 %_0.i.i123.i, !561923, !DIExpression(), !562057) + #dbg_value(i64 %_0.i5.i.i, !561922, !DIExpression(), !562055) + #dbg_value(i64 %_0.i5.i.i, !561925, !DIExpression(), !562057) + %_4.i1.i.i = sext i64 %_0.i.i123.i to i128, !dbg !562058 + %_5.i.i.i = sext i64 %_0.i5.i.i to i128, !dbg !562059 + %wide.i.i.i = mul nsw i128 %_4.i1.i.i, %_5.i.i.i, !dbg !562058 + #dbg_value(i128 %wide.i.i.i, !561926, !DIExpression(), !562060) + %kept.i.i.i = trunc i128 %wide.i.i.i to i64, !dbg !562061 + #dbg_value(i64 %kept.i.i.i, !561928, !DIExpression(), !562062) + %_8.i.i.i = lshr i128 %wide.i.i.i, 64, !dbg !562063 + %discarded.i.i.i = trunc nuw i128 %_8.i.i.i to i64, !dbg !562064 + #dbg_value(i64 %discarded.i.i.i, !561930, !DIExpression(), !562065) + %_10.i.i.i = ashr i64 %kept.i.i.i, 63, !dbg !562066 + %_9.i.i.i = xor i64 %_10.i.i.i, %discarded.i.i.i, !dbg !562067 + #dbg_value(i64 %_0.i.i128.i, !561881, !DIExpression(), !562068) + #dbg_value(i64 %_0.i.i128.i, !561889, !DIExpression(), !562046) + #dbg_value(i64 %_9.i.i.i, !561883, !DIExpression(), !562068) + store i64 %_0.i.i128.i, ptr %_3.i126.i, align 8, !dbg !562069, !alias.scope !562070, !noalias !561484 + #dbg_value(i64 %_9.i.i.i, !561469, !DIExpression(), !561472) + #dbg_value(ptr undef, !561463, !DIExpression(), !561472) + %77 = or i64 %_9.i.i.i, %accumulated.sroa.0.054.i, !dbg !562073 + #dbg_value(i64 %77, !561355, !DIExpression(), !561631) + #dbg_value(i64 %_16456.i, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + #dbg_value(ptr undef, !561426, !DIExpression(), !561451) + #dbg_value(ptr undef, !561414, !DIExpression(), !561447) + #dbg_value(ptr undef, !561430, !DIExpression(), !561452) + #dbg_value(ptr poison, !561433, !DIExpression(), !561452) + %_164.i = add i64 %_16456.i, 1, !dbg !562074 + #dbg_value(i64 poison, !561376, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !561961) + %exitcond.not.i = icmp eq i64 %_16456.i, %4, !dbg !561962 + br i1 %exitcond.not.i, label %bb54.i, label %bb28.i, !dbg !561963 + +bb59.i: ; preds = %bb24.i + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(48) %71, ptr noundef nonnull align 8 dereferenceable(48) %_59.i, i64 48, i1 false), !dbg !562075, !noalias !561484 + call void @llvm.lifetime.end.p0(i64 48, ptr nonnull %_59.i), !dbg !561556, !noalias !561565 + %_49.sroa.4.0..sroa_idx.i = getelementptr inbounds nuw i8, ptr %_0, i64 16, !dbg !561964 + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(24) %_49.sroa.4.0..sroa_idx.i, ptr noundef nonnull align 8 dereferenceable(24) %_57.i, i64 24, i1 false), !dbg !561556, !noalias !561605 + call void @llvm.lifetime.end.p0(i64 24, ptr nonnull %_57.i), !dbg !561556, !noalias !561565 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md new file mode 100644 index 00000000000..9d2c024bac0 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-i64-mul-dense-s.md @@ -0,0 +1,69 @@ + + + +# `candidate-i64-mul-dense.s` + +```s + movq -376(%rbp), %r15 +.Ltmp108913: + .loc 524 86 32 + testq %r15, %r15 + .loc 524 86 16 is_stmt 0 + je .LBB1673_26 +.Ltmp108914: + .loc 563 318 19 is_stmt 1 + xorq %r14, %rdi +.Ltmp108915: + .loc 563 0 19 is_stmt 0 + xorq %r14, %r10 +.Ltmp108916: + .loc 524 88 17 is_stmt 1 + orq %rdi, %r10 +.Ltmp108917: + jne .LBB1673_42 +.Ltmp108918: + .loc 182 1904 50 + testq %r14, %r14 +.Ltmp108919: + .loc 524 92 26 + je .LBB1673_41 +.Ltmp108920: + .loc 524 0 26 is_stmt 0 + movq -320(%rbp), %rsi +.Ltmp108921: + xorl %edi, %edi + xorl %ecx, %ecx +.Ltmp108922: + .p2align 4 +.LBB1673_25: + .loc 564 62 9 is_stmt 1 + movq (%r15,%rdi,8), %rax +.Ltmp108923: + .loc 565 193 24 + imulq (%rsi,%rdi,8) +.Ltmp108924: + .loc 207 475 9 + movq %rax, (%r9,%rdi,8) +.Ltmp108925: + .loc 565 197 26 + sarq $63, %rax +.Ltmp108926: + .loc 565 197 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp108927: + .loc 566 109 21 is_stmt 1 + orq %rax, %rcx +.Ltmp108928: + .loc 182 1904 50 + incq %rdi +.Ltmp108929: + cmpq %rdi, %r14 + jne .LBB1673_25 + jmp .LBB1673_62 +.Ltmp108930: +.LBB1673_26: + .loc 563 90 47 + cmpq %r14, %rdi + sete %cl + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md new file mode 100644 index 00000000000..05db6cccb49 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-ll.md @@ -0,0 +1,137 @@ + + + +# `candidate-u64-mul-dense.ll` + +```ll +terminate.i81.i: ; preds = %cleanup.i80.i + %114 = landingpad { ptr, i32 } + filter [0 x ptr] zeroinitializer +; call core::panicking::panic_in_cleanup + call void @_ZN4core9panicking16panic_in_cleanup17h8f68387bb6cbbf54E() #88, !dbg !588172, !noalias !587626 + unreachable, !dbg !588172 + +bb28.i: ; preds = %bb28.i, %bb28.lr.ph.i.new + %_16456.i = phi i64 [ 1, %bb28.lr.ph.i.new ], [ %_164.i.1, %bb28.i ] + %iter.sroa.0.055.i = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %_164.i, %bb28.i ] + %accumulated.sroa.0.054.i = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %120, %bb28.i ] + %niter = phi i64 [ 0, %bb28.lr.ph.i.new ], [ %niter.next.1, %bb28.i ] + #dbg_value(i64 %iter.sroa.0.055.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %accumulated.sroa.0.054.i, !587504, !DIExpression(), !587773) + #dbg_value(i64 %iter.sroa.0.055.i, !587527, !DIExpression(), !588173) + #dbg_value(ptr undef, !579662, !DIExpression(), !587607) + #dbg_value(i64 %iter.sroa.0.055.i, !579668, !DIExpression(), !587607) + #dbg_value(ptr poison, !580362, !DIExpression(), !588174) + #dbg_value(i64 %iter.sroa.0.055.i, !580367, !DIExpression(), !588174) + #dbg_value(ptr poison, !580362, !DIExpression(), !588176) + #dbg_value(i64 %iter.sroa.0.055.i, !580367, !DIExpression(), !588176) + #dbg_value(ptr poison, !587985, !DIExpression(), !588178) + #dbg_value(i64 %iter.sroa.0.055.i, !587986, !DIExpression(), !588178) + %115 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.055.i, !dbg !588180 + %_0.i5.i.i = load i64, ptr %115, align 8, !dbg !588180, !noalias !588181, !noundef !23 + %116 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.055.i, !dbg !588184 + %_0.i.i123.i = load i64, ptr %116, align 8, !dbg !588184, !noalias !588181, !noundef !23 + %_3.i126.i = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %iter.sroa.0.055.i, !dbg !588185 + #dbg_value(ptr poison, !588025, !DIExpression(), !588186) + #dbg_value(ptr poison, !588026, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i, !588027, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588186) + #dbg_value(i64 %_0.i5.i.i, !588027, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !588186) + #dbg_value(ptr %_3.i126.i, !588022, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i, !588023, !DIExpression(), !588188) + #dbg_value(i64 %_0.i.i123.i, !588010, !DIExpression(), !588189) + #dbg_value(i64 %_0.i5.i.i, !588024, !DIExpression(), !588188) + #dbg_value(i64 %_0.i5.i.i, !588011, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i, !588009, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i, !588036, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i123.i, !588001, !DIExpression(), !588193) + #dbg_value(i64 %_0.i5.i.i, !588002, !DIExpression(), !588193) + #dbg_value(i64 %_0.i.i123.i, !588067, !DIExpression(), !588195) + #dbg_value(i64 %_0.i.i123.i, !588073, !DIExpression(), !588197) + #dbg_value(i64 %_0.i5.i.i, !588070, !DIExpression(), !588195) + #dbg_value(i64 %_0.i5.i.i, !588076, !DIExpression(), !588197) + %_0.i3.i.i = mul i64 %_0.i.i123.i, %_0.i5.i.i, !dbg !588199 + #dbg_value(i64 %_0.i.i123.i, !587992, !DIExpression(), !588200) + #dbg_value(i64 %_0.i.i123.i, !587994, !DIExpression(), !588202) + #dbg_value(i64 %_0.i5.i.i, !587993, !DIExpression(), !588200) + #dbg_value(i64 %_0.i5.i.i, !587995, !DIExpression(), !588202) + %_5.i.i.i = zext i64 %_0.i.i123.i to i128, !dbg !588203 + %_6.i.i.i = zext i64 %_0.i5.i.i to i128, !dbg !588204 + %_4.i1.i.i = mul nuw i128 %_5.i.i.i, %_6.i.i.i, !dbg !588205 + %_3.i2.i.i = lshr i128 %_4.i1.i.i, 64, !dbg !588206 + %_0.i.i128.i = trunc nuw i128 %_3.i2.i.i to i64, !dbg !588207 + #dbg_value(i64 %_0.i3.i.i, !588012, !DIExpression(), !588208) + #dbg_value(i64 %_0.i3.i.i, !588037, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i128.i, !588014, !DIExpression(), !588208) + store i64 %_0.i3.i.i, ptr %_3.i126.i, align 8, !dbg !588209, !alias.scope !588210, !noalias !587626 + #dbg_value(i64 %_0.i.i128.i, !561469, !DIExpression(), !587614) + #dbg_value(ptr undef, !561463, !DIExpression(), !587614) + %117 = or i64 %accumulated.sroa.0.054.i, %_0.i.i128.i, !dbg !588213 + #dbg_value(i64 %117, !587504, !DIExpression(), !587773) + #dbg_value(i64 %_16456.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(ptr undef, !587575, !DIExpression(), !587600) + #dbg_value(ptr undef, !587563, !DIExpression(), !587596) + #dbg_value(ptr undef, !587579, !DIExpression(), !587601) + #dbg_value(ptr poison, !587582, !DIExpression(), !587601) + %_164.i = add i64 %_16456.i, 1, !dbg !588214 + #dbg_value(i64 poison, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %_16456.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + #dbg_value(i64 %_16456.i, !587527, !DIExpression(), !588173) + #dbg_value(i64 %_16456.i, !579668, !DIExpression(), !587607) + #dbg_value(i64 %_16456.i, !580367, !DIExpression(), !588174) + #dbg_value(i64 %_16456.i, !580367, !DIExpression(), !588176) + #dbg_value(i64 %_16456.i, !587986, !DIExpression(), !588178) + %118 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_16456.i, !dbg !588180 + %_0.i5.i.i.1 = load i64, ptr %118, align 8, !dbg !588180, !noalias !588181, !noundef !23 + %119 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_16456.i, !dbg !588184 + %_0.i.i123.i.1 = load i64, ptr %119, align 8, !dbg !588184, !noalias !588181, !noundef !23 + %_3.i126.i.1 = getelementptr inbounds nuw i64, ptr %ptr.i.i, i64 %_16456.i, !dbg !588185 + #dbg_value(i64 %_0.i.i123.i.1, !588027, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588186) + #dbg_value(i64 %_0.i5.i.i.1, !588027, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !588186) + #dbg_value(ptr %_3.i126.i.1, !588022, !DIExpression(), !588186) + #dbg_value(i64 %_0.i.i123.i.1, !588023, !DIExpression(), !588188) + #dbg_value(i64 %_0.i.i123.i.1, !588010, !DIExpression(), !588189) + #dbg_value(i64 %_0.i5.i.i.1, !588024, !DIExpression(), !588188) + #dbg_value(i64 %_0.i5.i.i.1, !588011, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i.1, !588009, !DIExpression(), !588189) + #dbg_value(ptr %_3.i126.i.1, !588036, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i123.i.1, !588001, !DIExpression(), !588193) + #dbg_value(i64 %_0.i5.i.i.1, !588002, !DIExpression(), !588193) + #dbg_value(i64 %_0.i.i123.i.1, !588067, !DIExpression(), !588195) + #dbg_value(i64 %_0.i.i123.i.1, !588073, !DIExpression(), !588197) + #dbg_value(i64 %_0.i5.i.i.1, !588070, !DIExpression(), !588195) + #dbg_value(i64 %_0.i5.i.i.1, !588076, !DIExpression(), !588197) + %_0.i3.i.i.1 = mul i64 %_0.i.i123.i.1, %_0.i5.i.i.1, !dbg !588199 + #dbg_value(i64 %_0.i.i123.i.1, !587992, !DIExpression(), !588200) + #dbg_value(i64 %_0.i.i123.i.1, !587994, !DIExpression(), !588202) + #dbg_value(i64 %_0.i5.i.i.1, !587993, !DIExpression(), !588200) + #dbg_value(i64 %_0.i5.i.i.1, !587995, !DIExpression(), !588202) + %_5.i.i.i.1 = zext i64 %_0.i.i123.i.1 to i128, !dbg !588203 + %_6.i.i.i.1 = zext i64 %_0.i5.i.i.1 to i128, !dbg !588204 + %_4.i1.i.i.1 = mul nuw i128 %_5.i.i.i.1, %_6.i.i.i.1, !dbg !588205 + %_3.i2.i.i.1 = lshr i128 %_4.i1.i.i.1, 64, !dbg !588206 + %_0.i.i128.i.1 = trunc nuw i128 %_3.i2.i.i.1 to i64, !dbg !588207 + #dbg_value(i64 %_0.i3.i.i.1, !588012, !DIExpression(), !588208) + #dbg_value(i64 %_0.i3.i.i.1, !588037, !DIExpression(), !588191) + #dbg_value(i64 %_0.i.i128.i.1, !588014, !DIExpression(), !588208) + store i64 %_0.i3.i.i.1, ptr %_3.i126.i.1, align 8, !dbg !588209, !alias.scope !588210, !noalias !587626 + #dbg_value(i64 %_0.i.i128.i.1, !561469, !DIExpression(), !587614) + %120 = or i64 %117, %_0.i.i128.i.1, !dbg !588213 + #dbg_value(i64 %120, !587504, !DIExpression(), !587773) + #dbg_value(i64 %_164.i, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + %_164.i.1 = add i64 %_16456.i, 2, !dbg !588214 + #dbg_value(i64 poison, !587525, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !588106) + %niter.next.1 = add i64 %niter, 2, !dbg !588108 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !588108 + br i1 %niter.ncmp.1, label %bb54.i.loopexit135.unr-lcssa, label %bb28.i, !dbg !588108 + +bb59.i: ; preds = %bb24.i + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(48) %111, ptr noundef nonnull align 8 dereferenceable(48) %_59.i, i64 48, i1 false), !dbg !588215, !noalias !587626 + call void @llvm.lifetime.end.p0(i64 48, ptr nonnull %_59.i), !dbg !587698, !noalias !587707 + %_49.sroa.4.0..sroa_idx.i = getelementptr inbounds nuw i8, ptr %_0, i64 16, !dbg !588109 + call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align 8 dereferenceable(24) %_49.sroa.4.0..sroa_idx.i, ptr noundef nonnull align 8 dereferenceable(24) %_57.i, i64 24, i1 false), !dbg !587698, !noalias !587747 + call void @llvm.lifetime.end.p0(i64 24, ptr nonnull %_57.i), !dbg !587698, !noalias !587707 + br label %bb61.i, !dbg !587940 + +bb39.i: ; preds = %bb2.i109.i, %"_ZN12vortex_array9scalar_fn3row7element5tuple18ArgColumn$LT$T$GT$14addresses_rows17h8cb4442712b37c9aE.exit.i.i" + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md new file mode 100644 index 00000000000..0676296c7a7 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/candidate-u64-mul-dense-s.md @@ -0,0 +1,70 @@ + + + +# `candidate-u64-mul-dense.s` + +```s + .loc 524 92 26 + andq $-2, %r15 + leaq (%r12,%rdx), %r11 + addq $8, %r11 + xorl %ecx, %ecx + xorl %r10d, %r10d +.Ltmp117915: +.LBB1693_70: + .loc 564 62 9 + movq (%r14,%r10,8), %rax +.Ltmp117916: + .loc 565 175 44 + mulq (%r8,%r10,8) +.Ltmp117917: + movq %rdx, %rdi +.Ltmp117918: + .loc 207 475 9 + movq %rax, -8(%r11,%r10,8) +.Ltmp117919: + .loc 564 62 9 + movq 8(%r14,%r10,8), %rax +.Ltmp117920: + .loc 565 175 44 + mulq 8(%r8,%r10,8) +.Ltmp117921: + .loc 566 109 21 + orq %rcx, %rdi +.Ltmp117922: + .loc 207 475 9 + movq %rax, (%r11,%r10,8) +.Ltmp117923: + .loc 565 175 44 + movq %rdx, %rcx +.Ltmp117924: + .loc 566 109 21 + orq %rdi, %rcx +.Ltmp117925: + .loc 524 92 26 + addq $2, %r10 + cmpq %r10, %r15 + jne .LBB1693_70 +.Ltmp117926: +.LBB1693_71: + testb $1, %sil + je .LBB1693_87 +.Ltmp117927: + .loc 564 62 9 + movq (%r14,%r10,8), %rax +.Ltmp117928: + .loc 565 175 44 + mulq (%r8,%r10,8) +.Ltmp117929: +.LBB1693_73: + .loc 207 475 9 + movq %rax, (%r9,%r10,8) +.Ltmp117930: + .loc 566 109 21 + orq %rdx, %rcx +.Ltmp117931: + .loc 566 0 21 is_stmt 0 + jmp .LBB1693_87 +.Ltmp117932: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md b/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md new file mode 100644 index 00000000000..185018ab6a9 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/copy-ablation.md @@ -0,0 +1,52 @@ + + + +# `Copy`-bound compiler ablation + +This note records the compact evidence behind the no-drop assertion in owned RowFn execution. + +## Timings + +All public `binary_ops` runs used CPU 8 and the same default repository flags. + +```text +no Copy bound mul_i32_constant 18.77 / 18.72 us +inert private marker bound mul_i32_constant 18.77 / 18.72 us +Output: Copy mul_i32_constant 29.94 / 29.93 us +Output: Copy, CGU=1 mul_i32_constant 29.87 / 29.89 us +i64/u64 controls unchanged +``` + +The inert marker rules out a generic “any where-clause/source change perturbs codegen” explanation. +`codegen-units=1` rules out the default partitioning choice as a repair. + +## Exact production functions + +Default-CGU DWARF identified the measured `i32 CheckedMul` monomorphs: + +```text +Copy: 0xe58c90..0xe59adc, size 0xe4c +no-Copy: 0xe7a1c0..0xe7b6d0, size 0x1510 + +Copy hot loop: 0xe58f90, 16-byte but not 32-byte aligned +no-Copy hot loop: 0xe7b260, 32-byte aligned +``` + +The Copy loop schedules the low `imul` before the widened-product chain. No-Copy schedules the +widened chain first and delays the low multiply. LLVM-MCA predicts Copy slightly better at 2.5 +cycles versus 2.7, contradicting scheduling as the cause of its 1.6x wall-time loss. + +Fresh Copy-plus-CGU1 optimized IR contains store-before-OR and still runs at 29.9 microseconds. +Therefore store-before-OR is not sufficient. Exact isolated loops also contradict causality: + +```text +LLVM-MCA: both orders 2.7 cycles +OR before store: 0.75-0.77 ns/row +store before OR: 0.823-0.825 ns/row +``` + +The standalone generic `MaybeUninit` loop produces identical Copy/no-Copy IR and assembly. The +remaining hypothesis is phase-order or code-quality sensitivity requiring the real trait, closure, +`Vec`, and monomorphization context. Do not label this a correctness bug or assign it to a specific +rustc/LLVM pass without a reduced reproducer. Reduce the real monomorph while retaining timing and +whole-function changes, then bisect MIR/LLVM passes and compiler versions. diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md new file mode 100644 index 00000000000..7513df3b465 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-ll.md @@ -0,0 +1,86 @@ + + + +# `final-i32-mul-constant.ll` + +```ll +bb27.preheader.i.split.us: ; preds = %bb27.preheader.i + br i1 %_3.i5.not.i.i, label %panic.i5.i5.i.invoke.i, label %bb27.i.us.preheader + +bb27.i.us.preheader: ; preds = %bb27.preheader.i.split.us + %57 = add nuw nsw i64 %len3.i.i.i, 1, !dbg !566996 + br label %bb27.i.us, !dbg !566996 + +bb27.i.us: ; preds = %bb27.i.us.preheader, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" + %_15854.i.us = phi i64 [ %_158.i.us, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ 1, %bb27.i.us.preheader ] + %iter.sroa.0.053.i.us = phi i64 [ %_15854.i.us, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ 0, %bb27.i.us.preheader ] + %accumulated.sroa.0.052.i.us = phi i1 [ %60, %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us" ], [ false, %bb27.i.us.preheader ] + #dbg_value(i64 %iter.sroa.0.053.i.us, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + #dbg_value(i64 %iter.sroa.0.053.i.us, !566732, !DIExpression(), !567057) + #dbg_value(ptr %columns.i, !545259, !DIExpression(), !567058) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545260, !DIExpression(), !567058) + #dbg_value(ptr %columns.i, !545249, !DIExpression(), !567059) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545250, !DIExpression(), !567059) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567060) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567062) + #dbg_value(ptr %columns.i, !545251, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567063) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545125, !DIExpression(), !567062) + %exitcond35.not = icmp eq i64 %_15854.i.us, %57, !dbg !566996 + br i1 %exitcond35.not, label %panic.i5.i5.i.invoke.i, label %"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us", !dbg !566996 + +"_ZN12vortex_array9scalar_fn3row7element9primitive83_$LT$impl$u20$vortex_array..scalar_fn..row..element..InputElement$u20$for$u20$T$GT$3get17h2f14351c5788fe60E.exit8.i.i.i.us": ; preds = %bb27.i.us + #dbg_value(ptr %columns.i, !545251, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567063) + #dbg_value(ptr %columns.i, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567062) + %58 = getelementptr inbounds nuw i32, ptr %data.i6.i.i.i, i64 %iter.sroa.0.053.i.us, !dbg !566996 + %_0.sroa.0.0.i.i.i.us = load i32, ptr %58, align 4, !dbg !567000, !noalias !566784, !noundef !23 + #dbg_value(ptr %14, !545249, !DIExpression(), !567064) + #dbg_value(i64 %iter.sroa.0.053.i.us, !545250, !DIExpression(), !567064) + #dbg_value(ptr %14, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567065) + #dbg_value(ptr %14, !545120, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567067) + #dbg_value(ptr %14, !545252, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !567068) + #dbg_value(i64 0, !545125, !DIExpression(), !567065) + %_0.sroa.0.0.i9.i.i.us = load i32, ptr %data.i6.i7.i.i, align 4, !dbg !567005, !noalias !566784, !noundef !23 + #dbg_value(ptr poison, !567031, !DIExpression(), !567069) + #dbg_value(ptr poison, !567032, !DIExpression(), !567069) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567033, !DIExpression(DW_OP_LLVM_fragment, 0, 32), !567069) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567033, !DIExpression(DW_OP_LLVM_fragment, 32, 32), !567069) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567029, !DIExpression(), !567070) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567030, !DIExpression(), !567070) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567020, !DIExpression(), !567071) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567021, !DIExpression(), !567071) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567015, !DIExpression(), !567072) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567010, !DIExpression(), !567073) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567016, !DIExpression(), !567072) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567011, !DIExpression(), !567073) + %_0.i.i160.i.us = mul i32 %_0.sroa.0.0.i9.i.i.us, %_0.sroa.0.0.i.i.i.us, !dbg !567007 + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567039, !DIExpression(), !567074) + #dbg_value(i32 %_0.sroa.0.0.i.i.i.us, !567041, !DIExpression(), !567075) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567040, !DIExpression(), !567074) + #dbg_value(i32 %_0.sroa.0.0.i9.i.i.us, !567042, !DIExpression(), !567075) + %_4.i1.i.i.us = sext i32 %_0.sroa.0.0.i.i.i.us to i64, !dbg !567035 + %_5.i.i.i.us = sext i32 %_0.sroa.0.0.i9.i.i.us to i64, !dbg !567046 + %product.i.i.i.us = mul nsw i64 %_5.i.i.i.us, %_4.i1.i.i.us, !dbg !567035 + #dbg_value(i64 %product.i.i.i.us, !567043, !DIExpression(), !567076) + %59 = add nsw i64 %product.i.i.i.us, -2147483648, !dbg !567047 + %_0.sroa.0.0.i.i161.i.us = icmp ult i64 %59, -4294967296, !dbg !567047 + #dbg_value(i1 %_0.sroa.0.0.i.i161.i.us, !566736, !DIExpression(DW_OP_LLVM_convert, 1, DW_ATE_unsigned, DW_OP_LLVM_convert, 8, DW_ATE_unsigned, DW_OP_stack_value), !567077) + #dbg_value(i32 %_0.i.i160.i.us, !566734, !DIExpression(), !567077) + %self34.i.us = getelementptr inbounds nuw i32, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.053.i.us, !dbg !567048 + #dbg_value(ptr %self34.i.us, !567052, !DIExpression(), !567078) + #dbg_value(i32 %_0.i.i160.i.us, !567053, !DIExpression(), !567078) + store i32 %_0.i.i160.i.us, ptr %self34.i.us, align 4, !dbg !567049, !noalias !566784 + #dbg_value(ptr undef, !541560, !DIExpression(), !566763) + #dbg_value(i1 %_0.sroa.0.0.i.i161.i.us, !541568, !DIExpression(DW_OP_LLVM_convert, 1, DW_ATE_unsigned, DW_OP_LLVM_convert, 8, DW_ATE_unsigned, DW_OP_stack_value), !566763) + %60 = or i1 %accumulated.sroa.0.052.i.us, %_0.sroa.0.0.i.i161.i.us, !dbg !567055 + #dbg_value(i8 poison, !566728, !DIExpression(), !566992) + #dbg_value(i64 %_15854.i.us, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + #dbg_value(ptr undef, !566753, !DIExpression(), !566756) + #dbg_value(ptr undef, !566744, !DIExpression(), !566749) + #dbg_value(ptr undef, !566757, !DIExpression(), !566761) + #dbg_value(ptr poison, !566760, !DIExpression(), !566761) + %_158.i.us = add i64 %_15854.i.us, 1, !dbg !567079 + #dbg_value(i64 poison, !566730, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !566993) + %exitcond.not.i.us = icmp eq i64 %_15854.i.us, %len3.i.i.i, !dbg !566994 + br i1 %exitcond.not.i.us, label %bb33.i, label %bb27.i.us, !dbg !566995 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md new file mode 100644 index 00000000000..f82f250cdd1 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i32-mul-constant-s.md @@ -0,0 +1,49 @@ + + + +# `final-i32-mul-constant.s` + +```s +.LBB1677_31: + .loc 564 47 9 is_stmt 1 + cmpq %rdi, %rdx + je .LBB1677_89 +.Ltmp110238: + .loc 564 47 9 is_stmt 0 + movslq (%r13,%rdi,4), %rcx +.Ltmp110239: + .loc 564 47 9 + movslq (%r12), %r8 +.Ltmp110240: + .loc 462 2133 13 is_stmt 1 + movl %r8d, %r10d + imull %ecx, %r10d +.Ltmp110241: + .loc 565 185 27 + imulq %rcx, %r8 +.Ltmp110242: + .loc 565 185 35 is_stmt 0 + addq $-2147483648, %r8 +.Ltmp110243: + cmpq %rax, %r8 + setb %cl +.Ltmp110244: + .loc 565 0 35 + movq -48(%rbp), %r8 +.Ltmp110245: + .loc 207 475 9 is_stmt 1 + movl %r10d, (%r8,%rdi,4) +.Ltmp110246: + .loc 566 821 53 + orb %cl, %r9b +.Ltmp110247: + .loc 182 1904 50 + incq %rdi +.Ltmp110248: + cmpq %rdi, %rdx +.Ltmp110249: + .loc 562 124 26 + jne .LBB1677_31 + jmp .LBB1677_64 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md new file mode 100644 index 00000000000..6b28482775c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-ll.md @@ -0,0 +1,96 @@ + + + +# `final-i64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb11.i, %bb15.i.i + %iter.sroa.0.012.i.i = phi i64 [ %_36.i.i, %bb15.i.i ], [ 0, %bb11.i ] + %failed.sroa.0.011.i.i = phi i64 [ %79, %bb15.i.i ], [ 0, %bb11.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !564425, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564479) + #dbg_value(i64 %failed.sroa.0.011.i.i, !564423, !DIExpression(), !564478) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564463, !DIExpression(), !564694) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564456, !DIExpression(), !564457) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564473, !DIExpression(), !564474) + %_36.i.i = add nuw i64 %iter.sroa.0.012.i.i, 1, !dbg !564695 + #dbg_value(i64 %_36.i.i, !564425, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564479) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564427, !DIExpression(), !564696) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564450, !DIExpression(), !564451) + #dbg_value(i64 %iter.sroa.0.012.i.i, !564697, !DIExpression(), !564701) + #dbg_value(ptr undef, !547076, !DIExpression(), !564445) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547082, !DIExpression(), !564445) + #dbg_value(ptr poison, !547154, !DIExpression(), !564703) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547155, !DIExpression(), !564703) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547147, !DIExpression(), !564705) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547139, !DIExpression(), !564707) + #dbg_value(ptr %column.val.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564705) + #dbg_value(ptr %column.val.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564707) + #dbg_value(i64 %len3.i.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564705) + #dbg_value(i64 %len3.i.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564707) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564709 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !564710, !noalias !564711, !noundef !23 + #dbg_value(ptr poison, !547154, !DIExpression(), !564715) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547155, !DIExpression(), !564715) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547147, !DIExpression(), !564717) + #dbg_value(i64 %iter.sroa.0.012.i.i, !547139, !DIExpression(), !564719) + #dbg_value(ptr %column5.val.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564717) + #dbg_value(ptr %column5.val.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564719) + #dbg_value(i64 %len3.i.i.i, !547146, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564717) + #dbg_value(i64 %len3.i.i.i, !547140, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564719) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !564721 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !564722 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564723 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !564724, !noalias !564711, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !564429, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564725) + #dbg_value(i64 %_0.i5.i.i.i, !564429, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564725) + #dbg_value(i64 %_0.i.i.i.i, !564726, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564734) + #dbg_value(i64 %_0.i5.i.i.i, !564726, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564734) + #dbg_value(ptr poison, !564315, !DIExpression(), !564736) + #dbg_value(ptr poison, !564316, !DIExpression(), !564736) + #dbg_value(i64 %_0.i.i.i.i, !564317, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564736) + #dbg_value(i64 %_0.i5.i.i.i, !564317, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564736) + #dbg_value(i64 %_0.i.i.i.i, !564313, !DIExpression(), !564738) + #dbg_value(i64 %_0.i5.i.i.i, !564314, !DIExpression(), !564738) + #dbg_value(i64 %_0.i.i.i.i, !564304, !DIExpression(), !564739) + #dbg_value(i64 %_0.i5.i.i.i, !564305, !DIExpression(), !564739) + #dbg_value(i64 %_0.i.i.i.i, !564293, !DIExpression(), !564741) + #dbg_value(i64 %_0.i.i.i.i, !564288, !DIExpression(), !564743) + #dbg_value(i64 %_0.i5.i.i.i, !564294, !DIExpression(), !564741) + #dbg_value(i64 %_0.i5.i.i.i, !564289, !DIExpression(), !564743) + %_0.i.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !564745 + #dbg_value(i64 %_0.i.i.i.i, !564325, !DIExpression(), !564746) + #dbg_value(i64 %_0.i.i.i.i, !564327, !DIExpression(), !564748) + #dbg_value(i64 %_0.i5.i.i.i, !564326, !DIExpression(), !564746) + #dbg_value(i64 %_0.i5.i.i.i, !564328, !DIExpression(), !564748) + %_4.i1.i.i.i.i = sext i64 %_0.i.i.i.i to i128, !dbg !564749 + %_5.i.i.i.i.i = sext i64 %_0.i5.i.i.i to i128, !dbg !564750 + %wide.i.i.i.i.i = mul nsw i128 %_5.i.i.i.i.i, %_4.i1.i.i.i.i, !dbg !564749 + #dbg_value(i128 %wide.i.i.i.i.i, !564329, !DIExpression(), !564751) + %kept.i.i.i.i.i = trunc i128 %wide.i.i.i.i.i to i64, !dbg !564752 + #dbg_value(i64 %kept.i.i.i.i.i, !564331, !DIExpression(), !564753) + %_8.i.i.i.i.i = lshr i128 %wide.i.i.i.i.i, 64, !dbg !564754 + %discarded.i.i.i.i.i = trunc nuw i128 %_8.i.i.i.i.i to i64, !dbg !564755 + #dbg_value(i64 %discarded.i.i.i.i.i, !564333, !DIExpression(), !564756) + %_10.i.i.i.i.i = ashr i64 %kept.i.i.i.i.i, 63, !dbg !564757 + %_9.i.i.i.i.i = xor i64 %_10.i.i.i.i.i, %discarded.i.i.i.i.i, !dbg !564758 + #dbg_value(i64 poison, !564431, !DIExpression(), !564759) + #dbg_value(i64 %_9.i.i.i.i.i, !564433, !DIExpression(), !564759) + #dbg_value(ptr undef, !564034, !DIExpression(), !564443) + #dbg_value(i64 %_9.i.i.i.i.i, !564040, !DIExpression(), !564443) + %79 = or i64 %_9.i.i.i.i.i, %failed.sroa.0.011.i.i, !dbg !564760 + #dbg_value(i64 %79, !564423, !DIExpression(), !564478) + #dbg_value(i64 %_0.i.i.i.i.i, !564431, !DIExpression(), !564759) + #dbg_value(ptr %_4.sroa.10.0.i.i, !564700, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !564701) + #dbg_value(i64 %index.i, !564700, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !564701) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !564761 + #dbg_value(ptr %self4.i.i, !564762, !DIExpression(), !564766) + #dbg_value(i64 %_0.i.i.i.i.i, !564765, !DIExpression(), !564766) + store i64 %_0.i.i.i.i.i, ptr %self4.i.i, align 8, !dbg !564768, !alias.scope !564439, !noalias !564769 + #dbg_value(ptr undef, !564467, !DIExpression(), !564480) + #dbg_value(ptr undef, !564462, !DIExpression(), !564481) + #dbg_value(ptr undef, !564482, !DIExpression(), !564486) + #dbg_value(ptr poison, !564485, !DIExpression(), !564486) + %exitcond.not.i.i = icmp eq i64 %_36.i.i, %len3.i4.i.i.fr, !dbg !564770 + br i1 %exitcond.not.i.i, label %bb33.i, label %bb15.i.i, !dbg !564488 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md new file mode 100644 index 00000000000..56206a6ec40 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-i64-mul-dense-s.md @@ -0,0 +1,34 @@ + + + +# `final-i64-mul-dense.s` + +```s +.LBB1675_25: + .loc 567 39 18 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp109477: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp109478: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp109479: + .loc 156 717 17 + incq %rsi +.Ltmp109480: + .loc 565 198 26 + sarq $63, %rax +.Ltmp109481: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp109482: + .loc 566 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp109483: + .loc 182 1904 50 + cmpq %rsi, %r9 + jne .LBB1675_25 + jmp .LBB1675_60 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md new file mode 100644 index 00000000000..4e40cfb8f2c --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-ll.md @@ -0,0 +1,322 @@ + + + +# `final-u64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb15.i.i, %bb15.i.i.preheader.new + %iter.sroa.0.012.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %116, %bb15.i.i ] + %niter = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %niter.next.1, %bb15.i.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %failed.sroa.0.011.i.i, !569873, !DIExpression(), !569927) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569912, !DIExpression(), !570143) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569905, !DIExpression(), !569906) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569922, !DIExpression(), !569923) + %_36.i.i = or disjoint i64 %iter.sroa.0.012.i.i, 1, !dbg !570144 + #dbg_value(i64 %_36.i.i, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569877, !DIExpression(), !570145) + #dbg_value(i64 %iter.sroa.0.012.i.i, !569899, !DIExpression(), !569900) + #dbg_value(i64 %iter.sroa.0.012.i.i, !570146, !DIExpression(), !570150) + #dbg_value(ptr undef, !551471, !DIExpression(), !569894) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551477, !DIExpression(), !569894) + #dbg_value(ptr poison, !551549, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551550, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551542, !DIExpression(), !570154) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570158 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(ptr poison, !551549, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551550, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551542, !DIExpression(), !570166) + #dbg_value(i64 %iter.sroa.0.012.i.i, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !570171 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570172 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(ptr poison, !569758, !DIExpression(), !570185) + #dbg_value(ptr poison, !569759, !DIExpression(), !570185) + #dbg_value(i64 %_0.i.i.i.i, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i = zext i64 %_0.i.i.i.i to i128, !dbg !570198 + %_6.i.i.i.i.i = zext i64 %_0.i5.i.i.i to i128, !dbg !570199 + %_4.i1.i.i.i.i = mul nuw i128 %_6.i.i.i.i.i, %_5.i.i.i.i.i, !dbg !570200 + %_3.i2.i.i.i.i = lshr i128 %_4.i1.i.i.i.i, 64, !dbg !570201 + %_0.i.i.i.i.i = trunc nuw i128 %_3.i2.i.i.i.i to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i, !569883, !DIExpression(), !570203) + #dbg_value(ptr undef, !564034, !DIExpression(), !569892) + #dbg_value(i64 %_0.i.i.i.i.i, !564040, !DIExpression(), !569892) + %115 = or i64 %failed.sroa.0.011.i.i, %_0.i.i.i.i.i, !dbg !570204 + #dbg_value(i64 %115, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !570205 + #dbg_value(ptr %self4.i.i, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i, ptr %self4.i.i, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + #dbg_value(ptr undef, !569916, !DIExpression(), !569929) + #dbg_value(ptr undef, !569911, !DIExpression(), !569930) + #dbg_value(ptr undef, !569931, !DIExpression(), !569935) + #dbg_value(ptr poison, !569934, !DIExpression(), !569935) + #dbg_value(i64 %_36.i.i, !569912, !DIExpression(), !570143) + #dbg_value(i64 %_36.i.i, !569905, !DIExpression(), !569906) + #dbg_value(i64 %_36.i.i, !569922, !DIExpression(), !569923) + %_36.i.i.1 = add nuw i64 %iter.sroa.0.012.i.i, 2, !dbg !570144 + #dbg_value(i64 %_36.i.i.1, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %_36.i.i, !569877, !DIExpression(), !570145) + #dbg_value(i64 %_36.i.i, !569899, !DIExpression(), !569900) + #dbg_value(i64 %_36.i.i, !570146, !DIExpression(), !570150) + #dbg_value(i64 %_36.i.i, !551477, !DIExpression(), !569894) + #dbg_value(i64 %_36.i.i, !551550, !DIExpression(), !570152) + #dbg_value(i64 %_36.i.i, !551542, !DIExpression(), !570154) + #dbg_value(i64 %_36.i.i, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_36.i.i, !dbg !570158 + %_0.i.i.i.i.1 = load i64, ptr %_4.i.i.i.i.1, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(i64 %_36.i.i, !551550, !DIExpression(), !570164) + #dbg_value(i64 %_36.i.i, !551542, !DIExpression(), !570166) + #dbg_value(i64 %_36.i.i, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i.1 = icmp ult i64 %_36.i.i, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.1), !dbg !570171 + %_4.i4.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_36.i.i, !dbg !570172 + %_0.i5.i.i.i.1 = load i64, ptr %_4.i4.i.i.i.1, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.1, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i.1, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i.1, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i.1, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(i64 %_0.i.i.i.i.1, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i.1, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i.1, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i.1, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i.1, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i.1, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i.1, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i.1, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i.1, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i.1, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i.1 = mul i64 %_0.i5.i.i.i.1, %_0.i.i.i.i.1, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i.1, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i.1, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i.1, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i.1, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i.1 = zext i64 %_0.i.i.i.i.1 to i128, !dbg !570198 + %_6.i.i.i.i.i.1 = zext i64 %_0.i5.i.i.i.1 to i128, !dbg !570199 + %_4.i1.i.i.i.i.1 = mul nuw i128 %_6.i.i.i.i.i.1, %_5.i.i.i.i.i.1, !dbg !570200 + %_3.i2.i.i.i.i.1 = lshr i128 %_4.i1.i.i.i.i.1, 64, !dbg !570201 + %_0.i.i.i.i.i.1 = trunc nuw i128 %_3.i2.i.i.i.i.1 to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.1, !569883, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.1, !564040, !DIExpression(), !569892) + %116 = or i64 %115, %_0.i.i.i.i.i.1, !dbg !570204 + #dbg_value(i64 %116, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_36.i.i, !dbg !570205 + #dbg_value(ptr %self4.i.i.1, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i.1, ptr %self4.i.i.1, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + %niter.next.1 = add i64 %niter, 2, !dbg !569937 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !569937 + br i1 %niter.ncmp.1, label %bb33.i.loopexit144.unr-lcssa, label %bb15.i.i, !dbg !569937 + +bb33.thread.i: ; preds = %bb26.preheader.i.thread, %bb11.i, %bb26.preheader.i + #dbg_value(i64 0, !569430, !DIExpression(), !570214) + #dbg_value(i64 %index.i, !569423, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !569651) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !570215, !noalias !569589 + #dbg_value(i64 0, !569401, !DIExpression(), !570217) + #dbg_declare(ptr poison, !569405, !DIExpression(), !570218) + #dbg_declare(ptr %value.i.i, !570219, !DIExpression(), !570222) + #dbg_value(ptr undef, !564775, !DIExpression(), !570225) + #dbg_value(ptr undef, !564776, !DIExpression(), !570225) + br label %bb36.i, !dbg !570226 + +bb33.i.loopexit.unr-lcssa: ; preds = %bb27.us.i.us, %bb27.us.i.us.preheader + %.lcssa.ph = phi i64 [ poison, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %iter.sroa.0.046.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %_158.us.i.us, %bb27.us.i.us ] + %accumulated.sroa.0.045.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %lcmp.mod148.not = icmp eq i64 %xtraiter147, 0, !dbg !569720 + br i1 %lcmp.mod148.not, label %bb33.i, label %bb27.us.i.us.epil, !dbg !569720 + +bb27.us.i.us.epil: ; preds = %bb33.i.loopexit.unr-lcssa + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + #dbg_value(i64 %accumulated.sroa.0.045.us.i.us.unr, !569447, !DIExpression(), !569717) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !569451, !DIExpression(), !569793) + #dbg_value(ptr %columns.i, !551276, !DIExpression(), !569794) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551277, !DIExpression(), !569794) + #dbg_value(ptr %columns.i, !551266, !DIExpression(), !569795) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551267, !DIExpression(), !569795) + #dbg_value(ptr %columns.i, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569798) + #dbg_value(i64 0, !551142, !DIExpression(), !569796) + #dbg_value(ptr %columns.i, !551269, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569827) + #dbg_value(ptr %columns.i, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569796) + %_0.sroa.0.0.i.i.us.i.us.epil = load i64, ptr %data.i.i.i.us.i, align 8, !dbg !569725, !noalias !569509, !noundef !23 + #dbg_value(ptr %14, !551266, !DIExpression(), !569800) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !551267, !DIExpression(), !569800) + #dbg_value(ptr %14, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569801) + #dbg_value(ptr %14, !551137, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569803) + #dbg_value(ptr %14, !551269, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !569804) + #dbg_value(i64 0, !551142, !DIExpression(), !569801) + %_0.sroa.0.0.i9.i.us.i.us.epil = load i64, ptr %data.i6.i7.i.us.i, align 8, !dbg !569730, !noalias !569509, !noundef !23 + #dbg_value(ptr poison, !569758, !DIExpression(), !569805) + #dbg_value(ptr poison, !569759, !DIExpression(), !569805) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569805) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !569805) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569756, !DIExpression(), !569806) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569757, !DIExpression(), !569806) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569747, !DIExpression(), !569807) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569748, !DIExpression(), !569807) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569740, !DIExpression(), !569808) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569735, !DIExpression(), !569809) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569741, !DIExpression(), !569808) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569736, !DIExpression(), !569809) + %_0.i3.i.us.i.us.epil = mul i64 %_0.sroa.0.0.i9.i.us.i.us.epil, %_0.sroa.0.0.i.i.us.i.us.epil, !dbg !569732 + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569766, !DIExpression(), !569810) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !569768, !DIExpression(), !569811) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569767, !DIExpression(), !569810) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !569769, !DIExpression(), !569811) + %_5.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i.i.us.i.us.epil to i128, !dbg !569762 + %_6.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i9.i.us.i.us.epil to i128, !dbg !569771 + %_4.i1.i.us.i.us.epil = mul nuw i128 %_6.i.i.us.i.us.epil, %_5.i.i.us.i.us.epil, !dbg !569772 + %_3.i2.i.us.i.us.epil = lshr i128 %_4.i1.i.us.i.us.epil, 64, !dbg !569773 + %_0.i.i159.us.i.us.epil = trunc nuw i128 %_3.i2.i.us.i.us.epil to i64, !dbg !569774 + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !569455, !DIExpression(), !569812) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !569453, !DIExpression(), !569812) + %self34.us.i.us.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.046.us.i.us.unr, !dbg !569775 + #dbg_value(ptr %self34.us.i.us.epil, !569779, !DIExpression(), !569813) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !569780, !DIExpression(), !569813) + store i64 %_0.i3.i.us.i.us.epil, ptr %self34.us.i.us.epil, align 8, !dbg !569776, !noalias !569509 + #dbg_value(ptr undef, !564034, !DIExpression(), !569482) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !564040, !DIExpression(), !569482) + %117 = or i64 %accumulated.sroa.0.045.us.i.us.unr, %_0.i.i159.us.i.us.epil, !dbg !569782 + #dbg_value(i64 %117, !569447, !DIExpression(), !569717) + #dbg_value(i64 poison, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + #dbg_value(ptr undef, !569472, !DIExpression(), !569475) + #dbg_value(ptr undef, !569463, !DIExpression(), !569468) + #dbg_value(ptr undef, !569476, !DIExpression(), !569480) + #dbg_value(ptr poison, !569479, !DIExpression(), !569480) + #dbg_value(i64 poison, !569449, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569718) + br label %bb33.i, !dbg !570227 + +bb33.i.loopexit144.unr-lcssa: ; preds = %bb15.i.i, %bb15.i.i.preheader + %.lcssa145.ph = phi i64 [ poison, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %iter.sroa.0.012.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %lcmp.mod.not = icmp eq i64 %xtraiter, 0, !dbg !569937 + br i1 %lcmp.mod.not, label %bb33.i, label %bb15.i.i.epil, !dbg !569937 + +bb15.i.i.epil: ; preds = %bb33.i.loopexit144.unr-lcssa + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569875, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %failed.sroa.0.011.i.i.unr, !569873, !DIExpression(), !569927) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569912, !DIExpression(), !570143) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569905, !DIExpression(), !569906) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569922, !DIExpression(), !569923) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569875, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 64), !569928) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569877, !DIExpression(), !570145) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !569899, !DIExpression(), !569900) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !570146, !DIExpression(), !570150) + #dbg_value(ptr undef, !551471, !DIExpression(), !569894) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551477, !DIExpression(), !569894) + #dbg_value(ptr poison, !551549, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551550, !DIExpression(), !570152) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551542, !DIExpression(), !570154) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551534, !DIExpression(), !570156) + #dbg_value(ptr %column.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570154) + #dbg_value(ptr %column.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570156) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570154) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570156) + %_4.i.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570158 + %_0.i.i.i.i.epil = load i64, ptr %_4.i.i.i.i.epil, align 8, !dbg !570159, !noalias !570160, !noundef !23 + #dbg_value(ptr poison, !551549, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551550, !DIExpression(), !570164) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551542, !DIExpression(), !570166) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !551534, !DIExpression(), !570168) + #dbg_value(ptr %column5.val.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570166) + #dbg_value(ptr %column5.val.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570168) + #dbg_value(i64 %len3.i.i.i, !551541, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570166) + #dbg_value(i64 %len3.i.i.i, !551535, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570168) + %_5.i3.i.i.i.epil = icmp ult i64 %iter.sroa.0.012.i.i.unr, %len3.i.i.i, !dbg !570170 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.epil), !dbg !570171 + %_4.i4.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570172 + %_0.i5.i.i.i.epil = load i64, ptr %_4.i4.i.i.i.epil, align 8, !dbg !570173, !noalias !570160, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.epil, !569879, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570174) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569879, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570174) + #dbg_value(i64 %_0.i.i.i.i.epil, !570175, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570183) + #dbg_value(i64 %_0.i5.i.i.i.epil, !570175, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570183) + #dbg_value(ptr poison, !569758, !DIExpression(), !570185) + #dbg_value(ptr poison, !569759, !DIExpression(), !570185) + #dbg_value(i64 %_0.i.i.i.i.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570185) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569760, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570185) + #dbg_value(i64 %_0.i.i.i.i.epil, !569756, !DIExpression(), !570187) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569757, !DIExpression(), !570187) + #dbg_value(i64 %_0.i.i.i.i.epil, !569747, !DIExpression(), !570188) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569748, !DIExpression(), !570188) + #dbg_value(i64 %_0.i.i.i.i.epil, !569740, !DIExpression(), !570190) + #dbg_value(i64 %_0.i.i.i.i.epil, !569735, !DIExpression(), !570192) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569741, !DIExpression(), !570190) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569736, !DIExpression(), !570192) + %_0.i3.i.i.i.i.epil = mul i64 %_0.i5.i.i.i.epil, %_0.i.i.i.i.epil, !dbg !570194 + #dbg_value(i64 %_0.i.i.i.i.epil, !569766, !DIExpression(), !570195) + #dbg_value(i64 %_0.i.i.i.i.epil, !569768, !DIExpression(), !570197) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569767, !DIExpression(), !570195) + #dbg_value(i64 %_0.i5.i.i.i.epil, !569769, !DIExpression(), !570197) + %_5.i.i.i.i.i.epil = zext i64 %_0.i.i.i.i.epil to i128, !dbg !570198 + %_6.i.i.i.i.i.epil = zext i64 %_0.i5.i.i.i.epil to i128, !dbg !570199 + %_4.i1.i.i.i.i.epil = mul nuw i128 %_6.i.i.i.i.i.epil, %_5.i.i.i.i.i.epil, !dbg !570200 + %_3.i2.i.i.i.i.epil = lshr i128 %_4.i1.i.i.i.i.epil, 64, !dbg !570201 + %_0.i.i.i.i.i.epil = trunc nuw i128 %_3.i2.i.i.i.i.epil to i64, !dbg !570202 + #dbg_value(i64 poison, !569881, !DIExpression(), !570203) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !569883, !DIExpression(), !570203) + #dbg_value(ptr undef, !564034, !DIExpression(), !569892) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !564040, !DIExpression(), !569892) + %118 = or i64 %failed.sroa.0.011.i.i.unr, %_0.i.i.i.i.i.epil, !dbg !570204 + #dbg_value(i64 %118, !569873, !DIExpression(), !569927) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !569881, !DIExpression(), !570203) + #dbg_value(ptr %_4.sroa.10.0.i.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !570150) + #dbg_value(i64 %index.i, !570149, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !570150) + %self4.i.i.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !570205 + #dbg_value(ptr %self4.i.i.epil, !570206, !DIExpression(), !570210) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !570209, !DIExpression(), !570210) + store i64 %_0.i3.i.i.i.i.epil, ptr %self4.i.i.epil, align 8, !dbg !570212, !alias.scope !569888, !noalias !570213 + #dbg_value(ptr undef, !569916, !DIExpression(), !569929) + #dbg_value(ptr undef, !569911, !DIExpression(), !569930) + #dbg_value(ptr undef, !569931, !DIExpression(), !569935) + #dbg_value(ptr poison, !569934, !DIExpression(), !569935) + br label %bb33.i, !dbg !570227 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md new file mode 100644 index 00000000000..e30db12f65a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/final-u64-mul-dense-s.md @@ -0,0 +1,71 @@ + + + +# `final-u64-mul-dense.s` + +```s +.LBB1679_67: + .loc 181 765 12 + andq $-2, %r10 + xorl %edi, %edi + xorl %ecx, %ecx + movq -48(%rbp), %r8 +.Ltmp111183: +.LBB1679_68: + .loc 567 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp111184: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp111185: + movq %rdx, %rsi +.Ltmp111186: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp111187: + .loc 567 39 18 + movq 8(%r13,%rdi,8), %rax +.Ltmp111188: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp111189: + .loc 566 821 53 + orq %rcx, %rsi +.Ltmp111190: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp111191: + .loc 156 717 17 + addq $2, %rdi +.Ltmp111192: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp111193: + .loc 566 821 53 + orq %rsi, %rcx +.Ltmp111194: + .loc 181 765 12 + cmpq %r10, %rdi + jne .LBB1679_68 +.Ltmp111195: +.LBB1679_69: + testb $1, %r9b + je .LBB1679_84 +.Ltmp111196: + .loc 567 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp111197: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp111198: + .loc 566 821 53 + orq %rdx, %rcx +.Ltmp111199: + .loc 566 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp111200: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp111201: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md new file mode 100644 index 00000000000..4cd73bddfad --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-ll.md @@ -0,0 +1,96 @@ + + + +# `indexed-i64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb11.i, %bb15.i.i + %iter.sroa.0.012.i.i = phi i64 [ %_36.i.i, %bb15.i.i ], [ 0, %bb11.i ] + %failed.sroa.0.011.i.i = phi i64 [ %79, %bb15.i.i ], [ 0, %bb11.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !576781, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !576835) + #dbg_value(i64 %failed.sroa.0.011.i.i, !576779, !DIExpression(), !576834) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576819, !DIExpression(), !577050) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576812, !DIExpression(), !576813) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576829, !DIExpression(), !576830) + %_36.i.i = add nuw i64 %iter.sroa.0.012.i.i, 1, !dbg !577051 + #dbg_value(i64 %_36.i.i, !576781, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !576835) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576783, !DIExpression(), !577052) + #dbg_value(i64 %iter.sroa.0.012.i.i, !576806, !DIExpression(), !576807) + #dbg_value(i64 %iter.sroa.0.012.i.i, !577053, !DIExpression(), !577057) + #dbg_value(ptr undef, !553722, !DIExpression(), !576801) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553728, !DIExpression(), !576801) + #dbg_value(ptr poison, !553800, !DIExpression(), !577059) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553801, !DIExpression(), !577059) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553793, !DIExpression(), !577061) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553785, !DIExpression(), !577063) + #dbg_value(ptr %column.val.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577061) + #dbg_value(ptr %column.val.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577063) + #dbg_value(i64 %len3.i.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577061) + #dbg_value(i64 %len3.i.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577063) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577065 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !577066, !noalias !577067, !noundef !23 + #dbg_value(ptr poison, !553800, !DIExpression(), !577071) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553801, !DIExpression(), !577071) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553793, !DIExpression(), !577073) + #dbg_value(i64 %iter.sroa.0.012.i.i, !553785, !DIExpression(), !577075) + #dbg_value(ptr %column5.val.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577073) + #dbg_value(ptr %column5.val.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577075) + #dbg_value(i64 %len3.i.i.i, !553792, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577073) + #dbg_value(i64 %len3.i.i.i, !553786, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577075) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !577077 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !577078 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577079 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !577080, !noalias !577067, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !576785, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577081) + #dbg_value(i64 %_0.i5.i.i.i, !576785, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577081) + #dbg_value(i64 %_0.i.i.i.i, !577082, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577090) + #dbg_value(i64 %_0.i5.i.i.i, !577082, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577090) + #dbg_value(ptr poison, !576671, !DIExpression(), !577092) + #dbg_value(ptr poison, !576672, !DIExpression(), !577092) + #dbg_value(i64 %_0.i.i.i.i, !576673, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577092) + #dbg_value(i64 %_0.i5.i.i.i, !576673, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577092) + #dbg_value(i64 %_0.i.i.i.i, !576669, !DIExpression(), !577094) + #dbg_value(i64 %_0.i5.i.i.i, !576670, !DIExpression(), !577094) + #dbg_value(i64 %_0.i.i.i.i, !576660, !DIExpression(), !577095) + #dbg_value(i64 %_0.i5.i.i.i, !576661, !DIExpression(), !577095) + #dbg_value(i64 %_0.i.i.i.i, !576649, !DIExpression(), !577097) + #dbg_value(i64 %_0.i.i.i.i, !576644, !DIExpression(), !577099) + #dbg_value(i64 %_0.i5.i.i.i, !576650, !DIExpression(), !577097) + #dbg_value(i64 %_0.i5.i.i.i, !576645, !DIExpression(), !577099) + %_0.i.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !577101 + #dbg_value(i64 %_0.i.i.i.i, !576681, !DIExpression(), !577102) + #dbg_value(i64 %_0.i.i.i.i, !576683, !DIExpression(), !577104) + #dbg_value(i64 %_0.i5.i.i.i, !576682, !DIExpression(), !577102) + #dbg_value(i64 %_0.i5.i.i.i, !576684, !DIExpression(), !577104) + %_4.i1.i.i.i.i = sext i64 %_0.i.i.i.i to i128, !dbg !577105 + %_5.i.i.i.i.i = sext i64 %_0.i5.i.i.i to i128, !dbg !577106 + %wide.i.i.i.i.i = mul nsw i128 %_5.i.i.i.i.i, %_4.i1.i.i.i.i, !dbg !577105 + #dbg_value(i128 %wide.i.i.i.i.i, !576685, !DIExpression(), !577107) + %kept.i.i.i.i.i = trunc i128 %wide.i.i.i.i.i to i64, !dbg !577108 + #dbg_value(i64 %kept.i.i.i.i.i, !576687, !DIExpression(), !577109) + %_8.i.i.i.i.i = lshr i128 %wide.i.i.i.i.i, 64, !dbg !577110 + %discarded.i.i.i.i.i = trunc nuw i128 %_8.i.i.i.i.i to i64, !dbg !577111 + #dbg_value(i64 %discarded.i.i.i.i.i, !576689, !DIExpression(), !577112) + %_10.i.i.i.i.i = ashr i64 %kept.i.i.i.i.i, 63, !dbg !577113 + %_9.i.i.i.i.i = xor i64 %_10.i.i.i.i.i, %discarded.i.i.i.i.i, !dbg !577114 + #dbg_value(i64 poison, !576787, !DIExpression(), !577115) + #dbg_value(i64 %_9.i.i.i.i.i, !576789, !DIExpression(), !577115) + #dbg_value(ptr undef, !576390, !DIExpression(), !576799) + #dbg_value(i64 %_9.i.i.i.i.i, !576396, !DIExpression(), !576799) + %79 = or i64 %_9.i.i.i.i.i, %failed.sroa.0.011.i.i, !dbg !577116 + #dbg_value(i64 %79, !576779, !DIExpression(), !576834) + #dbg_value(i64 %_0.i.i.i.i.i, !576787, !DIExpression(), !577115) + #dbg_value(ptr %_4.sroa.10.0.i.i, !577056, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577057) + #dbg_value(i64 %index.i, !577056, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577057) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !577117 + #dbg_value(ptr %self4.i.i, !577118, !DIExpression(), !577122) + #dbg_value(i64 %_0.i.i.i.i.i, !577121, !DIExpression(), !577122) + store i64 %_0.i.i.i.i.i, ptr %self4.i.i, align 8, !dbg !577124, !alias.scope !576795, !noalias !577125 + #dbg_value(ptr undef, !576823, !DIExpression(), !576836) + #dbg_value(ptr undef, !576818, !DIExpression(), !576837) + #dbg_value(ptr undef, !576838, !DIExpression(), !576842) + #dbg_value(ptr poison, !576841, !DIExpression(), !576842) + %exitcond.not.i.i = icmp eq i64 %_36.i.i, %len3.i4.i.i.fr, !dbg !577126 + br i1 %exitcond.not.i.i, label %bb33.i, label %bb15.i.i, !dbg !576844 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md new file mode 100644 index 00000000000..4f95e7d0668 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-i64-mul-dense-s.md @@ -0,0 +1,35 @@ + + + +# `indexed-i64-mul-dense.s` + +```s + .p2align 4 +.LBB1685_25: + .loc 566 39 18 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp113564: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp113565: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp113566: + .loc 156 717 17 + incq %rsi +.Ltmp113567: + .loc 565 198 26 + sarq $63, %rax +.Ltmp113568: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp113569: + .loc 568 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp113570: + .loc 182 1904 50 + cmpq %rsi, %r9 + jne .LBB1685_25 + jmp .LBB1685_60 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md new file mode 100644 index 00000000000..4b254e7d98a --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-ll.md @@ -0,0 +1,322 @@ + + + +# `indexed-u64-mul-dense.ll` + +```ll +bb15.i.i: ; preds = %bb15.i.i, %bb15.i.i.preheader.new + %iter.sroa.0.012.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %116, %bb15.i.i ] + %niter = phi i64 [ 0, %bb15.i.i.preheader.new ], [ %niter.next.1, %bb15.i.i ] + #dbg_value(i64 %iter.sroa.0.012.i.i, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %failed.sroa.0.011.i.i, !580571, !DIExpression(), !580625) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580610, !DIExpression(), !580841) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580603, !DIExpression(), !580604) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580620, !DIExpression(), !580621) + %_36.i.i = or disjoint i64 %iter.sroa.0.012.i.i, 1, !dbg !580842 + #dbg_value(i64 %_36.i.i, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580575, !DIExpression(), !580843) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580597, !DIExpression(), !580598) + #dbg_value(i64 %iter.sroa.0.012.i.i, !580844, !DIExpression(), !580848) + #dbg_value(ptr undef, !563670, !DIExpression(), !580592) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563676, !DIExpression(), !580592) + #dbg_value(ptr poison, !563748, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563749, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563741, !DIExpression(), !580852) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580856 + %_0.i.i.i.i = load i64, ptr %_4.i.i.i.i, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(ptr poison, !563748, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563749, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563741, !DIExpression(), !580864) + #dbg_value(i64 %iter.sroa.0.012.i.i, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i = icmp ult i64 %iter.sroa.0.012.i.i, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i), !dbg !580869 + %_4.i4.i.i.i = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580870 + %_0.i5.i.i.i = load i64, ptr %_4.i4.i.i.i, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(ptr poison, !580456, !DIExpression(), !580883) + #dbg_value(ptr poison, !580457, !DIExpression(), !580883) + #dbg_value(i64 %_0.i.i.i.i, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i = mul i64 %_0.i5.i.i.i, %_0.i.i.i.i, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i = zext i64 %_0.i.i.i.i to i128, !dbg !580896 + %_6.i.i.i.i.i = zext i64 %_0.i5.i.i.i to i128, !dbg !580897 + %_4.i1.i.i.i.i = mul nuw i128 %_6.i.i.i.i.i, %_5.i.i.i.i.i, !dbg !580898 + %_3.i2.i.i.i.i = lshr i128 %_4.i1.i.i.i.i, 64, !dbg !580899 + %_0.i.i.i.i.i = trunc nuw i128 %_3.i2.i.i.i.i to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i, !580581, !DIExpression(), !580901) + #dbg_value(ptr undef, !576390, !DIExpression(), !580590) + #dbg_value(i64 %_0.i.i.i.i.i, !576396, !DIExpression(), !580590) + %115 = or i64 %failed.sroa.0.011.i.i, %_0.i.i.i.i.i, !dbg !580902 + #dbg_value(i64 %115, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i, !dbg !580903 + #dbg_value(ptr %self4.i.i, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i, ptr %self4.i.i, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + #dbg_value(ptr undef, !580614, !DIExpression(), !580627) + #dbg_value(ptr undef, !580609, !DIExpression(), !580628) + #dbg_value(ptr undef, !580629, !DIExpression(), !580633) + #dbg_value(ptr poison, !580632, !DIExpression(), !580633) + #dbg_value(i64 %_36.i.i, !580610, !DIExpression(), !580841) + #dbg_value(i64 %_36.i.i, !580603, !DIExpression(), !580604) + #dbg_value(i64 %_36.i.i, !580620, !DIExpression(), !580621) + %_36.i.i.1 = add nuw i64 %iter.sroa.0.012.i.i, 2, !dbg !580842 + #dbg_value(i64 %_36.i.i.1, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %_36.i.i, !580575, !DIExpression(), !580843) + #dbg_value(i64 %_36.i.i, !580597, !DIExpression(), !580598) + #dbg_value(i64 %_36.i.i, !580844, !DIExpression(), !580848) + #dbg_value(i64 %_36.i.i, !563676, !DIExpression(), !580592) + #dbg_value(i64 %_36.i.i, !563749, !DIExpression(), !580850) + #dbg_value(i64 %_36.i.i, !563741, !DIExpression(), !580852) + #dbg_value(i64 %_36.i.i, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_36.i.i, !dbg !580856 + %_0.i.i.i.i.1 = load i64, ptr %_4.i.i.i.i.1, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(i64 %_36.i.i, !563749, !DIExpression(), !580862) + #dbg_value(i64 %_36.i.i, !563741, !DIExpression(), !580864) + #dbg_value(i64 %_36.i.i, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i.1 = icmp ult i64 %_36.i.i, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.1), !dbg !580869 + %_4.i4.i.i.i.1 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_36.i.i, !dbg !580870 + %_0.i5.i.i.i.1 = load i64, ptr %_4.i4.i.i.i.1, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.1, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i.1, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i.1, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i.1, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(i64 %_0.i.i.i.i.1, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i.1, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i.1, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i.1, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i.1, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i.1, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i.1, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i.1, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i.1, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i.1, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i.1 = mul i64 %_0.i5.i.i.i.1, %_0.i.i.i.i.1, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i.1, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i.1, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i.1, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i.1, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i.1 = zext i64 %_0.i.i.i.i.1 to i128, !dbg !580896 + %_6.i.i.i.i.i.1 = zext i64 %_0.i5.i.i.i.1 to i128, !dbg !580897 + %_4.i1.i.i.i.i.1 = mul nuw i128 %_6.i.i.i.i.i.1, %_5.i.i.i.i.i.1, !dbg !580898 + %_3.i2.i.i.i.i.1 = lshr i128 %_4.i1.i.i.i.i.1, 64, !dbg !580899 + %_0.i.i.i.i.i.1 = trunc nuw i128 %_3.i2.i.i.i.i.1 to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.1, !580581, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.1, !576396, !DIExpression(), !580590) + %116 = or i64 %115, %_0.i.i.i.i.i.1, !dbg !580902 + #dbg_value(i64 %116, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_36.i.i, !dbg !580903 + #dbg_value(ptr %self4.i.i.1, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i.1, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i.1, ptr %self4.i.i.1, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + %niter.next.1 = add i64 %niter, 2, !dbg !580635 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !580635 + br i1 %niter.ncmp.1, label %bb33.i.loopexit144.unr-lcssa, label %bb15.i.i, !dbg !580635 + +bb33.thread.i: ; preds = %bb26.preheader.i.thread, %bb11.i, %bb26.preheader.i + #dbg_value(i64 0, !580128, !DIExpression(), !580912) + #dbg_value(i64 %index.i, !580121, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !580349) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !580913, !noalias !580287 + #dbg_value(i64 0, !580099, !DIExpression(), !580915) + #dbg_declare(ptr poison, !580103, !DIExpression(), !580916) + #dbg_declare(ptr %value.i.i, !580917, !DIExpression(), !580920) + #dbg_value(ptr undef, !577131, !DIExpression(), !580923) + #dbg_value(ptr undef, !577132, !DIExpression(), !580923) + br label %bb36.i, !dbg !580924 + +bb33.i.loopexit.unr-lcssa: ; preds = %bb27.us.i.us, %bb27.us.i.us.preheader + %.lcssa.ph = phi i64 [ poison, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %iter.sroa.0.046.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %_157.us.i.us, %bb27.us.i.us ] + %accumulated.sroa.0.045.us.i.us.unr = phi i64 [ 0, %bb27.us.i.us.preheader ], [ %88, %bb27.us.i.us ] + %lcmp.mod148.not = icmp eq i64 %xtraiter147, 0, !dbg !580418 + br i1 %lcmp.mod148.not, label %bb33.i, label %bb27.us.i.us.epil, !dbg !580418 + +bb27.us.i.us.epil: ; preds = %bb33.i.loopexit.unr-lcssa + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + #dbg_value(i64 %accumulated.sroa.0.045.us.i.us.unr, !580145, !DIExpression(), !580415) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !580149, !DIExpression(), !580491) + #dbg_value(ptr %columns.i, !563475, !DIExpression(), !580492) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563476, !DIExpression(), !580492) + #dbg_value(ptr %columns.i, !563465, !DIExpression(), !580493) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563466, !DIExpression(), !580493) + #dbg_value(ptr %columns.i, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580496) + #dbg_value(i64 0, !563341, !DIExpression(), !580494) + #dbg_value(ptr %columns.i, !563468, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580525) + #dbg_value(ptr %columns.i, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580494) + %_0.sroa.0.0.i.i.us.i.us.epil = load i64, ptr %data.i.i.i.us.i, align 8, !dbg !580423, !noalias !580207, !noundef !23 + #dbg_value(ptr %14, !563465, !DIExpression(), !580498) + #dbg_value(i64 %iter.sroa.0.046.us.i.us.unr, !563466, !DIExpression(), !580498) + #dbg_value(ptr %14, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580499) + #dbg_value(ptr %14, !563336, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580501) + #dbg_value(ptr %14, !563468, !DIExpression(DW_OP_plus_uconst, 8, DW_OP_stack_value), !580502) + #dbg_value(i64 0, !563341, !DIExpression(), !580499) + %_0.sroa.0.0.i9.i.us.i.us.epil = load i64, ptr %data.i6.i7.i.us.i, align 8, !dbg !580428, !noalias !580207, !noundef !23 + #dbg_value(ptr poison, !580456, !DIExpression(), !580503) + #dbg_value(ptr poison, !580457, !DIExpression(), !580503) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580503) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580503) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580454, !DIExpression(), !580504) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580455, !DIExpression(), !580504) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580445, !DIExpression(), !580505) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580446, !DIExpression(), !580505) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580438, !DIExpression(), !580506) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580433, !DIExpression(), !580507) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580439, !DIExpression(), !580506) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580434, !DIExpression(), !580507) + %_0.i3.i.us.i.us.epil = mul i64 %_0.sroa.0.0.i9.i.us.i.us.epil, %_0.sroa.0.0.i.i.us.i.us.epil, !dbg !580430 + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580464, !DIExpression(), !580508) + #dbg_value(i64 %_0.sroa.0.0.i.i.us.i.us.epil, !580466, !DIExpression(), !580509) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580465, !DIExpression(), !580508) + #dbg_value(i64 %_0.sroa.0.0.i9.i.us.i.us.epil, !580467, !DIExpression(), !580509) + %_5.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i.i.us.i.us.epil to i128, !dbg !580460 + %_6.i.i.us.i.us.epil = zext i64 %_0.sroa.0.0.i9.i.us.i.us.epil to i128, !dbg !580469 + %_4.i1.i.us.i.us.epil = mul nuw i128 %_6.i.i.us.i.us.epil, %_5.i.i.us.i.us.epil, !dbg !580470 + %_3.i2.i.us.i.us.epil = lshr i128 %_4.i1.i.us.i.us.epil, 64, !dbg !580471 + %_0.i.i159.us.i.us.epil = trunc nuw i128 %_3.i2.i.us.i.us.epil to i64, !dbg !580472 + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !580151, !DIExpression(), !580510) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !580153, !DIExpression(), !580510) + #dbg_value(ptr undef, !576390, !DIExpression(), !580180) + #dbg_value(i64 %_0.i.i159.us.i.us.epil, !576396, !DIExpression(), !580180) + %117 = or i64 %accumulated.sroa.0.045.us.i.us.unr, %_0.i.i159.us.i.us.epil, !dbg !580473 + #dbg_value(i64 %117, !580145, !DIExpression(), !580415) + %self34.us.i.us.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.046.us.i.us.unr, !dbg !580474 + #dbg_value(ptr %self34.us.i.us.epil, !580478, !DIExpression(), !580511) + #dbg_value(i64 %_0.i3.i.us.i.us.epil, !580479, !DIExpression(), !580511) + store i64 %_0.i3.i.us.i.us.epil, ptr %self34.us.i.us.epil, align 8, !dbg !580475, !noalias !580207 + #dbg_value(i64 poison, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + #dbg_value(ptr undef, !580170, !DIExpression(), !580173) + #dbg_value(ptr undef, !580161, !DIExpression(), !580166) + #dbg_value(ptr undef, !580174, !DIExpression(), !580178) + #dbg_value(ptr poison, !580177, !DIExpression(), !580178) + #dbg_value(i64 poison, !580147, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580416) + br label %bb33.i, !dbg !580925 + +bb33.i.loopexit144.unr-lcssa: ; preds = %bb15.i.i, %bb15.i.i.preheader + %.lcssa145.ph = phi i64 [ poison, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %iter.sroa.0.012.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %_36.i.i.1, %bb15.i.i ] + %failed.sroa.0.011.i.i.unr = phi i64 [ 0, %bb15.i.i.preheader ], [ %116, %bb15.i.i ] + %lcmp.mod.not = icmp eq i64 %xtraiter, 0, !dbg !580635 + br i1 %lcmp.mod.not, label %bb33.i, label %bb15.i.i.epil, !dbg !580635 + +bb15.i.i.epil: ; preds = %bb33.i.loopexit144.unr-lcssa + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580573, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %failed.sroa.0.011.i.i.unr, !580571, !DIExpression(), !580625) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580610, !DIExpression(), !580841) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580603, !DIExpression(), !580604) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580620, !DIExpression(), !580621) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580573, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 64), !580626) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580575, !DIExpression(), !580843) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580597, !DIExpression(), !580598) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !580844, !DIExpression(), !580848) + #dbg_value(ptr undef, !563670, !DIExpression(), !580592) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563676, !DIExpression(), !580592) + #dbg_value(ptr poison, !563748, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563749, !DIExpression(), !580850) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563741, !DIExpression(), !580852) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563733, !DIExpression(), !580854) + #dbg_value(ptr %column.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580852) + #dbg_value(ptr %column.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580854) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580852) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580854) + %_4.i.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580856 + %_0.i.i.i.i.epil = load i64, ptr %_4.i.i.i.i.epil, align 8, !dbg !580857, !noalias !580858, !noundef !23 + #dbg_value(ptr poison, !563748, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563749, !DIExpression(), !580862) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563741, !DIExpression(), !580864) + #dbg_value(i64 %iter.sroa.0.012.i.i.unr, !563733, !DIExpression(), !580866) + #dbg_value(ptr %column5.val.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580864) + #dbg_value(ptr %column5.val.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580866) + #dbg_value(i64 %len3.i.i.i, !563740, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580864) + #dbg_value(i64 %len3.i.i.i, !563734, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580866) + %_5.i3.i.i.i.epil = icmp ult i64 %iter.sroa.0.012.i.i.unr, %len3.i.i.i, !dbg !580868 + tail call void @llvm.assume(i1 %_5.i3.i.i.i.epil), !dbg !580869 + %_4.i4.i.i.i.epil = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580870 + %_0.i5.i.i.i.epil = load i64, ptr %_4.i4.i.i.i.epil, align 8, !dbg !580871, !noalias !580858, !noundef !23 + #dbg_value(i64 %_0.i.i.i.i.epil, !580577, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580872) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580577, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580872) + #dbg_value(i64 %_0.i.i.i.i.epil, !580873, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580881) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580873, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580881) + #dbg_value(ptr poison, !580456, !DIExpression(), !580883) + #dbg_value(ptr poison, !580457, !DIExpression(), !580883) + #dbg_value(i64 %_0.i.i.i.i.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580883) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580458, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580883) + #dbg_value(i64 %_0.i.i.i.i.epil, !580454, !DIExpression(), !580885) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580455, !DIExpression(), !580885) + #dbg_value(i64 %_0.i.i.i.i.epil, !580445, !DIExpression(), !580886) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580446, !DIExpression(), !580886) + #dbg_value(i64 %_0.i.i.i.i.epil, !580438, !DIExpression(), !580888) + #dbg_value(i64 %_0.i.i.i.i.epil, !580433, !DIExpression(), !580890) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580439, !DIExpression(), !580888) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580434, !DIExpression(), !580890) + %_0.i3.i.i.i.i.epil = mul i64 %_0.i5.i.i.i.epil, %_0.i.i.i.i.epil, !dbg !580892 + #dbg_value(i64 %_0.i.i.i.i.epil, !580464, !DIExpression(), !580893) + #dbg_value(i64 %_0.i.i.i.i.epil, !580466, !DIExpression(), !580895) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580465, !DIExpression(), !580893) + #dbg_value(i64 %_0.i5.i.i.i.epil, !580467, !DIExpression(), !580895) + %_5.i.i.i.i.i.epil = zext i64 %_0.i.i.i.i.epil to i128, !dbg !580896 + %_6.i.i.i.i.i.epil = zext i64 %_0.i5.i.i.i.epil to i128, !dbg !580897 + %_4.i1.i.i.i.i.epil = mul nuw i128 %_6.i.i.i.i.i.epil, %_5.i.i.i.i.i.epil, !dbg !580898 + %_3.i2.i.i.i.i.epil = lshr i128 %_4.i1.i.i.i.i.epil, 64, !dbg !580899 + %_0.i.i.i.i.i.epil = trunc nuw i128 %_3.i2.i.i.i.i.epil to i64, !dbg !580900 + #dbg_value(i64 poison, !580579, !DIExpression(), !580901) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !580581, !DIExpression(), !580901) + #dbg_value(ptr undef, !576390, !DIExpression(), !580590) + #dbg_value(i64 %_0.i.i.i.i.i.epil, !576396, !DIExpression(), !580590) + %118 = or i64 %failed.sroa.0.011.i.i.unr, %_0.i.i.i.i.i.epil, !dbg !580902 + #dbg_value(i64 %118, !580571, !DIExpression(), !580625) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !580579, !DIExpression(), !580901) + #dbg_value(ptr %_4.sroa.10.0.i.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !580848) + #dbg_value(i64 %index.i, !580847, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !580848) + %self4.i.i.epil = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.012.i.i.unr, !dbg !580903 + #dbg_value(ptr %self4.i.i.epil, !580904, !DIExpression(), !580908) + #dbg_value(i64 %_0.i3.i.i.i.i.epil, !580907, !DIExpression(), !580908) + store i64 %_0.i3.i.i.i.i.epil, ptr %self4.i.i.epil, align 8, !dbg !580910, !alias.scope !580586, !noalias !580911 + #dbg_value(ptr undef, !580614, !DIExpression(), !580627) + #dbg_value(ptr undef, !580609, !DIExpression(), !580628) + #dbg_value(ptr undef, !580629, !DIExpression(), !580633) + #dbg_value(ptr poison, !580632, !DIExpression(), !580633) + br label %bb33.i, !dbg !580925 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md new file mode 100644 index 00000000000..f0981969a0e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/indexed-u64-mul-dense-s.md @@ -0,0 +1,71 @@ + + + +# `indexed-u64-mul-dense.s` + +```s +.LBB1688_67: + .loc 181 765 12 + andq $-2, %r10 + xorl %edi, %edi + xorl %ecx, %ecx + movq -48(%rbp), %r8 +.Ltmp114798: +.LBB1688_68: + .loc 566 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp114799: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp114800: + movq %rdx, %rsi +.Ltmp114801: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp114802: + .loc 566 39 18 + movq 8(%r13,%rdi,8), %rax +.Ltmp114803: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp114804: + .loc 568 821 53 + orq %rcx, %rsi +.Ltmp114805: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp114806: + .loc 156 717 17 + addq $2, %rdi +.Ltmp114807: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp114808: + .loc 568 821 53 + orq %rsi, %rcx +.Ltmp114809: + .loc 181 765 12 + cmpq %r10, %rdi + jne .LBB1688_68 +.Ltmp114810: +.LBB1688_69: + testb $1, %r9b + je .LBB1688_84 +.Ltmp114811: + .loc 566 39 18 + movq (%r13,%rdi,8), %rax +.Ltmp114812: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp114813: + .loc 568 821 53 + orq %rdx, %rcx +.Ltmp114814: + .loc 568 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp114815: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp114816: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md new file mode 100644 index 00000000000..c4a917571df --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-ll.md @@ -0,0 +1,84 @@ + + + +# `owned-i64-mul-dense.ll` + +```ll +terminate.i: ; preds = %bb57.i + %78 = landingpad { ptr, i32 } + filter [0 x ptr] zeroinitializer +; call core::panicking::panic_in_cleanup + call void @_ZN4core9panicking16panic_in_cleanup17h8f68387bb6cbbf54E() #88, !dbg !573712, !noalias !573567 + unreachable, !dbg !573712 + +bb18.i: ; preds = %bb18.i, %bb18.lr.ph.i + %_15552.i = phi i64 [ 1, %bb18.lr.ph.i ], [ %_155.i, %bb18.i ] + %iter.sroa.0.051.i = phi i64 [ 0, %bb18.lr.ph.i ], [ %_15552.i, %bb18.i ] + %failed.sroa.0.050.i = phi i64 [ 0, %bb18.lr.ph.i ], [ %81, %bb18.i ] + #dbg_value(i64 %iter.sroa.0.051.i, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + #dbg_value(i64 %failed.sroa.0.050.i, !573466, !DIExpression(), !573746) + #dbg_value(i64 %iter.sroa.0.051.i, !573479, !DIExpression(), !574113) + #dbg_value(ptr undef, !552190, !DIExpression(), !573546) + #dbg_value(i64 %iter.sroa.0.051.i, !552196, !DIExpression(), !573546) + #dbg_value(ptr poison, !552716, !DIExpression(), !574114) + #dbg_value(i64 %iter.sroa.0.051.i, !552717, !DIExpression(), !574114) + #dbg_value(ptr poison, !552716, !DIExpression(), !574116) + #dbg_value(i64 %iter.sroa.0.051.i, !552717, !DIExpression(), !574116) + %79 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.051.i, !dbg !574118 + %_0.i.i97.i = load i64, ptr %79, align 8, !dbg !574118, !noalias !574119, !noundef !23 + %80 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.051.i, !dbg !574122 + %_0.i5.i.i = load i64, ptr %80, align 8, !dbg !574122, !noalias !574119, !noundef !23 + #dbg_value(ptr poison, !573820, !DIExpression(), !574123) + #dbg_value(ptr poison, !573821, !DIExpression(), !574123) + #dbg_value(i64 %_0.i.i97.i, !573822, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !574123) + #dbg_value(i64 %_0.i5.i.i, !573822, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !574123) + #dbg_value(i64 %_0.i.i97.i, !573818, !DIExpression(), !574125) + #dbg_value(i64 %_0.i5.i.i, !573819, !DIExpression(), !574125) + #dbg_value(i64 %_0.i.i97.i, !573809, !DIExpression(), !574126) + #dbg_value(i64 %_0.i5.i.i, !573810, !DIExpression(), !574126) + #dbg_value(i64 %_0.i.i97.i, !573798, !DIExpression(), !574128) + #dbg_value(i64 %_0.i.i97.i, !573793, !DIExpression(), !574130) + #dbg_value(i64 %_0.i5.i.i, !573799, !DIExpression(), !574128) + #dbg_value(i64 %_0.i5.i.i, !573794, !DIExpression(), !574130) + %_0.i.i111.i = mul i64 %_0.i5.i.i, %_0.i.i97.i, !dbg !574132 + #dbg_value(i64 %_0.i.i97.i, !573830, !DIExpression(), !574133) + #dbg_value(i64 %_0.i.i97.i, !573832, !DIExpression(), !574135) + #dbg_value(i64 %_0.i5.i.i, !573831, !DIExpression(), !574133) + #dbg_value(i64 %_0.i5.i.i, !573833, !DIExpression(), !574135) + %_4.i1.i.i = sext i64 %_0.i.i97.i to i128, !dbg !574136 + %_5.i.i.i = sext i64 %_0.i5.i.i to i128, !dbg !574137 + %wide.i.i.i = mul nsw i128 %_5.i.i.i, %_4.i1.i.i, !dbg !574136 + #dbg_value(i128 %wide.i.i.i, !573834, !DIExpression(), !574138) + %kept.i.i.i = trunc i128 %wide.i.i.i to i64, !dbg !574139 + #dbg_value(i64 %kept.i.i.i, !573836, !DIExpression(), !574140) + %_8.i.i.i = lshr i128 %wide.i.i.i, 64, !dbg !574141 + %discarded.i.i.i = trunc nuw i128 %_8.i.i.i to i64, !dbg !574142 + #dbg_value(i64 %discarded.i.i.i, !573838, !DIExpression(), !574143) + %_10.i.i.i = ashr i64 %kept.i.i.i, 63, !dbg !574144 + %_9.i.i.i = xor i64 %_10.i.i.i, %discarded.i.i.i, !dbg !574145 + #dbg_value(i64 %_0.i.i111.i, !573481, !DIExpression(), !574146) + #dbg_value(i64 %_9.i.i.i, !573483, !DIExpression(), !574146) + #dbg_value(ptr undef, !573548, !DIExpression(), !573557) + #dbg_value(i64 %_9.i.i.i, !573554, !DIExpression(), !573557) + %81 = or i64 %_9.i.i.i, %failed.sroa.0.050.i, !dbg !574147 + #dbg_value(i64 %81, !573466, !DIExpression(), !573746) + %self32.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.051.i, !dbg !574148 + #dbg_value(ptr %self32.i, !573852, !DIExpression(), !574149) + #dbg_value(i64 %_0.i.i111.i, !573853, !DIExpression(), !574149) + store i64 %_0.i.i111.i, ptr %self32.i, align 8, !dbg !574151, !noalias !573567 + #dbg_value(i64 %_15552.i, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + #dbg_value(ptr undef, !573517, !DIExpression(), !573539) + #dbg_value(ptr undef, !573505, !DIExpression(), !573535) + #dbg_value(ptr undef, !573521, !DIExpression(), !573540) + #dbg_value(ptr poison, !573524, !DIExpression(), !573540) + %_155.i = add i64 %_15552.i, 1, !dbg !574152 + #dbg_value(i64 poison, !573477, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !573890) + %exitcond.not.i = icmp eq i64 %_15552.i, %len3.i4.i.i.fr, !dbg !574153 + br i1 %exitcond.not.i, label %bb38.i, label %bb18.i, !dbg !573891 + +bb38.thread.i: ; preds = %bb31.preheader.i.thread, %bb17.preheader.split.i, %bb31.preheader.i + #dbg_value(i64 0, !573466, !DIExpression(), !573746) + #dbg_value(i64 %index.i, !573459, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !573709) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !574154, !noalias !573647 + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md new file mode 100644 index 00000000000..db24d697b4e --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-i64-mul-dense-s.md @@ -0,0 +1,46 @@ + + + +# `owned-i64-mul-dense.s` + +```s + .loc 562 112 26 is_stmt 1 + je .LBB1685_70 +.Ltmp114190: + .loc 562 0 26 is_stmt 0 + movq -128(%rbp), %r13 +.Ltmp114191: + xorl %esi, %esi +.Ltmp114192: + xorl %ecx, %ecx + movq -48(%rbp), %rdi +.Ltmp114193: + .p2align 4 +.LBB1685_25: + .loc 564 62 9 is_stmt 1 + movq (%r13,%rsi,8), %rax +.Ltmp114194: + .loc 565 194 24 + imulq (%r12,%rsi,8) +.Ltmp114195: + .loc 207 475 9 + movq %rax, (%rdi,%rsi,8) +.Ltmp114196: + .loc 565 198 26 + sarq $63, %rax +.Ltmp114197: + .loc 565 198 13 is_stmt 0 + xorq %rdx, %rax +.Ltmp114198: + .loc 566 821 53 is_stmt 1 + orq %rax, %rcx +.Ltmp114199: + .loc 182 1904 50 + incq %rsi +.Ltmp114200: + cmpq %rsi, %r9 + jne .LBB1685_25 + jmp .LBB1685_60 +.Ltmp114201: + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md new file mode 100644 index 00000000000..0659a92c119 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-ll.md @@ -0,0 +1,121 @@ + + + +# `owned-u64-mul-dense.ll` + +```ll +bb18.i: ; preds = %bb18.i, %bb18.lr.ph.i.new + %_15552.i = phi i64 [ 1, %bb18.lr.ph.i.new ], [ %_155.i.1, %bb18.i ] + %iter.sroa.0.051.i = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %_155.i, %bb18.i ] + %failed.sroa.0.050.i = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %120, %bb18.i ] + %niter = phi i64 [ 0, %bb18.lr.ph.i.new ], [ %niter.next.1, %bb18.i ] + #dbg_value(i64 %iter.sroa.0.051.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %failed.sroa.0.050.i, !576953, !DIExpression(), !577231) + #dbg_value(i64 %iter.sroa.0.051.i, !576966, !DIExpression(), !577601) + #dbg_value(ptr undef, !561311, !DIExpression(), !577032) + #dbg_value(i64 %iter.sroa.0.051.i, !561317, !DIExpression(), !577032) + #dbg_value(ptr poison, !561836, !DIExpression(), !577602) + #dbg_value(i64 %iter.sroa.0.051.i, !561837, !DIExpression(), !577602) + #dbg_value(ptr poison, !561836, !DIExpression(), !577604) + #dbg_value(i64 %iter.sroa.0.051.i, !561837, !DIExpression(), !577604) + %115 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %iter.sroa.0.051.i, !dbg !577606 + %_0.i.i92.i = load i64, ptr %115, align 8, !dbg !577606, !noalias !577607, !noundef !23 + %116 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %iter.sroa.0.051.i, !dbg !577610 + %_0.i5.i.i = load i64, ptr %116, align 8, !dbg !577610, !noalias !577607, !noundef !23 + #dbg_value(ptr poison, !577301, !DIExpression(), !577611) + #dbg_value(ptr poison, !577302, !DIExpression(), !577611) + #dbg_value(i64 %_0.i.i92.i, !577303, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577611) + #dbg_value(i64 %_0.i5.i.i, !577303, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577611) + #dbg_value(i64 %_0.i.i92.i, !577299, !DIExpression(), !577613) + #dbg_value(i64 %_0.i5.i.i, !577300, !DIExpression(), !577613) + #dbg_value(i64 %_0.i.i92.i, !577290, !DIExpression(), !577614) + #dbg_value(i64 %_0.i5.i.i, !577291, !DIExpression(), !577614) + #dbg_value(i64 %_0.i.i92.i, !577283, !DIExpression(), !577616) + #dbg_value(i64 %_0.i.i92.i, !577278, !DIExpression(), !577618) + #dbg_value(i64 %_0.i5.i.i, !577284, !DIExpression(), !577616) + #dbg_value(i64 %_0.i5.i.i, !577279, !DIExpression(), !577618) + %_0.i3.i.i = mul i64 %_0.i5.i.i, %_0.i.i92.i, !dbg !577620 + #dbg_value(i64 %_0.i.i92.i, !577309, !DIExpression(), !577621) + #dbg_value(i64 %_0.i.i92.i, !577311, !DIExpression(), !577623) + #dbg_value(i64 %_0.i5.i.i, !577310, !DIExpression(), !577621) + #dbg_value(i64 %_0.i5.i.i, !577312, !DIExpression(), !577623) + %_5.i.i.i = zext i64 %_0.i.i92.i to i128, !dbg !577624 + %_6.i.i.i = zext i64 %_0.i5.i.i to i128, !dbg !577625 + %_4.i1.i.i = mul nuw i128 %_6.i.i.i, %_5.i.i.i, !dbg !577626 + %_3.i2.i.i = lshr i128 %_4.i1.i.i, 64, !dbg !577627 + %_0.i.i106.i = trunc nuw i128 %_3.i2.i.i to i64, !dbg !577628 + #dbg_value(i64 %_0.i3.i.i, !576968, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i, !576970, !DIExpression(), !577629) + #dbg_value(ptr undef, !573548, !DIExpression(), !577036) + #dbg_value(i64 %_0.i.i106.i, !573554, !DIExpression(), !577036) + %117 = or i64 %failed.sroa.0.050.i, %_0.i.i106.i, !dbg !577630 + #dbg_value(i64 %117, !576953, !DIExpression(), !577231) + %self32.i = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %iter.sroa.0.051.i, !dbg !577631 + #dbg_value(ptr %self32.i, !577323, !DIExpression(), !577632) + #dbg_value(i64 %_0.i3.i.i, !577324, !DIExpression(), !577632) + store i64 %_0.i3.i.i, ptr %self32.i, align 8, !dbg !577634, !noalias !577052 + #dbg_value(i64 %_15552.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(ptr undef, !577003, !DIExpression(), !577025) + #dbg_value(ptr undef, !576991, !DIExpression(), !577021) + #dbg_value(ptr undef, !577007, !DIExpression(), !577026) + #dbg_value(ptr poison, !577010, !DIExpression(), !577026) + %_155.i = add i64 %_15552.i, 1, !dbg !577635 + #dbg_value(i64 poison, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %_15552.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + #dbg_value(i64 %_15552.i, !576966, !DIExpression(), !577601) + #dbg_value(i64 %_15552.i, !561317, !DIExpression(), !577032) + #dbg_value(i64 %_15552.i, !561837, !DIExpression(), !577602) + #dbg_value(i64 %_15552.i, !561837, !DIExpression(), !577604) + %118 = getelementptr inbounds nuw i64, ptr %column.val.i.i, i64 %_15552.i, !dbg !577606 + %_0.i.i92.i.1 = load i64, ptr %118, align 8, !dbg !577606, !noalias !577607, !noundef !23 + %119 = getelementptr inbounds nuw i64, ptr %column5.val.i.i, i64 %_15552.i, !dbg !577610 + %_0.i5.i.i.1 = load i64, ptr %119, align 8, !dbg !577610, !noalias !577607, !noundef !23 + #dbg_value(i64 %_0.i.i92.i.1, !577303, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577611) + #dbg_value(i64 %_0.i5.i.i.1, !577303, !DIExpression(DW_OP_LLVM_fragment, 64, 64), !577611) + #dbg_value(i64 %_0.i.i92.i.1, !577299, !DIExpression(), !577613) + #dbg_value(i64 %_0.i5.i.i.1, !577300, !DIExpression(), !577613) + #dbg_value(i64 %_0.i.i92.i.1, !577290, !DIExpression(), !577614) + #dbg_value(i64 %_0.i5.i.i.1, !577291, !DIExpression(), !577614) + #dbg_value(i64 %_0.i.i92.i.1, !577283, !DIExpression(), !577616) + #dbg_value(i64 %_0.i.i92.i.1, !577278, !DIExpression(), !577618) + #dbg_value(i64 %_0.i5.i.i.1, !577284, !DIExpression(), !577616) + #dbg_value(i64 %_0.i5.i.i.1, !577279, !DIExpression(), !577618) + %_0.i3.i.i.1 = mul i64 %_0.i5.i.i.1, %_0.i.i92.i.1, !dbg !577620 + #dbg_value(i64 %_0.i.i92.i.1, !577309, !DIExpression(), !577621) + #dbg_value(i64 %_0.i.i92.i.1, !577311, !DIExpression(), !577623) + #dbg_value(i64 %_0.i5.i.i.1, !577310, !DIExpression(), !577621) + #dbg_value(i64 %_0.i5.i.i.1, !577312, !DIExpression(), !577623) + %_5.i.i.i.1 = zext i64 %_0.i.i92.i.1 to i128, !dbg !577624 + %_6.i.i.i.1 = zext i64 %_0.i5.i.i.1 to i128, !dbg !577625 + %_4.i1.i.i.1 = mul nuw i128 %_6.i.i.i.1, %_5.i.i.i.1, !dbg !577626 + %_3.i2.i.i.1 = lshr i128 %_4.i1.i.i.1, 64, !dbg !577627 + %_0.i.i106.i.1 = trunc nuw i128 %_3.i2.i.i.1 to i64, !dbg !577628 + #dbg_value(i64 %_0.i3.i.i.1, !576968, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i.1, !576970, !DIExpression(), !577629) + #dbg_value(i64 %_0.i.i106.i.1, !573554, !DIExpression(), !577036) + %120 = or i64 %117, %_0.i.i106.i.1, !dbg !577630 + #dbg_value(i64 %120, !576953, !DIExpression(), !577231) + %self32.i.1 = getelementptr inbounds nuw i64, ptr %_4.sroa.10.0.i.i, i64 %_15552.i, !dbg !577631 + #dbg_value(ptr %self32.i.1, !577323, !DIExpression(), !577632) + #dbg_value(i64 %_0.i3.i.i.1, !577324, !DIExpression(), !577632) + store i64 %_0.i3.i.i.1, ptr %self32.i.1, align 8, !dbg !577634, !noalias !577052 + #dbg_value(i64 %_155.i, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + %_155.i.1 = add i64 %_15552.i, 2, !dbg !577635 + #dbg_value(i64 poison, !576964, !DIExpression(DW_OP_LLVM_fragment, 0, 64), !577378) + %niter.next.1 = add i64 %niter, 2, !dbg !577379 + %niter.ncmp.1 = icmp eq i64 %niter.next.1, %unroll_iter, !dbg !577379 + br i1 %niter.ncmp.1, label %bb38.i.loopexit144.unr-lcssa, label %bb18.i, !dbg !577379 + +bb38.thread.i: ; preds = %bb31.preheader.i.thread, %bb17.preheader.split.i, %bb31.preheader.i + #dbg_value(i64 0, !576953, !DIExpression(), !577231) + #dbg_value(i64 %index.i, !576946, !DIExpression(DW_OP_LLVM_fragment, 128, 64), !577194) + call void @llvm.lifetime.start.p0(i64 24, ptr nonnull %value.i.i), !dbg !577636, !noalias !577132 + #dbg_value(i64 0, !576924, !DIExpression(), !577638) + #dbg_declare(ptr poison, !576928, !DIExpression(), !577639) + #dbg_declare(ptr %value.i.i, !577640, !DIExpression(), !577643) + #dbg_value(ptr undef, !574157, !DIExpression(), !577646) + #dbg_value(ptr undef, !574158, !DIExpression(), !577646) + br label %bb41.i, !dbg !577647 + + +``` diff --git a/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md new file mode 100644 index 00000000000..47957a445c2 --- /dev/null +++ b/research/rowfn-x86-2026-08-07/codegen/owned-u64-mul-dense-s.md @@ -0,0 +1,72 @@ + + + +# `owned-u64-mul-dense.s` + +```s +.LBB1688_67: + .loc 562 112 26 + andq $-2, %r10 + xorl %ecx, %ecx + xorl %edi, %edi + movq -48(%rbp), %r8 +.Ltmp115437: +.LBB1688_68: + .loc 564 62 9 + movq (%r13,%rdi,8), %rax +.Ltmp115438: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp115439: + movq %rdx, %rsi +.Ltmp115440: + .loc 207 475 9 + movq %rax, (%r8,%rdi,8) +.Ltmp115441: + .loc 564 62 9 + movq 8(%r13,%rdi,8), %rax +.Ltmp115442: + .loc 565 176 44 + mulq 8(%r12,%rdi,8) +.Ltmp115443: + .loc 566 821 53 + orq %rcx, %rsi +.Ltmp115444: + .loc 565 176 44 + movq %rdx, %rcx +.Ltmp115445: + .loc 566 821 53 + orq %rsi, %rcx +.Ltmp115446: + .loc 207 475 9 + movq %rax, 8(%r8,%rdi,8) +.Ltmp115447: + .loc 562 112 26 + addq $2, %rdi + cmpq %rdi, %r10 + jne .LBB1688_68 +.Ltmp115448: +.LBB1688_69: + testb $1, %r9b + je .LBB1688_84 +.Ltmp115449: + .loc 564 62 9 + movq (%r13,%rdi,8), %rax +.Ltmp115450: + .loc 565 176 44 + mulq (%r12,%rdi,8) +.Ltmp115451: + .loc 566 821 53 + orq %rdx, %rcx +.Ltmp115452: + .loc 566 0 53 is_stmt 0 + movq -48(%rbp), %rdx +.Ltmp115453: + .loc 207 475 9 is_stmt 1 + movq %rax, (%rdx,%rdi,8) +.Ltmp115454: +.LBB1688_84: + .loc 182 1868 54 + testq %rcx, %rcx + +``` diff --git a/vortex-array/Cargo.toml b/vortex-array/Cargo.toml index d00b811a387..2da7f59309f 100644 --- a/vortex-array/Cargo.toml +++ b/vortex-array/Cargo.toml @@ -129,6 +129,10 @@ harness = false name = "binary_ops" harness = false +[[bench]] +name = "row_fn_executor" +harness = false + [[bench]] name = "kleene_bool" harness = false @@ -282,3 +286,7 @@ harness = false [[bench]] name = "slice_dict_primitive" harness = false + +[[bench]] +name = "strict_validity" +harness = false diff --git a/vortex-array/benches/like.rs b/vortex-array/benches/like.rs index 68219724717..e83fae69b28 100644 --- a/vortex-array/benches/like.rs +++ b/vortex-array/benches/like.rs @@ -87,13 +87,9 @@ fn like_regex(bencher: Bencher) { bench_like(bencher, "h_llo%w%d", LikeOptions::default()); } -#[divan::bench] -fn like_per_row_patterns(bencher: Bencher) { +fn bench_per_row_patterns(bencher: Bencher, patterns: ArrayRef) { let session = vortex_array::array_session(); let array = strings(); - // A non-constant pattern child takes the per-row path; repeated patterns hit the - // compile cache. - let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "hello%")).into_array(); bencher .with_inputs(|| { ( @@ -109,6 +105,44 @@ fn like_per_row_patterns(bencher: Bencher) { .bench_values(|(array, mut ctx)| array.execute::(&mut ctx).unwrap()); } +#[divan::bench] +fn like_per_row_patterns(bencher: Bencher) { + // A non-constant pattern child takes the per-row path; repeated patterns hit the + // compile cache. + let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "hello%")).into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// The per-row path with the compile cache hit on every row, carrying the infix pattern that +/// [`like_per_row_distinct_patterns`] varies. Both compile the same shape and match the same way, +/// so the only difference between them is how often a pattern is compiled. +#[divan::bench] +fn like_per_row_repeated_patterns(bencher: Bencher) { + let patterns = VarBinViewArray::from_iter_str((0..ARRAY_SIZE).map(|_| "%aaa%")).into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// The per-row path with the compile cache defeated: every row carries a distinct pattern of the +/// same shape, so each row pays one [`LikePattern`] compilation. +/// +/// Paired with [`like_per_row_repeated_patterns`] this isolates the cost of compiling a pattern from +/// the cost of matching against it, which is what any kernel that cannot cache across rows pays. +#[divan::bench] +fn like_per_row_distinct_patterns(bencher: Bencher) { + let patterns = VarBinViewArray::from_iter_str( + (0..ARRAY_SIZE).map(|i| format!("%{}%", distinct_trigram(i))), + ) + .into_array(); + bench_per_row_patterns(bencher, patterns); +} + +/// A distinct three-letter lowercase infix per row, so `ARRAY_SIZE` rows never repeat a pattern +/// while every pattern keeps the same shape and compiles the same way. +fn distinct_trigram(i: usize) -> String { + let letter = |shift: usize| char::from(b'a' + u8::try_from((i >> shift) % 26).unwrap()); + [letter(0), letter(5), letter(10)].iter().collect() +} + #[divan::bench] fn ilike_contains(bencher: Bencher) { bench_like( diff --git a/vortex-array/benches/row_fn_executor.rs b/vortex-array/benches/row_fn_executor.rs new file mode 100644 index 00000000000..c66865ef26d --- /dev/null +++ b/vortex-array/benches/row_fn_executor.rs @@ -0,0 +1,419 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Compares cheap primitive row functions with the specialized columnar implementation. + +#![expect(clippy::unwrap_used)] + +use std::mem::MaybeUninit; +use std::sync::LazyLock; + +use divan::Bencher; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::array_session; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::DeferredError; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::OutputSink; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_buffer::BufferMut; +use vortex_error::VortexResult; +use vortex_error::vortex_err; +use vortex_session::VortexSession; +use vortex_session::registry::CachedId; + +const ROWS: usize = 65_536; + +static SESSION: LazyLock = LazyLock::new(array_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +#[derive(Clone)] +struct RowWrappingAdd; + +impl RowFn for RowWrappingAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_wrapping_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs.wrapping_add(rhs), + ) + } +} + +#[derive(Clone)] +struct RowCheckedAdd; + +struct CheckedAddSink { + values: BufferMut, + row_count: usize, +} + +struct CheckedAddRows<'a> { + values: &'a mut [MaybeUninit], +} + +struct CheckedAddRow<'a> { + value: &'a mut MaybeUninit, +} + +impl CheckedAddRow<'_> { + fn write(self, lhs: i64, rhs: i64) -> bool { + let value = lhs.wrapping_add(rhs); + let error = (lhs ^ value) & (rhs ^ value); + self.value.write(value); + error < 0 + } +} + +impl OutputSink for CheckedAddSink { + const ERRORS_ARE_DEFERRED: bool = true; + + type Rows<'a> = CheckedAddRows<'a>; + type Row<'a> = CheckedAddRow<'a>; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::from(i64::PTYPE)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self { + values: BufferMut::with_capacity(rows), + row_count: rows, + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + CheckedAddRows { + values: &mut self.values.spare_capacity_mut()[..self.row_count], + } + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.values.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + CheckedAddRow { + value: &mut rows.values[index], + } + } + + fn finish(mut self, error: DeferredError) -> VortexResult { + if error.occurred() { + return Err(vortex_err!("integer overflow in row checked add")); + } + + // SAFETY: dense execution writes every slot before `finish` is called. This sink does not + // support branch-and-skip, and filtered execution allocates exactly one slot per valid row. + unsafe { self.values.set_len(self.row_count) }; + Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) + } +} + +impl RowFn for RowCheckedAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_checked_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), CheckedAddSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| output.write(lhs, rhs), + ) + } +} + +struct I64Sink(BufferMut); + +impl OutputSink for I64Sink { + type Rows<'a> = &'a mut [i64]; + type Row<'a> = &'a mut i64; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::from(i64::PTYPE)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self(BufferMut::zeroed(rows))) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.0.as_mut_slice() + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + &mut rows[index] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(PrimitiveArray::new(self.0.freeze(), Validity::NonNullable).into_array()) + } +} + +#[derive(Clone)] +struct RowSinkWrappingAdd; + +impl RowFn for RowSinkWrappingAdd { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.row_sink_wrapping_add"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), I64Sink, _, _>( + |_| (), + |&(), (lhs, rhs), out| { + *out = lhs.wrapping_add(rhs); + }, + ) + } +} + +fn inputs() -> (ArrayRef, ArrayRef) { + let lhs = (0..ROWS) + .map(|index| index as i64) + .collect::>() + .into_array(); + let rhs = (0..ROWS) + .map(|index| (index % 17) as i64) + .collect::>() + .into_array(); + (lhs, rhs) +} + +fn constant_inputs() -> (ArrayRef, ArrayRef) { + let (lhs, _) = inputs(); + let rhs = ConstantArray::new(Scalar::from(7i64), ROWS).into_array(); + (lhs, rhs) +} + +fn nullable_inputs() -> (ArrayRef, ArrayRef) { + let lhs = PrimitiveArray::new( + (0..ROWS).map(|index| index as i64).collect::>(), + Validity::from_iter((0..ROWS).map(|index| !index.is_multiple_of(5))), + ) + .into_array(); + let rhs = PrimitiveArray::new( + (0..ROWS) + .map(|index| (index % 17) as i64) + .collect::>(), + Validity::from_iter((0..ROWS).map(|index| !index.is_multiple_of(7))), + ) + .into_array(); + (lhs, rhs) +} + +fn bench_row_fn(bencher: Bencher, row_fn: F) +where + F: RowFn, +{ + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + row_fn + .clone() + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn specialized_checked_add(bencher: Bencher) { + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add(bencher: Bencher) { + bench_row_fn(bencher, RowWrappingAdd); +} + +#[divan::bench] +fn row_sink_wrapping_add(bencher: Bencher) { + bench_row_fn(bencher, RowSinkWrappingAdd); +} + +#[divan::bench] +fn handrolled_sink_wrapping_add(bencher: Bencher) { + bencher + .with_inputs(inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + let lhs = lhs + .execute::(&mut ctx) + .unwrap() + .into_buffer::(); + let rhs = rhs + .execute::(&mut ctx) + .unwrap() + .into_buffer::(); + let mut output = BufferMut::zeroed(ROWS); + for ((out, lhs), rhs) in output + .as_mut_slice() + .iter_mut() + .zip(lhs.as_slice()) + .zip(rhs.as_slice()) + { + *out = lhs.wrapping_add(*rhs); + } + PrimitiveArray::new(output.freeze(), Validity::NonNullable).into_array() + }); +} + +#[divan::bench] +fn row_checked_add(bencher: Bencher) { + bench_row_fn(bencher, RowCheckedAdd); +} + +#[divan::bench] +fn specialized_checked_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowWrappingAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_checked_add_constant(bencher: Bencher) { + bencher + .with_inputs(constant_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowCheckedAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn specialized_checked_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + lhs.binary(rhs, Operator::Add) + .unwrap() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_checked_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowCheckedAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} + +#[divan::bench] +fn row_wrapping_add_nullable(bencher: Bencher) { + bencher + .with_inputs(nullable_inputs) + .bench_local_values(|(lhs, rhs)| { + let mut ctx = SESSION.create_execution_ctx(); + RowWrappingAdd + .try_new_array(ROWS, EmptyOptions, [lhs, rhs]) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap() + }); +} diff --git a/vortex-array/benches/strict_validity.rs b/vortex-array/benches/strict_validity.rs new file mode 100644 index 00000000000..c4e7ca77d8e --- /dev/null +++ b/vortex-array/benches/strict_validity.rs @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Benchmarks how the row lifting applies validity to a dense kernel. +//! +//! Both arms run the same kernel over the same nullable input and differ only in how the output's +//! nulls are restored: +//! +//! - `lazy` is what the lifting behind [`RowFn`] does: conjoin the input validities (itself lazy) +//! and hand the resulting boolean array straight to `mask`. +//! - `eager` is the strategy it replaced: materialize the conjunction into a `Mask`, copy it into a +//! `BitBuffer`, wrap that in a `BoolArray`, and mask with it. +//! +//! `chain` composes three of the same function, which is where a per-call materialization compounds. + +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] + +use std::sync::LazyLock; + +use divan::Bencher; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::array_session; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::PType; +use vortex_array::expr::Expression; +use vortex_array::expr::union_child_validities; +use vortex_array::scalar_fn::Arity; +use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_session::VortexSession; +use vortex_session::registry::CachedId; + +static SESSION: LazyLock = LazyLock::new(array_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +const SIZES: &[usize] = &[65536, 1 << 20]; + +/// The shared kernel: double every lane, ignoring validity. +fn doubled(input: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let input = input.clone().execute::(ctx)?; + let values: Buffer = input + .as_slice::() + .iter() + .map(|value| value.wrapping_mul(2)) + .collect(); + Ok(PrimitiveArray::new(values, Validity::NonNullable).into_array()) +} + +/// Dense row function: the lifting decides how validity is applied. +/// +/// Its [`reduce_encoded`](RowFn::reduce_encoded) answers with [`doubled`] before the row loop is +/// reached, so this arm runs exactly the kernel [`EagerDouble`] runs and the two differ only in +/// validity. The row closure below is what makes it a [`RowFn`] at all, and never executes. +#[derive(Clone)] +struct LazyDouble; + +impl RowFn for LazyDouble { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.lazy_double"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i32,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = value.wrapping_mul(2), + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + args: &[ArrayRef], + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + doubled(&args[0], ctx).map(Some) + } +} + +/// The same function, applying validity the way the adapter used to: materialize a mask first. +#[derive(Clone)] +struct EagerDouble; + +impl ScalarFnVTable for EagerDouble { + type Options = EmptyOptions; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("bench.eager_double"); + *ID + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(1) + } + + fn child_name(&self, _options: &Self::Options, _child_idx: usize) -> ChildName { + ChildName::from("input") + } + + fn return_dtype(&self, _options: &Self::Options, args: &[DType]) -> VortexResult { + Ok(DType::Primitive(PType::I32, args[0].nullability())) + } + + fn execute( + &self, + _options: &Self::Options, + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let input = args.get(0)?; + let valid = input.validity()?.execute_mask(args.row_count(), ctx)?; + let values = doubled(&input, ctx)?; + + if valid.all_true() { + return values.cast(DType::Primitive(PType::I32, input.dtype().nullability())); + } + + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + values.mask(mask) + } + + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + union_child_validities(expression) + } + + fn is_strict(&self, _options: &Self::Options) -> bool { + true + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + false + } +} + +/// A nullable i32 column with array-backed validity (~10% nulls). +fn nullable_input(len: usize) -> ArrayRef { + PrimitiveArray::new( + (0..len as i32).collect::>(), + Validity::from_iter((0..len).map(|i| !i.is_multiple_of(10))), + ) + .into_array() +} + +fn bench_depth(bencher: Bencher, scalar_fn: V, len: usize, depth: usize) +where + V: ScalarFnVTable + Clone, +{ + bencher + .with_inputs(|| nullable_input(len)) + .bench_local_values(|input| { + let mut ctx = SESSION.create_execution_ctx(); + let mut array = input; + for _ in 0..depth { + array = scalar_fn + .clone() + .try_new_array(len, EmptyOptions, [array]) + .unwrap() + .into_array(); + } + array.execute::(&mut ctx).unwrap() + }); +} + +#[divan::bench(args = SIZES)] +fn lazy(bencher: Bencher, len: usize) { + bench_depth(bencher, LazyDouble, len, 1); +} + +#[divan::bench(args = SIZES)] +fn eager(bencher: Bencher, len: usize) { + bench_depth(bencher, EagerDouble, len, 1); +} + +#[divan::bench(args = SIZES)] +fn lazy_chain(bencher: Bencher, len: usize) { + bench_depth(bencher, LazyDouble, len, 3); +} + +#[divan::bench(args = SIZES)] +fn eager_chain(bencher: Bencher, len: usize) { + bench_depth(bencher, EagerDouble, len, 3); +} diff --git a/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs b/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs index 6b389a0554b..d7bcc8d0b4c 100644 --- a/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs +++ b/vortex-array/src/scalar/typed_view/primitive/numeric_operator.rs @@ -10,7 +10,7 @@ use vortex_error::vortex_err; use crate::scalar_fn::fns::operators::Operator; -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] /// Binary element-wise operations. pub enum NumericOperator { /// Binary element-wise addition of two arrays or of two scalars. diff --git a/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs b/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs index 3bfb11a266e..58fb22ed73e 100644 --- a/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs +++ b/vortex-array/src/scalar_fn/fns/binary/compare/primitive.rs @@ -4,6 +4,7 @@ //! Native comparison of primitive arrays via bit-packing lane kernels. use vortex_buffer::BitBuffer; +use vortex_buffer::Buffer; use vortex_error::VortexResult; use vortex_error::vortex_bail; @@ -11,18 +12,20 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::IntoArray; use crate::arrays::BoolArray; +use crate::arrays::Constant; use crate::arrays::ConstantArray; +use crate::arrays::PrimitiveArray; use crate::dtype::DType; use crate::dtype::NativePType; use crate::dtype::Nullability; use crate::dtype::PType; use crate::match_each_native_ptype; use crate::scalar::Scalar; -use crate::scalar_fn::fns::binary::PrimitiveOperand; use crate::scalar_fn::fns::binary::compare::collect_bits; use crate::scalar_fn::fns::binary::compare::collect_zip_bits; use crate::scalar_fn::fns::binary::compare::compare_validity; use crate::scalar_fn::fns::operators::CompareOperator; +use crate::validity::Validity; /// Compare two primitive arrays of the same [`PType`]. /// @@ -128,3 +131,67 @@ fn compare_slice_constant(lhs: &[T], rhs: T, op: CompareOperator CompareOperator::Lte => collect_bits(lhs, |a: T| a.is_le(rhs)), } } + +/// A primitive binary-operator operand: a materialized buffer, a non-null constant, or an +/// all-null constant. +/// +/// Splitting the constant out of the buffer is what lets the lane kernels above hoist it into a +/// register instead of reading it back per lane. +enum PrimitiveOperand { + /// A decoded column, one value per row. + Array { + values: Buffer, + validity: Validity, + }, + + /// The same non-null value in every row. + Constant { + value: T, + len: usize, + validity: Validity, + }, + + /// A null in every row, carrying only the row count. + Null(usize), +} + +impl PrimitiveOperand { + fn try_new(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + if let Some(constant) = array.as_opt::() { + return Ok( + match constant.scalar().as_primitive().try_typed_value::()? { + Some(value) => Self::Constant { + value, + len: array.len(), + validity: if constant.scalar().dtype().is_nullable() { + Validity::AllValid + } else { + Validity::NonNullable + }, + }, + None => Self::Null(array.len()), + }, + ); + } + + let array = array.clone().execute::(ctx)?; + let validity = array.validity()?; + let values = array.into_buffer::(); + Ok(Self::Array { values, validity }) + } + + fn len(&self) -> usize { + match self { + Self::Array { values, .. } => values.len(), + Self::Constant { len, .. } | Self::Null(len) => *len, + } + } + + fn validity(&self) -> Validity { + match self { + Self::Array { validity, .. } => validity.clone(), + Self::Constant { validity, .. } => validity.clone(), + Self::Null(_) => Validity::AllInvalid, + } + } +} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs index afb709abe1c..47c7d1351b9 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Checked-lane execution for numeric kernels, driven by the shared +//! Checked-lane execution for the decimal kernels, driven by the shared //! `vortex-compute` lane kernels. - -use std::ops::BitOrAssign; +//! +//! The primitive widths do not come through here: they are computed one row at a time by +//! [`row`](super::row), which writes a value for every row and reduces failure as one bit rather +//! than scanning lanes. use vortex_buffer::Buffer; use vortex_buffer::BufferMut; @@ -13,34 +15,22 @@ use vortex_compute::lane_kernels::IndexedSourceExt; use vortex_mask::AllOr; use vortex_mask::Mask; -/// Evidence that a lane failed, anything other than [`Default`] meaning failure. -/// -/// `bool` is the ordinary choice; [`map_checked_into`] explains why the wider members exist and -/// asserts the width bound that membership here does **not** imply. -/// -/// [`map_checked_into`]: IndexedSourceExt::map_checked_into -pub(super) trait Failure: Copy + Default + PartialEq + BitOrAssign {} - -impl Failure for bool {} -impl Failure for u8 {} -impl Failure for u16 {} -impl Failure for u32 {} -impl Failure for u64 {} - /// Apply the fallible `apply` over every lane of `source`, returning -/// `Err(first_failing_valid_lane)` only when it returns `None` on a _valid_ lane. +/// `Err(first_failing_valid_lane)` only when it returns `None` on a valid lane. /// /// `apply` also runs on invalid lanes, whose failures are masked out and whose values are /// unspecified, so it must be total: no panics or side effects on any stored lane value. /// -/// This drives the one-pass early-exit kernels, which abort at the end of the enclosing 64-lane -/// chunk. Use it when per-lane failure handling is cheap relative to the operation, as in integer -/// division. Prefer [`checked_apply_lanes`] when the failure check itself vectorizes. +/// This drives the one-pass early-exit kernels: failures abort at the end of the enclosing +/// 64-lane chunk. It suits an operation whose per-lane failure handling is cheap relative to the +/// operation itself, which is what the decimal kernels and their per-lane casts are. /// -/// `#[inline]`: this must inline into the caller that builds the closure, so that a captured -/// constant operand flattens into a register rather than living behind a pointer the loop reloads -/// on every lane, which blocks vectorization. -#[inline] +/// `#[inline(always)]`: this wrapper and its kernel calls must inline into the caller that +/// constructs the closure, so the closure environment (e.g. a captured constant operand) +/// flattens into registers. Left to its own devices under `codegen-units > 1`, the compiler +/// keeps the environment behind a pointer, and reloading a captured constant on every lane +/// blocks vectorization of the whole loop. +#[inline(always)] pub(super) fn checked_lanes( source: S, valid_rows: &Mask, @@ -61,7 +51,6 @@ where }; let mut values = BufferMut::::with_capacity(len); - let out = &mut values.spare_capacity_mut()[..len]; match valid_bits { None => source.try_map_into(out, apply)?, @@ -73,55 +62,3 @@ where Ok(values.freeze()) } - -/// Apply the split value/failure `apply` over every lane of `source`, returning -/// `Err(first_failing_valid_lane)` only when it flags a _valid_ lane. -/// -/// The hot pass writes every value unconditionally and OR-reduces one piece of evidence, leaving -/// the loop free of per-lane selects and per-chunk exit branches. Only if a lane flagged does a -/// cold second pass re-run `apply` through the early-exit kernels, which drop null-lane failures -/// and attribute the first valid one. -/// -/// Like [`checked_lanes`], `apply` runs on invalid lanes and must be total. -/// -/// `#[inline]`: see [`checked_lanes`]. -#[inline] -pub(super) fn checked_apply_lanes( - source: S, - valid_rows: &Mask, - mut apply: Apply, -) -> Result, usize> -where - S: IndexedSource + Copy, - T: Copy + Default, - Fail: Failure, - Apply: FnMut(S::Item) -> (T, Fail), -{ - let len = source.len(); - debug_assert_eq!(len, valid_rows.len()); - - let valid_bits = match valid_rows.bit_buffer() { - AllOr::All => None, - AllOr::None => return Ok(Buffer::zeroed(len)), - AllOr::Some(valid_bits) => Some(valid_bits), - }; - - let mut values = BufferMut::::with_capacity(len); - - let out = &mut values.spare_capacity_mut()[..len]; - if source.map_checked_into(out, &mut apply) != Fail::default() { - let mut checked = |item: S::Item| { - let (value, failure) = apply(item); - (failure == Fail::default()).then_some(value) - }; - match valid_bits { - None => source.try_map_into(out, &mut checked)?, - Some(valid_bits) => source.try_map_masked_into(valid_bits, out, &mut checked)?, - } - } - - // SAFETY: the kernels initialize every lane in `out`. - unsafe { values.set_len(len) }; - - Ok(values.freeze()) -} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs index 6622e08f82b..21db5dc8c8e 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/mod.rs @@ -4,17 +4,21 @@ //! Native execution of the arithmetic operators (Add/Sub/Mul/Div) of the [`Binary`] scalar //! function. There is no Arrow fallback. //! +//! The primitive widths are computed by a [`RowFn`](crate::scalar_fn::RowFn), which owns null +//! handling, constants and validity for them; see [`row`]. Decimal keeps its own columnar +//! implementation in [`decimal`]. +//! //! [`Binary`]: super::Binary mod checked; mod decimal; mod primitive; +mod row; #[cfg(test)] mod tests; use decimal::execute_numeric_decimal; -pub(crate) use primitive::PrimitiveOperand; -use primitive::execute_numeric_primitive; +use row::execute_numeric_primitive; use vortex_error::VortexResult; use vortex_error::vortex_ensure; diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs index 357547f25b8..7a6c1de4641 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs @@ -1,65 +1,57 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use vortex_buffer::Buffer; -use vortex_compute::lane_kernels::IndexedSource; -use vortex_compute::lane_kernels::LaneZip; -use vortex_error::VortexResult; -use vortex_error::vortex_err; -use vortex_mask::Mask; - -use super::checked::Failure; -use super::checked::checked_apply_lanes; -use super::checked::checked_lanes; -use crate::ArrayRef; -use crate::ExecutionCtx; -use crate::IntoArray; -use crate::arrays::Constant; -use crate::arrays::ConstantArray; -use crate::arrays::PrimitiveArray; -use crate::builtins::ArrayBuiltins; -use crate::dtype::DType; +//! The checked arithmetic one row of a primitive column is computed with. +//! +//! Each operator is a type implementing [`CheckedPrimitiveOp`] at every native width, and each +//! width implements [`CheckedArithmetic`] with the value and failure evidence written separately. +//! Keeping them apart is what lets [`row`](super::row) write a value for every row and reduce the +//! evidence without a branch, so the loop vectorizes. + +use std::ops::BitOrAssign; + use crate::dtype::NativePType; -use crate::dtype::PType; use crate::dtype::half::f16; -use crate::match_each_native_ptype; -use crate::scalar::NumericOperator; -use crate::scalar::Scalar; -use crate::validity::Validity; -struct CheckedAdd; +/// Checked addition, failing on integer overflow. +pub(super) struct CheckedAdd; -struct CheckedSub; +/// Checked subtraction, failing on integer overflow. +pub(super) struct CheckedSub; -struct CheckedMul; +/// Checked multiplication, failing on integer overflow. +pub(super) struct CheckedMul; -struct CheckedDiv; +/// Checked division, failing on integer division by zero and on `MIN / -1`. +pub(super) struct CheckedDiv; -trait CheckedPrimitiveOp: Sized { - /// Message for the error raised when this operation fails on a valid lane. +/// Evidence that some row failed, in a form that OR-reduces across the batch. +/// +/// A plain `bool` is the obvious choice and the right one for most operations. Unsigned +/// multiplication is the exception: deriving a `bool` from the widened product costs a comparison, +/// and LLVM rewrites that comparison plus the product into `llvm.umul.with.overflow`, which has no +/// vector form and scalarizes the whole loop. Carrying the discarded high half instead means the row +/// never compares, so the multiply stays a widening vector multiply and the reduction stays a +/// vector OR. **The width must not exceed the element's**, or the reduction becomes the loop's +/// bottleneck instead of the arithmetic. +pub(super) trait Failure: Copy + Default + PartialEq + BitOrAssign {} + +impl Failure for T {} + +/// One arithmetic operator at one width, as a value and its failure evidence. +/// +/// The pair rather than an `Option` is what a row can write unconditionally: the value is stored +/// whatever the evidence says, and a failing row is either masked away as null or turned into a +/// batch error before anything reads it. +pub(super) trait CheckedPrimitiveOp: 'static + Sized { + /// The error reported for a batch in which some valid row failed. const ERROR: &'static str; - /// Whether to check inside the value loop and exit early, rather than reducing evidence over - /// the whole batch and re-running only once some lane has flagged. - const CHECKED_VALUE_LOOP: bool = false; - - /// How this operation reports a failing lane. See [`Failure`]. + /// How this operation reports a failing row. See [`Failure`]. type Failure: Failure; - /// Compute a lane's value and its failure evidence together. - /// - /// Overflowing-style rather than `Option`, so the vectorizable kernels can write every - /// lane's value unconditionally and reduce the evidence separately. [`Self::checked`] is the - /// shape for the scalar and early-exit paths. + /// The result of this operation, paired with evidence of whether the row failed. fn apply(lhs: T, rhs: T) -> (T, Self::Failure); - - /// [`Self::apply`] folded into the `Option` that the early-exit kernels take. - #[inline(always)] - fn checked(lhs: T, rhs: T) -> Option { - let (value, failed) = Self::apply(lhs, rhs); - - (failed == Self::Failure::default()).then_some(value) - } } impl CheckedPrimitiveOp for CheckedAdd { @@ -97,12 +89,6 @@ impl CheckedPrimitiveOp for CheckedMul { impl CheckedPrimitiveOp for CheckedDiv { const ERROR: &'static str = "integer division by zero or overflow in checked div"; - // Integer division still lowers to scalar divides, so the split - // value/error-scan loop used to auto-vectorize add/sub/mul only adds a - // second full scan. Use the one-pass early-exit checked kernel for integer - // division, matching Arrow/Velox. Float division has no checked errors and - // stays on the split/vectorizable default path. - const CHECKED_VALUE_LOOP: bool = T::DIV_CHECKS_IN_VALUE_LOOP; type Failure = bool; @@ -116,207 +102,21 @@ impl CheckedPrimitiveOp for CheckedDiv { }; (value, failed) } - - #[inline(always)] - fn checked(lhs: T, rhs: T) -> Option { - lhs.div_checked(rhs) - } } -/// Execute a numeric operation between two primitive-typed arrays. -pub(super) fn execute_numeric_primitive( - lhs: &ArrayRef, - rhs: &ArrayRef, - op: NumericOperator, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let ptype = PType::try_from(lhs.dtype())?; - - match_each_native_ptype!(ptype, |T| { - match op { - NumericOperator::Add => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Sub => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Mul => execute_checked_typed::(lhs, rhs, ctx), - NumericOperator::Div => execute_checked_typed::(lhs, rhs, ctx), - } - }) -} - -fn execute_checked_typed( - lhs: &ArrayRef, - rhs: &ArrayRef, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: NativePType, - Op: CheckedPrimitiveOp, - Scalar: From, - Scalar: From>, -{ - let result_dtype = lhs - .dtype() - .with_nullability(lhs.dtype().nullability() | rhs.dtype().nullability()); - let lhs = PrimitiveOperand::::try_new(lhs, ctx)?; - let rhs = PrimitiveOperand::::try_new(rhs, ctx)?; - let len = lhs.len(); - debug_assert_eq!(len, rhs.len()); - - let validity = lhs.validity().and(rhs.validity())?; - let valid_rows = validity.execute_mask(len, ctx)?; - - let values = match (&lhs, &rhs) { - ( - PrimitiveOperand::Array { values: lhs, .. }, - PrimitiveOperand::Array { values: rhs, .. }, - ) => checked_op_lanes::<_, T, Op>( - LaneZip::new(lhs.as_slice(), rhs.as_slice()), - &valid_rows, - |(lhs, rhs)| (lhs, rhs), - ), - ( - PrimitiveOperand::Array { values: lhs, .. }, - PrimitiveOperand::Constant { value: rhs, .. }, - ) => { - // Capture the constant by value so it stays hoisted out of the lane loop. - let rhs = *rhs; - checked_op_lanes::<_, T, Op>(lhs.as_slice(), &valid_rows, move |lhs| (lhs, rhs)) - } - ( - PrimitiveOperand::Constant { value: lhs, .. }, - PrimitiveOperand::Array { values: rhs, .. }, - ) => { - let lhs = *lhs; - checked_op_lanes::<_, T, Op>(rhs.as_slice(), &valid_rows, move |rhs| (lhs, rhs)) - } - ( - PrimitiveOperand::Constant { value: lhs, .. }, - PrimitiveOperand::Constant { value: rhs, .. }, - ) => { - let value = Op::checked(*lhs, *rhs) - .ok_or_else(|| vortex_err!(InvalidArgument: "{}", Op::ERROR))?; - return Ok(constant_result_array(value, len, &result_dtype)); - } - (PrimitiveOperand::Null(_), _) | (_, PrimitiveOperand::Null(_)) => Ok(Buffer::zeroed(len)), - } - .map_err(|_lane| vortex_err!(InvalidArgument: "{}", Op::ERROR))?; - - primitive_result_array::(values, validity, &result_dtype) -} - -/// Run `Op` over the lanes of `source` in the loop shape it declares: the one-pass early-exit -/// kernel when `Op::CHECKED_VALUE_LOOP` is set, and the split value/failure kernel otherwise. +/// The per-width arithmetic behind [`CheckedPrimitiveOp`], with each operation split into the value +/// it produces and whether producing it failed. /// -/// `#[inline]`: see [`checked_lanes`]. -#[inline] -fn checked_op_lanes( - source: S, - valid_rows: &Mask, - mut to_operands: impl FnMut(S::Item) -> (T, T), -) -> Result, usize> -where - S: IndexedSource + Copy, - T: NativePType, - Op: CheckedPrimitiveOp, -{ - if Op::CHECKED_VALUE_LOOP { - checked_lanes(source, valid_rows, |item| { - let (lhs, rhs) = to_operands(item); - Op::checked(lhs, rhs) - }) - } else { - checked_apply_lanes(source, valid_rows, |item| { - let (lhs, rhs) = to_operands(item); - Op::apply(lhs, rhs) - }) - } -} - -fn primitive_result_array( - values: Buffer, - validity: Validity, - dtype: &DType, -) -> VortexResult { - let array = PrimitiveArray::new(values, validity).into_array(); - if array.dtype() == dtype { - return Ok(array); - } - array.cast(dtype.clone()) -} - -fn constant_result_array(value: T, len: usize, dtype: &DType) -> ArrayRef -where - T: NativePType, - Scalar: From + From>, -{ - if dtype.is_nullable() { - ConstantArray::new(Some(value), len).into_array() - } else { - ConstantArray::new(value, len).into_array() - } -} - -/// A primitive binary-operator operand: a materialized buffer, a non-null constant, or an -/// all-null constant. -pub(crate) enum PrimitiveOperand { - Array { - values: Buffer, - validity: Validity, - }, - Constant { - value: T, - len: usize, - validity: Validity, - }, - Null(usize), -} - -impl PrimitiveOperand { - pub(crate) fn try_new(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { - if let Some(constant) = array.as_opt::() { - return Ok( - match constant.scalar().as_primitive().try_typed_value::()? { - Some(value) => Self::Constant { - value, - len: array.len(), - validity: if constant.scalar().dtype().is_nullable() { - Validity::AllValid - } else { - Validity::NonNullable - }, - }, - None => Self::Null(array.len()), - }, - ); - } - - let array = array.clone().execute::(ctx)?; - let validity = array.validity()?; - let values = array.into_buffer::(); - Ok(Self::Array { values, validity }) - } - - pub(crate) fn len(&self) -> usize { - match self { - Self::Array { values, .. } => values.len(), - Self::Constant { len, .. } | Self::Null(len) => *len, - } - } - - pub(crate) fn validity(&self) -> Validity { - match self { - Self::Array { validity, .. } => validity.clone(), - Self::Constant { validity, .. } => validity.clone(), - Self::Null(_) => Validity::AllInvalid, - } - } -} - -trait CheckedArithmetic: NativePType { - const DIV_CHECKS_IN_VALUE_LOOP: bool; - - /// How multiplication reports a failing lane, which is width-dependent in a way the other - /// operations are not. `impl_checked_unsigned!` and `impl_checked_signed!` say why each family - /// reports what it reports. +/// Every `_value` method **must** be total: it is called for rows behind nulls, whose operands are +/// arbitrary, so it may not panic or trap. Integer division is the one that needs care, and +/// [`CheckedDiv`] supplies the default instead of dividing when the divisor is rejected. +pub(super) trait CheckedArithmetic: NativePType { + /// How multiplication reports a failing row. + /// + /// `Self` for the unsigned widths that have a widening multiply, so the row can hand back the + /// discarded high half rather than comparing. `bool` everywhere else: the narrow signed widths + /// already vectorize through a two-sided range check, floats never overflow, and the 64-bit + /// widths use a full-width evidence word. type MulFailure: Failure; fn add_value(self, rhs: Self) -> Self; @@ -327,16 +127,10 @@ trait CheckedArithmetic: NativePType { fn mul_failure(self, rhs: Self) -> Self::MulFailure; fn div_value(self, rhs: Self) -> Self; fn div_error(self, rhs: Self) -> bool; - fn div_checked(self, rhs: Self) -> Option; } /// The integer arithmetic every width shares, given the two things that actually differ between -/// them: how multiplication reports a failing lane, and how add/sub/div detect one. -/// -/// Only `mul_failure` genuinely varies per width, so it is the macro's parameter and everything -/// else is written once. The `$mul_failure_ty` and body are supplied by the caller because the -/// choice between a widened high half and a `bool` is exactly the vectorization decision described -/// on [`Failure`]. +/// them: how multiplication reports a failing row, and how add/sub/div detect one. macro_rules! impl_checked_integer { ( $ty:ty, @@ -347,8 +141,6 @@ macro_rules! impl_checked_integer { = |$mf_lhs:ident, $mf_rhs:ident| $mul_failure:expr, ) => { impl CheckedArithmetic for $ty { - const DIV_CHECKS_IN_VALUE_LOOP: bool = true; - type MulFailure = $mul_failure_ty; #[inline(always)] @@ -395,19 +187,12 @@ macro_rules! impl_checked_integer { let ($div_lhs, $div_rhs) = (self, rhs); $div_error } - - #[inline(always)] - fn div_checked(self, rhs: Self) -> Option { - self.checked_div(rhs) - } } }; } -/// The unsigned widths. `add`, `sub` and `div` are written once here, and `widening_mul` covers -/// every width including the 64-bit one, which widens into `u128`: the discarded high half of the -/// widened product is the failure evidence, and costs none of the comparison LLVM folds into -/// `umul.with.overflow`. +/// The unsigned widths. The discarded high half of the widened product is the failure evidence, +/// and costs none of the comparison LLVM folds into `umul.with.overflow`. macro_rules! impl_checked_unsigned { ($ty:ty, widening_mul: $wide:ty) => { impl_checked_integer!( @@ -420,12 +205,8 @@ macro_rules! impl_checked_unsigned { }; } -/// The signed widths, on the same principle. `widening_mul` is the shorthand for the narrow widths, -/// whose two-sided range check over a wider product is not the shape LLVM folds into an overflow -/// intrinsic, so they vectorize while reporting a plain `bool`. The 64-bit width cannot: deriving a -/// `bool` there costs the comparison that scalarizes the loop, so `high_half_mul` hands back the -/// discarded high half as a word. Both arms derive every shift and bound from `$ty`, so -/// instantiating one at a new width cannot silently keep another width's constants. +/// The signed widths. The narrow widths report a two-sided range check as `bool`; the 64-bit width +/// reports the discarded high half as a word so deriving the evidence does not scalarize the loop. macro_rules! impl_checked_signed { ($ty:ty, widening_mul: $wide:ty) => { impl_checked_signed!($ty, mul_failure: bool = |lhs, rhs| { @@ -433,9 +214,6 @@ macro_rules! impl_checked_signed { product < <$ty>::MIN as $wide || product > <$ty>::MAX as $wide }); }; - // Zero exactly when the product fits: a signed multiply overflows iff the high half of the true - // product differs from the sign extension of the half that was kept, so the two XOR to zero on - // the lanes that fit. `tests::test_multiply_overflow_boundaries` pins the boundaries. ($ty:ty, high_half_mul: $wide:ty => $failure:ty) => { impl_checked_signed!($ty, mul_failure: #[expect( clippy::cast_possible_truncation, @@ -451,7 +229,7 @@ macro_rules! impl_checked_signed { ( $ty:ty, mul_failure: $(#[$mul_failure_attr:meta])* $mul_failure_ty:ty - = |$l:ident, $r:ident| $mul_failure:expr + = |$lhs:ident, $rhs:ident| $mul_failure:expr ) => { impl_checked_integer!( $ty, @@ -464,7 +242,7 @@ macro_rules! impl_checked_signed { ((lhs ^ rhs) & (lhs ^ value)) < 0 }, div_error: |lhs, rhs| rhs == 0 || (lhs == <$ty>::MIN && rhs == -1), - mul_failure: $(#[$mul_failure_attr])* $mul_failure_ty = |$l, $r| $mul_failure, + mul_failure: $(#[$mul_failure_attr])* $mul_failure_ty = |$lhs, $rhs| $mul_failure, ); }; } @@ -473,8 +251,6 @@ macro_rules! impl_checked_float { ($($ty:ty),+ $(,)?) => { $( impl CheckedArithmetic for $ty { - const DIV_CHECKS_IN_VALUE_LOOP: bool = false; - type MulFailure = bool; #[inline(always)] @@ -516,11 +292,6 @@ macro_rules! impl_checked_float { fn div_error(self, _rhs: Self) -> bool { false } - - #[inline(always)] - fn div_checked(self, rhs: Self) -> Option { - Some(self / rhs) - } } )+ }; @@ -543,27 +314,26 @@ mod tests { /// Values whose pairwise products are worth probing: the saturating boundaries, the sign-change /// pivots, and a spread of magnitudes that straddles the 64-bit split. const PROBES: &[i64] = &[ - 0, // - 1, // - -1, // - 2, // - -2, // - 3, // - i64::MIN, // - i64::MIN + 1, // - i64::MAX, // - i64::MAX - 1, // - 1 << 31, // - 1 << 32, // - 1 << 62, // - -(1 << 62), // - 0x7FFF_FFFF, // - -0x8000_0000, // + 0, + 1, + -1, + 2, + -2, + 3, + i64::MIN, + i64::MIN + 1, + i64::MAX, + i64::MAX - 1, + 1 << 31, + 1 << 32, + 1 << 62, + -(1 << 62), + 0x7FFF_FFFF, + -0x8000_0000, ]; - /// Every `mul_failure` impl is either a bit trick or a two-sided range check, so hold each - /// against the obvious reference: `reference` is the width's own `checked_mul`, whose `None` - /// _is_ the definition of overflow. + /// Every `mul_failure` implementation is either a bit trick or a two-sided range check, so + /// hold each against `checked_mul`, whose `None` is the definition of overflow. #[track_caller] fn assert_agrees_with_checked_mul(lhs: T, rhs: T, reference: Option) { let failed = lhs.mul_failure(rhs) != ::default(); @@ -578,14 +348,13 @@ mod tests { assert_agrees_with_checked_mul(lhs, rhs, lhs.checked_mul(rhs)); let (lhs, rhs) = (lhs as u64, rhs as u64); - assert_agrees_with_checked_mul(lhs, rhs, lhs.checked_mul(rhs)); } } } - /// The 8-bit widths are cheap enough to check exhaustively, which pins the shift of the - /// unsigned formula and the range check of the signed one against every product that exists. + /// The 8-bit widths are cheap enough to check exhaustively, pinning the unsigned shift and the + /// signed range check against every product that exists. #[test] fn mul_failure_agrees_with_checked_mul_exhaustively_at_8_bits() { for lhs in u8::MIN..=u8::MAX { diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs new file mode 100644 index 00000000000..cbbfa9e22da --- /dev/null +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/row.rs @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The primitive arithmetic operators as a [`RowFn`]. +//! +//! [`Binary`] keeps its ID, options serialization, and semantic contracts. It delegates only the +//! execution of primitive `Add`, `Sub`, `Mul`, and `Div` to [`NumericBinary`]. The helper is not +//! registered and appears in no serialized expression. +//! +//! Shared lifting owns decoding, constant handling, output allocation, nullability, validity, and +//! nullable retry. The declaration below contains only type dispatch and the per-row operation. +//! +//! [`Binary`]: crate::scalar_fn::fns::binary::Binary + +use vortex_error::VortexResult; +use vortex_error::vortex_err; +use vortex_session::registry::CachedId; + +use super::primitive::CheckedAdd; +use super::primitive::CheckedDiv; +use super::primitive::CheckedMul; +use super::primitive::CheckedPrimitiveOp; +use super::primitive::CheckedSub; +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::dtype::NativePType; +use crate::dtype::PType; +use crate::match_each_native_ptype; +use crate::scalar::NumericOperator; +use crate::scalar_fn::RowFn; +use crate::scalar_fn::RowVisitor; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::VecExecutionArgs; + +/// Execute a numeric operation between two primitive-typed arrays. +pub(super) fn execute_numeric_primitive( + lhs: &ArrayRef, + rhs: &ArrayRef, + op: NumericOperator, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = VecExecutionArgs::new(vec![lhs.clone(), rhs.clone()], lhs.len()); + + ScalarFnVTable::execute(&NumericBinary, &op, &args, ctx) +} + +/// The primitive arithmetic operators as a row function. +#[derive(Clone)] +struct NumericBinary; + +impl RowFn for NumericBinary { + type Options = NumericOperator; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + // Fallibility is declared before dispatch knows the primitive width. The float widths inherit + // this conservative declaration at no execution cost. + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.numeric_binary"); + *ID + } + + fn dispatch( + &self, + op: &Self::Options, + args: &[DType], + visitor: Visitor, + ) -> VortexResult { + let ptype = operand_ptype(args)?; + + match_each_native_ptype!(ptype, |Primitive| { + match op { + NumericOperator::Add => visit_checked::(visitor), + NumericOperator::Sub => visit_checked::(visitor), + NumericOperator::Mul => visit_checked::(visitor), + NumericOperator::Div => visit_checked::(visitor), + } + }) + } +} + +/// Return the primitive width selected by the left operand. +/// +/// The visited `(Primitive, Primitive)` tuple validates both operands against this width. +fn operand_ptype(args: &[DType]) -> VortexResult { + let lhs = args + .first() + .ok_or_else(|| vortex_err!("a numeric operator takes two operands, got none"))?; + + PType::try_from(lhs) +} + +/// Visit two primitive columns and defer one OR-reducible failure word per row. +fn visit_checked(visitor: Visitor) -> VortexResult +where + Primitive: NativePType, + Operator: CheckedPrimitiveOp, + Visitor: RowVisitor, +{ + visitor.visit_prepared_deferred::<(Primitive, Primitive), Primitive, _, Operator::Failure>( + |_| (), + |&(), (lhs, rhs)| Operator::apply(lhs, rhs), + |failure| { + if failure != ::default() { + return Err(vortex_err!(InvalidArgument: "{}", Operator::ERROR)); + } + + Ok(()) + }, + ) +} diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs b/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs index 7ee98ae717e..08d3e2daca9 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs @@ -297,13 +297,13 @@ fn test_multiply_overflow_on_null_lane_ignored( Ok(()) } -/// The hot pass OR-reduces evidence across whole 64-lane chunks before anything looks at it, so an -/// overflow in a late chunk must still be caught, and must still be suppressed when its lane is -/// null. Every other test here fits in a single chunk and cannot show either. +/// The hot pass OR-reduces evidence across the whole row loop before anything looks at it, so an +/// overflow late in the batch must still be caught, and must still be suppressed when its lane is +/// null. #[rstest] #[case::reported(true)] #[case::suppressed_by_null(false)] -fn test_multiply_overflow_survives_chunk_reduction( +fn test_multiply_overflow_survives_batch_reduction( #[case] lane_is_valid: bool, ) -> VortexResult<()> { const LEN: u32 = 1000; diff --git a/vortex-array/src/scalar_fn/fns/list_length.rs b/vortex-array/src/scalar_fn/fns/list_length.rs index 415a65416db..a971fe71141 100644 --- a/vortex-array/src/scalar_fn/fns/list_length.rs +++ b/vortex-array/src/scalar_fn/fns/list_length.rs @@ -350,6 +350,25 @@ mod tests { Ok(()) } + /// A non-nullable fixed-size list has one length for the whole column, so the result stays a + /// constant rather than materializing one `u64` per row. + #[test] + fn test_fixed_size_list_length_stays_constant() -> VortexResult<()> { + let fsl = create_fixed_size_list(Validity::NonNullable); + let mut ctx = array_session().create_execution_ctx(); + + let result = fsl + .apply(&list_length(root()))? + .execute::(&mut ctx)?; + + assert_eq!( + result.as_constant(), + Some(Scalar::primitive(2u64, Nullability::NonNullable)), + "expected a constant length column" + ); + Ok(()) + } + #[test] fn test_fixed_size_list_length_nullable() -> VortexResult<()> { let fsl = create_fixed_size_list(Validity::Array( diff --git a/vortex-array/src/scalar_fn/mod.rs b/vortex-array/src/scalar_fn/mod.rs index 003dbc2ca48..11bcefe0325 100644 --- a/vortex-array/src/scalar_fn/mod.rs +++ b/vortex-array/src/scalar_fn/mod.rs @@ -6,6 +6,68 @@ //! This module contains the [`ScalarFnVTable`] trait and all built-in scalar function //! implementations. Expressions ([`crate::expr::Expression`]) reference scalar functions //! at each node. +//! +//! # Choosing a trait +//! +//! Two traits reach this vtable, and [`RowFn`] derives the whole of [`ScalarFnVTable`] from a row +//! closure. Implement `RowFn` when the function fits it, and `ScalarFnVTable` when it does not. +//! +//! [`RowFn`] is for a kernel whose value at a row is determined by that row alone, and which has to +//! read every row anyway: the arithmetic operators over primitive columns, `vortex.tensor.l2_norm`, +//! `vortex.tensor.inner_product`, `vortex.tensor.cosine_similarity`, `vortex.geo.distance`, +//! `vortex.geo.contains`. Name the element types and write the row closure, and the rest is +//! derived, including which rows get visited. +//! +//! Its *input* side is open. [`InputElement::Elem`] is a GAT, so an element can hand the closure +//! borrowed variable-length data (a byte-string element yielding `&[u8]`) or drill through a wrapper +//! (`vortex-tensor`'s `TensorRow` yields a slice of an extension array's storage). Covering a new +//! type family, a list row included, is one impl. +//! +//! Its output is always an [`OutputSink`], allocated once per batch and handing the closure one row +//! to write. [`ElementSink`] is the standard sink for one owned [`OutputElement`] per row. A custom +//! sink carries runtime-shaped output, such as a tensor whose width comes from its input dtype, or a +//! future string transform appending every row into one shared byte buffer. +//! +//! When part of the kernel's work depends only on an operand that is constant for the batch (the +//! norm of a broadcast query vector, a prepared form of a constant geometry), do that work in +//! [`RowVisitor::visit_prepared_into`]'s once-per-batch prepare step. Pass `|_| ()` when there is +//! nothing to prepare. Prepare **must not** be load-bearing for validation: an empty batch decodes +//! every operand as non-constant, so a prepare that validated its constant would silently not run. +//! +//! Null handling is derived too, null-strategy selection included: a nullable batch runs densely +//! (compute every row, mask after), by branch-and-skip (decode full length, compute only the +//! conjoined-valid rows, mask after), or by filtering (shrink the inputs to the valid rows, +//! compute, scatter back), and the framework picks per batch. Function authors do nothing. The one +//! input to that choice an element controls is [`InputElement::FILTERED_DECODE_COST`]: set it when +//! decoding a column does expensive per-row work (parsing a geometry), so sparse batches keep the +//! filter strategy's shrunken decode. Costs from separate arguments are additive. +//! +//! Two things no output sink covers, and they are what actually send a function to +//! [`ScalarFnVTable`]: +//! +//! - **A result that aliases an input.** Sinks own their output bytes. Trimming strings is the +//! example, where the ideal kernel keeps the input's data buffer and writes new views over it, +//! copying no bytes, which only a columnar kernel can express. +//! - **A null result for a non-null row.** Sinks build an all-valid column, so +//! `vortex.list.sum` cannot be a row function: a valid empty list sums to null. +//! +//! [`ScalarFnVTable`] takes the whole column instead, and everything a row function gets derived is +//! then hand-written: null propagation, constant folding, nullability, validity, and options serde. +//! Besides the two cases above and the functions that are simply not strict (Kleene logic, or a +//! strictness that depends on the options), reach for it when a row loop *could* express the +//! function but would do avoidable work: +//! +//! - **The answer is already an array, or is one value for the whole column.** +//! `vortex.list.length` hands back a `ListViewArray`'s sizes child, and a single `ConstantArray` +//! for a `FixedSizeListArray`. A row loop would rebuild that one `u64` at a time, even given a +//! list-length element that reads the size out of the layout rather than the list. +//! - **A row is not the natural unit of work.** `vortex.not` is one `!` per 64-bit word, in place +//! when the bit buffer is unshared, against 64 loop iterations and 64 bit writes, and its +//! encoding-aware fallback pushes the inversion down instead of canonicalizing. +//! - **The row's value is cheaper to read than the row.** `vortex.byte_length` was tried as a row +//! function and measured 7.6x slower than its columnar implementation, because the length is a +//! field of the view and the row loop paid to resolve the bytes it never looked at. Being +//! row-determined is necessary but not sufficient. use vortex_session::registry::Id; @@ -35,6 +97,9 @@ pub use options::*; mod signature; pub use signature::*; +mod row; +pub use row::*; + pub mod fns; pub mod internal; pub mod session; diff --git a/vortex-array/src/scalar_fn/row/element/bool.rs b/vortex-array/src/scalar_fn/row/element/bool.rs new file mode 100644 index 00000000000..d4fc51c769c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/bool.rs @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_buffer::BitBuffer; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::BoolArray; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar_fn::InputElement; +use crate::scalar_fn::OutputElement; +use crate::validity::Validity; + +impl InputElement for bool { + type Column = BitBuffer; + type Varying<'a> = &'a BitBuffer; + type Elem<'a> = bool; + + // Every bit of the buffer is readable, valid or not. + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + matches!(dtype, DType::Bool(_)), + "expected a Bool column, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(array.execute::(ctx)?.into_bit_buffer()) + } + + fn get(column: &Self::Column, index: usize) -> bool { + column.value(index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> bool + where + Self: 'a, + { + column.value(index) + } +} + +impl OutputElement for bool { + fn element_dtype() -> DType { + DType::Bool(Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + // `From>` packs through the multiversioned SIMD path; `from_iter` would set one + // bit at a time, which measures 6.6-7.9x slower on the packing step alone. + BoolArray::new(BitBuffer::from(values), Validity::NonNullable).into_array() + } + + fn placeholder() -> Self { + false + } +} diff --git a/vortex-array/src/scalar_fn/row/element/conformance.rs b/vortex-array/src/scalar_fn/row/element/conformance.rs new file mode 100644 index 00000000000..45687ea9c1a --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/conformance.rs @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! A shared conformance check every [`InputElement`] should be run through. + +use std::hint::black_box; + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::InputElement; + +/// Assert that `E` honors its [`InputElement`] contract over `array`, and rejects `rejected_dtype`. +/// +/// The part worth checking mechanically is [`InputElement::DENSE_SAFE`]. An element claiming it will +/// be read at rows that are *null*, where an array guarantees nothing about the payload, and getting +/// the `const` wrong is either an out-of-bounds panic in production (the failure mode of +/// [#9090](https://github.com/vortex-data/vortex/issues/9090)) or an unnecessary valid-only +/// execution path. Nothing else verifies it, since the framework reads the `const` rather than +/// testing the claim. +/// +/// So `array` **must** contain at least one null row, and its payload behind those nulls **must** be +/// deliberately extreme rather than zeroed, or the check passes vacuously. Build that safely by +/// putting the extreme values in the array first and masking those rows afterwards, as the callers of +/// this function do. +/// +/// What this cannot check: [`DECODE_FALLIBLE`](InputElement::DECODE_FALLIBLE), which needs data that +/// is legal but malformed, and whether `validate` accepts everything it *should*, since only the +/// element knows its full dtype domain. Pass one representative rejection. +#[track_caller] +pub fn assert_element_conforms( + array: ArrayRef, + rejected_dtype: &DType, + ctx: &mut ExecutionCtx, +) -> VortexResult<()> { + let dtype = array.dtype().clone(); + E::validate(&dtype)?; + + assert!( + E::validate(rejected_dtype).is_err(), + "element accepted {rejected_dtype}, which it was expected to reject", + ); + + let len = array.len(); + let valid = array.validity()?.execute_mask(len, ctx)?; + assert!( + !valid.all_true(), + "conformance needs a null row to read behind, but every row of the {dtype} input is valid", + ); + + let column = E::decode(array, ctx)?; + let varying = E::varying(&column); + assert_eq!( + E::varying_len(&varying), + len, + "varying element view changed the decoded row count", + ); + + // The claim under test. Reading a null row may yield garbage, but it must not fault, so an + // element that secretly follows a per-row offset panics here instead of in production. + if E::DENSE_SAFE { + for index in 0..len { + black_box(E::get(&column, index)); + black_box(E::get_varying(&varying, index)); + } + } else { + for index in 0..len { + if valid.value(index) { + black_box(E::get(&column, index)); + black_box(E::get_varying(&varying, index)); + } + } + } + + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/element/mod.rs b/vortex-array/src/scalar_fn/row/element/mod.rs new file mode 100644 index 00000000000..384d8bc3aa7 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/mod.rs @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The element types a row function can read and produce. +//! +//! Both traits are open, and this module holds one file per type family, so covering a new one is a +//! sibling file and every row function gains it. The families are not confined to this crate: +//! `vortex-tensor`'s `TensorRow` drills through an extension wrapper into its storage. +//! +//! The two directions are deliberately asymmetric. [`InputElement::Elem`] is a GAT, so an input row +//! can borrow out of the decoded column, while an [`OutputElement`] is one owned value returned by +//! an owned row computation or written through +//! [`ElementSink`](crate::scalar_fn::ElementSink); runtime-shaped output uses a custom +//! [`OutputSink`](crate::scalar_fn::OutputSink). + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; + +mod bool; + +#[cfg(any(test, feature = "_test-harness"))] +mod conformance; +#[cfg(any(test, feature = "_test-harness"))] +pub use conformance::assert_element_conforms; + +mod primitive; + +mod tuple; +pub use tuple::ElementTuple; +pub use tuple::IndexedElementTuple; +pub(super) use tuple::batch_constant; + +/// An element type that can be read row-wise out of an input column. +pub trait InputElement: 'static { + /// The decoded column representation supporting `O(1)` row access. + type Column; + + /// The view of a varying decoded column read by the hot row loop. + /// + /// This may borrow a cheaper representation than [`Column`](Self::Column). Primitive elements, + /// for example, expose a slice so its pointer and length are loop invariants rather than + /// re-reading a [`Buffer`](vortex_buffer::Buffer) descriptor for every row. + type Varying<'a>; + + /// The borrowed element value handed to the row closure a [`RowFn`](crate::scalar_fn::RowFn) + /// visits with. + type Elem<'a>; + + /// Whether [`decode`](Self::decode) and [`get`](Self::get) tolerate rows that are null in the + /// input. + /// + /// Arrays only guarantee their contents for _valid_ rows, so this is `false` for any element + /// that follows an offset or pointer stored in the array: behind a null row that value is + /// arbitrary and may not address anything. Reading a whole value out of a flat buffer is `true`, + /// since the value is garbage but the read cannot fault. + /// + /// Dense execution requires this of every argument; otherwise the row layer executes only + /// valid rows. + const DENSE_SAFE: bool = false; + + /// Whether [`decode`](Self::decode) can fail on _legal_ input data. + /// + /// `false` for an element read straight out of a buffer: decoding can still fail for + /// infrastructural reasons (IO, allocation), but never because of the values. `true` for an + /// element that parses its bytes, since a malformed WKB geometry in a _valid_ row is a domain + /// error, which makes a function over that element + /// [fallible](crate::scalar_fn::ScalarFnVTable::is_fallible) however infallible its own row + /// computation is. + const DECODE_FALLIBLE: bool = true; + + /// A relative unit count for per-row decode work avoided by filtering this argument first. + /// + /// Use `1` for an element whose decode _parses_ every row (a geometry built from coordinate + /// storage): decoding only the survivors of a sparse validity mask is genuinely cheaper than + /// decoding everyone. Keep the default `0` for a bulk canonicalization (bytes, bools, + /// primitives), whose decode is a memcpy-shaped pass that filtering barely shrinks. Larger + /// values may express a proportionally more expensive decode. + /// + /// The lifting reads this when it picks a null strategy for a batch with a mixed + /// validity mask: filtering the inputs first only pays off when it shrinks a per-row decode, + /// so elements that leave this at zero always take the cheaper branch-and-skip strategy. + /// Getting it wrong is a performance bug, never a correctness bug. + const FILTERED_DECODE_COST: usize = 0; + + /// Validate that `dtype` is an acceptable input column dtype for this element type. + fn validate(dtype: &DType) -> VortexResult<()>; + + /// Decode `array` into its column representation. Called once per batch. + /// + /// This is where every per-batch cost belongs: resolving the dtype, downcasting the buffer, + /// checking the ptype, and anything else that does not vary by row. [`Column`](Self::Column) is + /// the type to widen if that means carrying more, since it is chosen by the element. + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Decode `array` _without_ assuming every row is valid, or `Ok(None)` when this element + /// cannot for this particular array. + /// + /// An element with [`DENSE_SAFE`](Self::DENSE_SAFE) set **should not** override this: its + /// ordinary decode already tolerates null payloads, so the default is already correct and an + /// override just restates it. Overriding is for an element that is *not* dense-safe but can + /// still write an arbitrary placeholder into null slots; the caller guarantees + /// [`get`](Self::get) is never called for such a row. It is what the branch-and-skip null + /// strategy decodes with. + /// + /// Return `Ok(None)` rather than an error when an array has no null-tolerant decode; the lifting + /// falls back to the filter strategy. + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if Self::DENSE_SAFE { + Self::decode(array, ctx).map(Some) + } else { + Ok(None) + } + } + + /// Read the element at `index`, the one function called once per row. + /// + /// `O(1)` is necessary but **not sufficient**: this must not repeat work that is constant across + /// the batch, however cheap that work looks per call. An `O(1)` ptype check and buffer downcast + /// per row cost `l2_norm` 2x at width 2, invisible in the call because it read like a getter. Do + /// that work in [`decode`](Self::decode) and leave this an offset computation. + fn get(column: &Self::Column, index: usize) -> Self::Elem<'_>; + + /// Borrow the representation used when this argument varies within the batch. + /// + /// Called once before the hot loop. Constants do not use this view because the tuple adapter + /// keeps their one-row decoded representation separate. + fn varying(column: &Self::Column) -> Self::Varying<'_>; + + /// Number of rows addressable through a [`Varying`](Self::Varying) view. + fn varying_len(column: &Self::Varying<'_>) -> usize; + + /// Read one row from a [`Varying`](Self::Varying) view. + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> Self::Elem<'a> + where + Self: 'a; +} + +/// An element type that a row computation can produce, buildable into an all-valid column. +/// +/// [`Clone`] is required so [`ElementSink`](crate::scalar_fn::ElementSink) can allocate through +/// `vec![placeholder; rows]`, which is what lets a zero placeholder reach the allocator's zeroed +/// path instead of costing a write pass over the output. +pub trait OutputElement: 'static + Sized + Clone { + /// The dtype of columns built from this element type. Must be non-nullable: nullability is + /// derived from the inputs by the lifting. + /// + /// Taking no arguments confines an element's dtype to a property of its Rust type, so an output + /// whose dtype depends on runtime data (a tensor, whose dtype carries its shape) cannot be an + /// element. Such an output uses an [`OutputSink`](crate::scalar_fn::OutputSink), whose + /// [`sink_dtype`](crate::scalar_fn::OutputSink::sink_dtype) does see the input dtypes. + fn element_dtype() -> DType; + + /// Build a column from one value per row. Called once per batch. + fn build(values: Vec) -> ArrayRef; + + /// An arbitrary value of this element, pre-filled into the output slots that the + /// branch-and-skip null strategy skips. + /// + /// The value is never observable: the lifting masks every slot holding it before the + /// result escapes. It only has to be cheap to construct and legal to + /// [`build`](Self::build) with. + fn placeholder() -> Self; +} diff --git a/vortex-array/src/scalar_fn/row/element/primitive.rs b/vortex-array/src/scalar_fn/row/element/primitive.rs new file mode 100644 index 00000000000..d54c922bf2d --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/primitive.rs @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure_eq; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::PrimitiveArray; +use crate::dtype::DType; +use crate::dtype::NativePType; +use crate::dtype::Nullability; +use crate::scalar_fn::InputElement; +use crate::scalar_fn::OutputElement; +use crate::validity::Validity; + +impl InputElement for T { + type Column = Buffer; + type Varying<'a> = &'a [T]; + type Elem<'a> = T; + + // Every lane of the buffer holds a `T`, valid or not. + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + let expected = T::PTYPE; + let DType::Primitive(ptype, _) = dtype else { + vortex_bail!("expected a {expected} column, got {dtype}"); + }; + vortex_ensure_eq!( + *ptype, + expected, + "expected a {expected} column, got {dtype}" + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(array.execute::(ctx)?.into_buffer::()) + } + + fn get(column: &Self::Column, index: usize) -> T { + column[index] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column.as_slice() + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> T + where + Self: 'a, + { + column[index] + } +} + +impl OutputElement for T { + fn element_dtype() -> DType { + DType::Primitive(T::PTYPE, Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + PrimitiveArray::new(values, Validity::NonNullable).into_array() + } + + fn placeholder() -> Self { + T::default() + } +} diff --git a/vortex-array/src/scalar_fn/row/element/tuple.rs b/vortex-array/src/scalar_fn/row/element/tuple.rs new file mode 100644 index 00000000000..fad02639e1a --- /dev/null +++ b/vortex-array/src/scalar_fn/row/element/tuple.rs @@ -0,0 +1,396 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Argument lists built from [`InputElement`]s, and the per-argument decode behind them. + +use vortex_compute::lane_kernels::IndexedSource; +use vortex_compute::lane_kernels::LaneZip; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::arrays::Extension; +use crate::arrays::Masked; +use crate::arrays::extension::ExtensionArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; +use crate::dtype::DType; +use crate::dtype::NativePType; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::InputElement; + +mod private { + pub trait Sealed {} +} + +/// One decoded input column of an [`ElementTuple`]. +/// +/// A constant operand holds the same value in every row, so it is decoded once as a single row and +/// read at index 0 forever. That is what stops a constant argument costing one decode per row, which +/// matters whenever the decode is more than a buffer read: parsing a geometry from WKB, or +/// canonicalizing an extension row. +pub struct ArgColumn(ArgColumnKind); + +enum ArgColumnKind { + Varying(T::Column), + Constant(T::Column), +} + +impl ArgColumn { + /// Decode one input column, collapsing a constant operand to its single distinct row. + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + // An empty input has no row 0 to slice, and its row loop runs zero times either way. + if let Some(constant) = batch_constant(&array) + && !array.is_empty() + { + return Ok(Self(ArgColumnKind::Constant(T::decode( + constant.slice(0..1)?, + ctx, + )?))); + } + + Ok(Self(ArgColumnKind::Varying(T::decode(array, ctx)?))) + } + + /// Like [`decode`](Self::decode), but a varying column decodes null-tolerantly through + /// [`InputElement::decode_null_tolerant`]. `Ok(None)` means the element cannot, and the + /// caller falls back to the filter strategy. + /// + /// A constant operand still takes the ordinary decode: the lifting short-circuits null + /// constants before any strategy runs, so a constant reaching here is non-null. + fn decode_null_tolerant(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult> { + if let Some(constant) = batch_constant(&array) + && !array.is_empty() + { + return Ok(Some(Self(ArgColumnKind::Constant(T::decode( + constant.slice(0..1)?, + ctx, + )?)))); + } + + Ok(T::decode_null_tolerant(array, ctx)? + .map(ArgColumnKind::Varying) + .map(Self)) + } + + /// Read the element at `index`, which for a constant operand is always its single row. + fn get(&self, index: usize) -> T::Elem<'_> { + match &self.0 { + ArgColumnKind::Varying(column) => T::get(column, index), + ArgColumnKind::Constant(column) => T::get(column, 0), + } + } + + /// The decoded full column, or `None` when this argument was collapsed to one constant row. + fn varying(&self) -> Option<&T::Column> { + match &self.0 { + ArgColumnKind::Varying(column) => Some(column), + ArgColumnKind::Constant(_) => None, + } + } + + /// Whether this argument addresses exactly `row_count` rows. + /// + /// A constant operand was collapsed to its one distinct row and is read at index 0 forever, so + /// it addresses any row count and is exempt. + fn addresses_rows(&self, row_count: usize) -> bool { + match &self.0 { + ArgColumnKind::Varying(column) => T::varying_len(&T::varying(column)) == row_count, + ArgColumnKind::Constant(_) => true, + } + } + + /// The single decoded element of a constant operand, or `None` for a real column. + /// + /// `Some` exactly when [`decode`](Self::decode) collapsed the operand to its one distinct row, + /// in which case the value returned is the element every row of the batch reads. + fn constant(&self) -> Option> { + match &self.0 { + ArgColumnKind::Varying(_) => None, + ArgColumnKind::Constant(column) => Some(T::get(column, 0)), + } + } +} + +/// The array whose every row holds one distinct value, when `array` is constant for the batch. +/// +/// Beyond the constant encoding itself this sees one level through two wrappers that spell "the +/// same value in every row" without being the constant encoding: +/// +/// - [`Masked`], how the compressor spells an all-same-with-nulls chunk: the child carries the +/// value, the wrapper carries only validity. Reading the child's value for a null row is sound +/// here because the lifting owns validity entirely; the row loop's output behind a null +/// row is masked away (dense) or never computed (filter), so which value the loop read there +/// cannot be observed. An all-null constant never reaches decode at all, since the lifting +/// short-circuits it to an all-null result first. +/// - [`Extension`] over constant storage, the shape an extension-typed builder produces before +/// `ExtensionConstantRule` normalizes it to a top-level constant. Every row wraps the same +/// storage value, so the whole array (sliced to one row, keeping its extension dtype) is the +/// constant. +pub(in crate::scalar_fn::row) fn batch_constant(array: &ArrayRef) -> Option { + if array.as_constant().is_some() { + return Some(array.clone()); + } + + if let Some(masked) = array.as_opt::() { + return Some(masked.child().clone()).filter(|child| child.as_constant().is_some()); + } + + array + .as_opt::() + .is_some_and(|ext| ext.storage_array().as_constant().is_some()) + .then(|| array.clone()) +} + +/// Tuples of [`InputElement`]s forming the typed argument list a [`RowFn`](crate::scalar_fn::RowFn) +/// visits with. Implemented for `()` and tuples of one through twelve elements. This trait is +/// framework-only; add a new decode primitive by implementing [`InputElement`], then use it inside +/// one of those tuples. +/// +/// The arities past the widest function in tree are deliberate. This trait is **sealed**, so a +/// downstream crate cannot add the one it needs, and an unused arity costs only its own macro +/// expansion: no monomorphization happens until something instantiates it. +pub trait ElementTuple: 'static + private::Sealed { + /// The decoded column representations. + type Columns; + + /// Direct references to decoded columns when every argument varies within the batch. + type VaryingColumns<'a>; + + /// The borrowed row of element values. + type Elems<'a>; + + /// The batch-constant element values: [`Elems`](Self::Elems) with every argument wrapped in + /// `Option`. + /// + /// `Some` marks an argument whose operand is constant for the batch and carries the element + /// every row reads; `None` marks one that varies by row. This is what + /// [`RowVisitor`](crate::scalar_fn::RowVisitor) hands to a visit's prepare closure, so a kernel + /// can hoist work that depends only on a constant argument out of the row loop. + type ConstElems<'a>; + + /// The number of arguments. + const ARITY: usize; + + /// Whether every argument is [`InputElement::DENSE_SAFE`]. + const DENSE_SAFE: bool; + + /// Whether _any_ argument is [`InputElement::DECODE_FALLIBLE`]. + const DECODE_FALLIBLE: bool; + + /// The additive cost of per-row decode work avoided by filtering the arguments first. + const FILTERED_DECODE_COST: usize; + + /// Validate the input dtypes, including that `dtypes` has exactly `ARITY` entries. + /// + /// The expression layer checks the count against [`Arity`](crate::scalar_fn::Arity) before it + /// builds a call, but this is also the entry point of the public + /// [`return_dtype`](crate::scalar_fn::ScalarFnVTable::return_dtype), so the count is enforced + /// here rather than assumed. + fn validate(dtypes: &[DType]) -> VortexResult<()>; + + /// Decode every input column once. Called once per batch. + fn decode(args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx) -> VortexResult; + + /// Decode every input column once, tolerating null rows, or `Ok(None)` when some argument + /// cannot. Called once per batch by the branch-and-skip null strategy. + fn decode_null_tolerant( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult>; + + /// Read the row of elements at `index`. Must be `O(1)`: it is called in the row loop. + fn get(columns: &Self::Columns, index: usize) -> Self::Elems<'_>; + + /// Borrow every decoded column directly, or `None` when any argument is batch-constant. + /// + /// This is selected once outside the hot loop. Keeping `ArgColumn` out of the resulting tuple + /// gives the optimizer ordinary contiguous column access without a per-row constant check. + fn varying(columns: &Self::Columns) -> Option>; + + /// Whether every varying column contains exactly `row_count` rows. + fn varying_len_matches(columns: &Self::VaryingColumns<'_>, row_count: usize) -> bool; + + /// Whether every argument that varies within the batch contains exactly `row_count` rows. + /// + /// The same guarantee as [`varying_len_matches`](Self::varying_len_matches), for the mixed case + /// [`varying`](Self::varying) declines: a batch-constant argument is exempt because it was + /// collapsed to one row, while every argument beside it still has to address the whole batch. + fn decoded_lens_match(columns: &Self::Columns, row_count: usize) -> bool; + + /// Read one row from columns already known to vary within the batch. + fn get_varying<'a>(columns: &Self::VaryingColumns<'a>, index: usize) -> Self::Elems<'a>; + + /// Read the batch-constant elements out of the decoded columns. Called once per batch. + fn constants(columns: &Self::Columns) -> Self::ConstElems<'_>; +} + +/// An argument tuple that can expose independent indexed reads after one length validation. +/// +/// This trait is sealed through [`ElementTuple`]. Tuples without a natural indexed source continue +/// to use ordinary row access and output sinks. +pub trait IndexedElementTuple: ElementTuple { + /// The source shared execution uses for a dense all-varying loop. + /// + /// Its length must be the common varying-column length. For every valid index it must preserve + /// row order, return the same value as [`ElementTuple::get_varying`], and uphold the unchecked + /// read contract of [`IndexedSource`]. + type Source<'a>: IndexedSource>; + + /// Borrow a source from columns already validated to vary within the batch. + fn indexed_source<'a>(columns: &Self::VaryingColumns<'a>) -> Self::Source<'a>; +} + +impl private::Sealed for () {} + +impl ElementTuple for () { + type Columns = (); + type VaryingColumns<'a> = (); + type Elems<'a> = (); + type ConstElems<'a> = (); + + const ARITY: usize = 0; + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + const FILTERED_DECODE_COST: usize = 0; + + fn validate(dtypes: &[DType]) -> VortexResult<()> { + vortex_ensure_eq!( + dtypes.len(), + 0, + "expected 0 argument dtypes, got {}", + dtypes.len(), + ); + Ok(()) + } + + fn decode(_args: &dyn ExecutionArgs, _ctx: &mut ExecutionCtx) -> VortexResult { + Ok(()) + } + + fn decode_null_tolerant( + _args: &dyn ExecutionArgs, + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(())) + } + + fn get(_columns: &Self::Columns, _index: usize) -> Self::Elems<'_> {} + + fn varying(_columns: &Self::Columns) -> Option> { + Some(()) + } + + fn varying_len_matches(_columns: &Self::VaryingColumns<'_>, _row_count: usize) -> bool { + true + } + + fn decoded_lens_match(_columns: &Self::Columns, _row_count: usize) -> bool { + true + } + + fn get_varying<'a>(_columns: &Self::VaryingColumns<'a>, _index: usize) -> Self::Elems<'a> {} + + fn constants(_columns: &Self::Columns) -> Self::ConstElems<'_> {} +} + +macro_rules! element_tuple { + ($arity:literal; $($t:ident : $idx:tt),+) => { + impl<$($t: InputElement),+> private::Sealed for ($($t,)+) {} + + impl<$($t: InputElement),+> ElementTuple for ($($t,)+) { + type Columns = ($(ArgColumn<$t>,)+); + type VaryingColumns<'a> = ($($t::Varying<'a>,)+); + type Elems<'a> = ($($t::Elem<'a>,)+); + type ConstElems<'a> = ($(Option<$t::Elem<'a>>,)+); + + const ARITY: usize = $arity; + const DENSE_SAFE: bool = $($t::DENSE_SAFE &&)+ true; + const DECODE_FALLIBLE: bool = $($t::DECODE_FALLIBLE ||)+ false; + const FILTERED_DECODE_COST: usize = $($t::FILTERED_DECODE_COST +)+ 0; + + fn validate(dtypes: &[DType]) -> VortexResult<()> { + vortex_ensure_eq!( + dtypes.len(), + $arity, + "expected {} argument dtypes, got {}", + $arity, + dtypes.len(), + ); + + $($t::validate(&dtypes[$idx])?;)+ + Ok(()) + } + + fn decode( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + Ok(($(ArgColumn::<$t>::decode(args.get($idx)?, ctx)?,)+)) + } + + fn decode_null_tolerant( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(( + $(match ArgColumn::<$t>::decode_null_tolerant(args.get($idx)?, ctx)? { + Some(column) => column, + None => return Ok(None), + },)+ + ))) + } + + fn get(columns: &Self::Columns, index: usize) -> Self::Elems<'_> { + ($(columns.$idx.get(index),)+) + } + + fn varying(columns: &Self::Columns) -> Option> { + Some(($($t::varying(columns.$idx.varying()?),)+)) + } + + fn varying_len_matches( + columns: &Self::VaryingColumns<'_>, + row_count: usize, + ) -> bool { + $($t::varying_len(&columns.$idx) == row_count &&)+ true + } + + fn decoded_lens_match(columns: &Self::Columns, row_count: usize) -> bool { + $(columns.$idx.addresses_rows(row_count) &&)+ true + } + + fn get_varying<'a>( + columns: &Self::VaryingColumns<'a>, + index: usize, + ) -> Self::Elems<'a> { + ($($t::get_varying(&columns.$idx, index),)+) + } + + fn constants(columns: &Self::Columns) -> Self::ConstElems<'_> { + ($(columns.$idx.constant(),)+) + } + } + }; +} + +element_tuple!(1; A:0); +element_tuple!(2; A:0, B:1); +element_tuple!(3; A:0, B:1, C:2); +element_tuple!(4; A:0, B:1, C:2, D:3); +element_tuple!(5; A:0, B:1, C:2, D:3, E:4); +element_tuple!(6; A:0, B:1, C:2, D:3, E:4, F:5); +element_tuple!(7; A:0, B:1, C:2, D:3, E:4, F:5, G:6); +element_tuple!(8; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7); +element_tuple!(9; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8); +element_tuple!(10; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9); +element_tuple!(11; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10); +element_tuple!(12; A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7, I:8, J:9, K:10, L:11); + +impl IndexedElementTuple for (A, B) { + type Source<'a> = LaneZip<&'a [A], &'a [B]>; + + fn indexed_source<'a>(columns: &Self::VaryingColumns<'a>) -> Self::Source<'a> { + LaneZip::new(columns.0, columns.1) + } +} diff --git a/vortex-array/src/scalar_fn/row/execute.rs b/vortex-array/src/scalar_fn/row/execute.rs new file mode 100644 index 00000000000..a92e3e43998 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/execute.rs @@ -0,0 +1,277 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The pieces every row function is built from, whatever its `dispatch` chooses. +//! +//! These back the blanket impls in [`row_fn`](super::row_fn) and are deliberately not public: +//! [`RowFn`](crate::scalar_fn::RowFn) is the abstraction, these are its internals. + +use std::mem::needs_drop; +use std::ops::BitOrAssign; + +use vortex_compute::lane_kernels::IndexedSourceExt; +use vortex_error::VortexError; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_mask::AllOr; +use vortex_mask::Mask; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::DeferredError; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::IndexedElementTuple; +use crate::scalar_fn::OutputElement; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::SinkResult; + +/// The value path out of a row executor, keeping a deferred row error distinct from structural +/// execution errors so nullable lifting retries only the former. +pub(super) enum RowExecution { + /// A successfully built output column. + Output(ArrayRef), + + /// A batch-wide row error that nullable lifting may retry over only the valid rows. + DeferredError(VortexError), +} + +impl RowExecution { + /// Return the output or surface its deferred row error. + pub(super) fn into_result(self) -> VortexResult { + match self { + Self::Output(output) => Ok(output), + Self::DeferredError(error) => Err(error), + } + } +} + +/// Validate the input dtypes of an owned-output row function and return its output dtype. +pub(super) fn validate_row_output( + args: &[DType], +) -> VortexResult { + A::validate(args)?; + let dtype = O::element_dtype(); + vortex_ensure!( + !dtype.is_nullable(), + "row output elements must declare a non-nullable dtype, got {dtype}", + ); + Ok(dtype) +} + +/// Validate the input dtypes of a sink-writing row function and return the dtype its sink builds. +/// +/// The output dtype may be a function of the inputs. A sink can also own a batch-wide builder, such +/// as the shared byte and view buffers of a future string transform. +pub(super) fn validate_row_sink( + args: &[DType], +) -> VortexResult { + A::validate(args)?; + let dtype = S::sink_dtype(args)?; + vortex_ensure!( + !dtype.is_nullable(), + "row output sinks must declare a non-nullable dtype, got {dtype}", + ); + Ok(dtype) +} + +/// Decode every input column once, then store owned row outputs and reduce deferred failures. +pub(super) fn execute_row_output_prepared( + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, F), + finish_failure: impl FnOnce(F) -> VortexResult<()>, +) -> VortexResult +where + A: IndexedElementTuple, + O: OutputElement, + F: Copy + Default + BitOrAssign, +{ + const { + assert!( + !needs_drop::(), + "owned deferred outputs must not require drop glue" + ) + }; + + let row_count = args.row_count(); + let mut values = Vec::::with_capacity(row_count); + let columns = A::decode(args, ctx)?; + let state = prepare(A::constants(&columns)); + let failed; + + { + let output = &mut values.spare_capacity_mut()[..row_count]; + + if let Some(varying) = A::varying(&columns) { + vortex_ensure!( + A::varying_len_matches(&varying, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + failed = + A::indexed_source(&varying).map_checked_into(output, |elems| apply(&state, elems)); + } else { + vortex_ensure!( + A::decoded_lens_match(&columns, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + let mut accumulated = F::default(); + for index in 0..row_count { + let (value, failure) = apply(&state, A::get(&columns, index)); + output[index].write(value); + accumulated |= failure; + } + failed = accumulated; + } + } + + // SAFETY: normal completion of either loop initializes every slot in `0..row_count` exactly + // once, and `values` was allocated with at least `row_count` capacity. + unsafe { values.set_len(row_count) }; + + match finish_failure(failed) { + Ok(()) => Ok(RowExecution::Output(O::build(values))), + Err(error) => Ok(RowExecution::DeferredError(error)), + } +} + +/// Decode every input column once, allocate the sink once, then write one row at a time. +/// +/// The sink lives here rather than in the closure, so `apply` stays [`Fn`] and the loop keeps the +/// unconditional shape that lets it vectorize. Monomorphic in `A`, `S` and `R`, so `apply` and +/// [`OutputSink::row`] both inline. +pub(super) fn execute_row_sink_prepared( + args: &dyn ExecutionArgs, + sink_dtype: &DType, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult { + let row_count = args.row_count(); + let mut sink = S::with_capacity(row_count, sink_dtype)?; + let columns = A::decode(args, ctx)?; + let state = prepare(A::constants(&columns)); + let mut accumulated = R::Accumulated::default(); + + { + let mut rows = sink.rows(); + vortex_ensure!( + S::row_count_matches(&rows, row_count), + "the output sink does not address exactly {row_count} rows", + ); + + if let Some(varying) = A::varying(&columns) { + vortex_ensure!( + A::varying_len_matches(&varying, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + for index in 0..row_count { + apply( + &state, + A::get_varying(&varying, index), + S::row(&mut rows, index), + ) + .accumulate(&mut accumulated)?; + } + } else { + vortex_ensure!( + A::decoded_lens_match(&columns, row_count), + "a decoded row input does not address exactly {row_count} rows", + ); + + for index in 0..row_count { + apply(&state, A::get(&columns, index), S::row(&mut rows, index)) + .accumulate(&mut accumulated)?; + } + } + } + + finish_sink(sink, DeferredError::new(R::occurred(accumulated))) +} + +/// Run a prepared sink over only the rows set in `valid`, or decline when the sink cannot skip. +pub(super) fn execute_row_sink_branch( + args: &dyn ExecutionArgs, + sink_dtype: &DType, + valid: &Mask, + ctx: &mut ExecutionCtx, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, +) -> VortexResult> { + if !S::SUPPORTS_SKIPPED_ROWS { + return Ok(None); + } + + let Some(columns) = A::decode_null_tolerant(args, ctx)? else { + return Ok(None); + }; + let state = prepare(A::constants(&columns)); + let row_count = args.row_count(); + let mut sink = S::with_capacity(row_count, sink_dtype)?; + let mut accumulated = R::Accumulated::default(); + + let AllOr::Some(valid) = valid.bit_buffer() else { + vortex_bail!("execute_row_sink_branch requires a mixed mask"); + }; + + { + let mut rows = sink.rows(); + vortex_ensure!( + S::row_count_matches(&rows, row_count), + "the output sink does not address exactly {row_count} rows", + ); + + let varying = A::varying(&columns); + let lens_match = match &varying { + Some(varying) => A::varying_len_matches(varying, row_count), + None => A::decoded_lens_match(&columns, row_count), + }; + vortex_ensure!( + lens_match, + "a decoded row input does not address exactly {row_count} rows", + ); + + let mut error = None; + valid.for_each_set_index(|index| { + if error.is_some() { + return; + } + + let result = match &varying { + Some(varying) => apply( + &state, + A::get_varying(varying, index), + S::row(&mut rows, index), + ), + None => apply(&state, A::get(&columns, index), S::row(&mut rows, index)), + }; + if let Err(err) = result.accumulate(&mut accumulated) { + error = Some(err); + } + }); + + if let Some(error) = error { + return Err(error); + } + } + + finish_sink(sink, DeferredError::new(R::occurred(accumulated))).map(Some) +} + +/// Finish a sink while preserving whether its error came from the deferred row accumulator. +fn finish_sink( + sink: S, + deferred_error: DeferredError, +) -> VortexResult { + match sink.finish(deferred_error) { + Ok(output) => Ok(RowExecution::Output(output)), + Err(error) if deferred_error.occurred() => Ok(RowExecution::DeferredError(error)), + Err(error) => Err(error), + } +} diff --git a/vortex-array/src/scalar_fn/row/lift.rs b/vortex-array/src/scalar_fn/row/lift.rs new file mode 100644 index 00000000000..0a44b066826 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/lift.rs @@ -0,0 +1,699 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Lifting a kernel over non-null values into a full [`ScalarFnVTable::execute`]. +//! +//! A [`RowFn`] hands the framework a kernel that only ever computes rows valid in every argument. +//! Everything between that kernel and [`ScalarFnVTable::execute`] lives here: null propagation, +//! constant folding, nullability widening, output dtype reconciliation, and the per-batch choice +//! between dense execution and the two mechanisms that execute only valid rows. +//! +//! This is machinery, not an interface. It takes the kernel as a pair of closures rather than a +//! trait because the one trait that ever occupied the slot (a public `StrictScalarFnVTable`, with +//! [`RowFn`] blanket-implementing it) never found a second implementor, and the indirection cost +//! more than it explained. Extract a trait if and when a non-row user appears. +//! +//! [`RowFn`]: crate::scalar_fn::RowFn +//! [`ScalarFnVTable::execute`]: crate::scalar_fn::ScalarFnVTable::execute + +use smallvec::SmallVec; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_error::vortex_ensure_eq; +use vortex_mask::AllOr; +use vortex_mask::Mask; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::BoolArray; +use crate::arrays::ConstantArray; +use crate::arrays::MaskedArray; +use crate::arrays::PrimitiveArray; +use crate::builtins::ArrayBuiltins; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar::Scalar; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::SinkResult; +use crate::scalar_fn::row::element::batch_constant; +use crate::scalar_fn::row::execute::RowExecution; +use crate::validity::Validity; + +struct BorrowedExecutionArgs<'a> { + inputs: &'a [ArrayRef], + row_count: usize, +} + +impl<'a> BorrowedExecutionArgs<'a> { + fn new(inputs: &'a [ArrayRef], row_count: usize) -> Self { + Self { inputs, row_count } + } +} + +impl ExecutionArgs for BorrowedExecutionArgs<'_> { + fn get(&self, index: usize) -> VortexResult { + self.inputs.get(index).cloned().ok_or_else(|| { + vortex_error::vortex_err!( + "Input index {} out of bounds (num_inputs={})", + index, + self.inputs.len() + ) + }) + } + + fn num_inputs(&self) -> usize { + self.inputs.len() + } + + fn row_count(&self) -> usize { + self.row_count + } +} + +/// The arguments handed to one kernel invocation. +/// +/// `arrays` may be filtered or sliced, while `dtypes` and `output_dtype` always describe the +/// original planned batch. Keeping them together prevents an execution path from accidentally +/// pairing an input view with unrelated planning metadata. +#[derive(Clone, Copy)] +pub(super) struct KernelArgs<'a> { + /// The executor-facing view, including the row count for this invocation. + pub(super) execution: &'a dyn ExecutionArgs, + + /// The same inputs as concrete arrays for encoding-aware rewrites. + pub(super) arrays: &'a [ArrayRef], + + /// The original input dtypes used to select the row implementation. + pub(super) dtypes: &'a [DType], + + /// The non-nullable dtype built by the selected output capability. + pub(super) output_dtype: &'a DType, +} + +/// The execution policy and output dtype selected by a planning visit. +pub(super) struct BatchPlan { + /// The non-nullable dtype built by the selected output capability. + pub(super) output_dtype: DType, + + /// How this concrete dispatch executes nullable rows. + pub(super) policy: RowPolicy, +} + +/// The nullable execution policy derived from one concrete dispatch. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum RowPolicy { + /// Evaluate all rows and mask the result. + Dense, + + /// Evaluate all rows, retrying only valid rows if a deferred error is raised. + DenseWithRetry, + + /// Execute only valid rows, choosing branch-and-skip or filtering from the mask and decode + /// cost. + ValidOnly { filtered_decode_cost: usize }, +} + +impl RowPolicy { + /// The policy for an owned output carrying batch-deferred failure evidence. + pub(super) const fn for_deferred_output() -> Self { + if A::DENSE_SAFE && !A::DECODE_FALLIBLE { + Self::DenseWithRetry + } else { + Self::ValidOnly { + filtered_decode_cost: A::FILTERED_DECODE_COST, + } + } + } + + /// The policy one concrete dispatch executes nullable rows under. + /// + /// Note what is deliberately **not** read here: [`OutputSink::SUPPORTS_SKIPPED_ROWS`]. Hoisting + /// it into the plan so that a non-skipping sink never enters the branch path looks like a free + /// win, and #9130 records it as one, but it is not: the branch path probes + /// [`reduce_encoded`](crate::scalar_fn::RowFn::reduce_encoded) against the _original_ arrays + /// before it ever consults the sink, and that is the only probe that sees them still encoded. + /// Skipping the path early would leave such a function with only the filtered probe, whose + /// canonical arrays match no encoding fast path. For a function whose reduction is defined to + /// answer differently from its row loop, that is a wrong answer rather than a slow one. + /// + /// [`OutputSink::SUPPORTS_SKIPPED_ROWS`]: crate::scalar_fn::OutputSink::SUPPORTS_SKIPPED_ROWS + pub(super) const fn for_sink() -> Self { + if A::DENSE_SAFE && !A::DECODE_FALLIBLE && !R::FALLIBLE { + if R::DEFERRED { + Self::DenseWithRetry + } else { + Self::Dense + } + } else { + Self::ValidOnly { + filtered_decode_cost: A::FILTERED_DECODE_COST, + } + } + } +} + +/// How far [`Batch::resolve_validity`] got before a mixed-mask strategy became necessary. +enum ResolvedMask { + /// The batch was answered without one: every row valid, or every row null. + Decided(ArrayRef), + + /// A mask with both set and unset bits, which a strategy must now execute. + Mixed(Mask), +} + +/// One batch of inputs, with everything the lifting reads off them before the kernel runs. +pub(super) struct Batch<'a> { + /// The function being executed, named in the errors this raises. + id: ScalarFnId, + + /// The arguments as the execution layer handed them over. Every path but the filter strategy + /// gives the kernel these untouched, so it sees the original encodings. + args: &'a dyn ExecutionArgs, + + /// The input columns, collected once: constant folding inspects them and the filter strategy + /// filters them. + inputs: SmallVec<[ArrayRef; 4]>, + + /// The input dtypes, collected with the columns and reused by both planning and execution. + arg_dtypes: SmallVec<[DType; 4]>, + + /// The conjoined input validity, so a row of the output is valid iff it is valid in every + /// input. Conjoining is lazy, and nothing materializes it unless the null handling asks. + validity: Validity, + + /// The dtype the function declares for these inputs, which the kernel's output is reconciled + /// against. Already widened to nullable if any input is nullable. + result_dtype: DType, + + /// The non-nullable dtype the dispatched output capability builds, computed while planning. + output_dtype: DType, + + /// How the concrete dispatch executes nullable rows. + policy: RowPolicy, +} + +impl<'a> Batch<'a> { + /// Collect `args` and read the lifting's facts off them, `return_dtype` being the function's + /// declared return dtype for the input dtypes it is handed. + /// + /// **Not** for a nullary function: with no inputs there is no validity to propagate and no + /// per-row work to fold, and the all-constant check below would vacuously pass. + pub(super) fn new( + id: ScalarFnId, + args: &'a dyn ExecutionArgs, + plan: impl FnOnce(&[DType]) -> VortexResult, + ) -> VortexResult { + let inputs: SmallVec<[ArrayRef; 4]> = (0..args.num_inputs()) + .map(|i| args.get(i)) + .collect::>()?; + + let arg_dtypes: SmallVec<[DType; 4]> = + inputs.iter().map(|input| input.dtype().clone()).collect(); + let plan = plan(&arg_dtypes)?; + let nullability = plan.output_dtype.nullability() + | Nullability::from(arg_dtypes.iter().any(DType::is_nullable)); + let result_dtype = plan.output_dtype.with_nullability(nullability); + + let mut validity = Validity::NonNullable; + for input in &inputs { + validity = validity.and(input.validity()?)?; + } + + Ok(Self { + id, + args, + inputs, + arg_dtypes, + validity, + result_dtype, + output_dtype: plan.output_dtype, + policy: plan.policy, + }) + } + + /// Run `kernel` over this batch, adding everything the kernel does not do: the null-constant + /// short circuit, the all-constant fold, and the null handling. + /// + /// `kernel` computes the whole column from the arguments it is handed. Those are this batch's + /// arguments untouched, except under the filter strategy, where they are filtered copies, and + /// in the all-constant fold, where they are one row each. What it may assume: + /// + /// - No input is a null constant, and the inputs are not all constant. + /// - Under valid-only execution, every row of every input is valid. + /// - Under dense execution, rows behind nulls hold arbitrary values, and their results are + /// discarded. + /// + /// Either way the kernel can ignore input validity, and its output **must** equal + /// `return_dtype` up to nullability. A kernel that returns nulls of its own keeps them, unioned + /// with the ones the lifting applies, which requires its declared dtype to be nullable. + /// + /// `branch` computes only the rows set in the conjoined mask, over the _unfiltered_ arguments, + /// writing an arbitrary placeholder everywhere else; `Ok(None)` means it cannot for these + /// inputs, which sends the batch to the filter strategy. It is only ever called with a mixed + /// mask, and it **must not** run its row computation (nor any per-row fallible decode) on an + /// unset row, since those rows hold arbitrary values and a fallible kernel would spuriously + /// fail on them. + pub(super) fn execute( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Strictness: any null-constant input forces an all-null result without evaluating the + // kernel. + if self + .inputs + .iter() + .any(|input| input.as_constant().is_some_and(|scalar| scalar.is_null())) + { + return Ok(self.all_null()); + } + + // All inputs constant, and their conjoined validity proves every row non-null. This sees + // through extension and masked wrappers just like argument decoding does. + if self.args.row_count() > 0 + && self.validity.definitely_no_nulls() + && self + .inputs + .iter() + .all(|input| batch_constant(input).is_some()) + { + return self.broadcast_one_row(kernel, ctx); + } + + match self.policy { + RowPolicy::Dense => self.execute_dense(kernel, false, ctx), + RowPolicy::DenseWithRetry => self.execute_dense(kernel, true, ctx), + RowPolicy::ValidOnly { + filtered_decode_cost, + } => self.execute_filtered(kernel, branch, filtered_decode_cost, ctx), + } + } + + /// Evaluate a single row of all-constant inputs and broadcast its value. + /// + /// Reconciling the row's dtype before reading the scalar keeps this path on the same + /// kernel/declaration agreement check as the dense and filter paths, rather than letting `cast` + /// paper over a disagreement. + fn broadcast_one_row( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let one_row: SmallVec<[ArrayRef; 4]> = self + .inputs + .iter() + .map(|input| input.slice(0..1)) + .collect::>()?; + + let args = BorrowedExecutionArgs::new(&one_row, 1); + let result = kernel(self.kernel_args(&args, &one_row), ctx)?.into_result()?; + let scalar = self.with_return_dtype(result, 1)?.execute_scalar(0, ctx)?; + + Ok(ConstantArray::new(scalar, self.args.row_count()).into_array()) + } + + /// Run the kernel over every row, including the rows behind nulls, then mask its result. + /// + /// The arguments reach the kernel untouched, so the inputs keep their original encoding, and + /// the conjoined validity is handed to `mask` as an array rather than materialized into a + /// [`Mask`] first. + fn execute_dense( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + retry_deferred_error: bool, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Every row is null, so the kernel has nothing to contribute. + if matches!(self.validity, Validity::AllInvalid) { + return Ok(self.all_null()); + } + + let values = match kernel(self.kernel_args(self.args, &self.inputs), ctx)? { + RowExecution::Output(values) => values, + RowExecution::DeferredError(error) if retry_deferred_error => { + let valid = self + .validity + .clone() + .execute_mask(self.args.row_count(), ctx)?; + + // The same shortcut pair as `resolve_validity`, with different outcomes: every + // row valid means some valid row genuinely failed, and no row valid means every + // failure was behind a null. An empty mask is both all-true and all-false, but + // cannot reach this arm: a zero-row loop accumulates no evidence, so a zero-row + // batch never reports a deferred error. + if valid.all_true() { + return Err(error); + } + if valid.all_false() { + return Ok(self.all_null()); + } + + // Filtering unconditionally, rather than consulting `branch_beats_filter`. Not + // because branch-and-skip is unavailable in principle: `ERRORS_ARE_DEFERRED` and + // `SUPPORTS_SKIPPED_ROWS` are independent, and a sink may legally set both. It is + // that `execute_dense` is not handed the `branch` closure at all, so filtering is + // the only strategy reachable from here. This is the cold path, taken only after a + // batch has already reported an error, so the choice has not been worth plumbing + // for. + return self.filter_and_scatter(kernel, &valid, ctx); + } + RowExecution::DeferredError(error) => return Err(error), + }; + + match self.validity.clone() { + Validity::NonNullable | Validity::AllValid => { + self.with_return_dtype(values, self.args.row_count()) + } + Validity::Array(valid) => { + self.with_return_dtype(values.mask(valid)?, self.args.row_count()) + } + // Handled by the guard above, before the kernel ran. + Validity::AllInvalid => Ok(self.all_null()), + } + } + + /// Materialize the conjoined validity and resolve everything that does not need a mixed-mask + /// strategy, so that the production selector and the forced-strategy test seam cannot drift + /// apart on the shortcuts they share. The deferred-error retry in + /// [`execute_dense`](Self::execute_dense) repeats the same materialize-then-shortcut shape + /// with different outcomes — all-true is an error there, all-false is all-null — so it stays + /// open-coded, with its own note on why the ordering is safe. + fn resolve_validity( + &self, + kernel: &impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let valid = self + .validity + .clone() + .execute_mask(self.args.row_count(), ctx)?; + + // Check all-true before all-false: an empty mask is both, and must not be treated as + // all-null (a zero-length non-nullable execution keeps its non-nullable dtype). + if valid.all_true() { + return self + .with_return_dtype( + kernel(self.kernel_args(self.args, &self.inputs), ctx)?.into_result()?, + self.args.row_count(), + ) + .map(ResolvedMask::Decided); + } + + if valid.all_false() { + return Ok(ResolvedMask::Decided(self.all_null())); + } + + Ok(ResolvedMask::Mixed(valid)) + } + + /// Materialize the conjoined validity once, take the all-true and all-false shortcuts, and + /// pick a strategy per batch for a mixed mask. + /// + /// Two strategies can execute a mixed mask, and neither is visible to the kernel: + /// + /// - **Branch-and-skip** ([`execute_branched`](Self::execute_branched)): hand the _unfiltered_ + /// arguments plus the mask to `branch`, which computes only the valid rows, then mask the + /// full-length result exactly as the dense path does. This skips the filter and the scatter + /// entirely, at the price of decoding full-length columns. + /// - **Filter** ([`filter_and_scatter`](Self::filter_and_scatter)): filter every input down to + /// the conjoined-valid rows, run the kernel over those, and scatter its results back into a + /// null-padded output. Always available, never encoding-preserving. + /// + /// Branch-and-skip is preferred whenever [`branch_beats_filter`] says so, and the filter + /// strategy is also the fallback for a kernel with no branch execution. + fn execute_filtered( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + filtered_decode_cost: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let valid = match self.resolve_validity(&kernel, ctx)? { + ResolvedMask::Decided(result) => return Ok(result), + ResolvedMask::Mixed(valid) => valid, + }; + + if branch_beats_filter(filtered_decode_cost, &valid) + && let Some(result) = self.execute_branched(branch, &valid, ctx)? + { + return Ok(result); + } + + self.filter_and_scatter(kernel, &valid, ctx) + } + + /// Try the branch-and-skip strategy for a mixed mask: the kernel computes only the rows set in + /// `valid` over the unfiltered inputs, and the full-length result is masked exactly as the + /// dense path masks. `Ok(None)` means the kernel has no branch execution for these inputs, and + /// the caller falls back to [`filter_and_scatter`](Self::filter_and_scatter). + fn execute_branched( + &self, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + valid: &Mask, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let Some(values) = branch(self.kernel_args(self.args, &self.inputs), valid, ctx)? else { + return Ok(None); + }; + let values = values.into_result()?; + + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + self.with_return_dtype(values.mask(mask)?, valid.len()) + .map(Some) + } + + /// The filter strategy for a mixed mask: filter every input down to the rows set in `valid`, + /// run the kernel over those, and scatter its results back into a null-padded output. + fn filter_and_scatter( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + valid: &Mask, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let filtered: SmallVec<[ArrayRef; 4]> = self + .inputs + .iter() + .map(|input| input.filter(valid.clone())) + .collect::>()?; + + let args = BorrowedExecutionArgs::new(&filtered, valid.true_count()); + let values = kernel(self.kernel_args(&args, &filtered), ctx)?.into_result()?; + + self.with_return_dtype(self.scatter_valid(values, valid)?, valid.len()) + } + + /// An all-null result of the function's declared return dtype. + fn all_null(&self) -> ArrayRef { + ConstantArray::new( + Scalar::null(self.result_dtype.clone()), + self.args.row_count(), + ) + .into_array() + } + + /// Pair an input view with this batch's planning metadata. + fn kernel_args<'b>( + &'b self, + execution: &'b dyn ExecutionArgs, + arrays: &'b [ArrayRef], + ) -> KernelArgs<'b> { + KernelArgs { + execution, + arrays, + dtypes: &self.arg_dtypes, + output_dtype: &self.output_dtype, + } + } + + /// Reconcile the kernel's output dtype with the function's declared return dtype. + /// + /// The kernel may ignore nullability, so a nullability difference is cast away. Any other + /// difference means the declared dtype and the kernel disagree, which is a bug worth naming + /// rather than silently casting away. + fn with_return_dtype(&self, values: ArrayRef, expected_len: usize) -> VortexResult { + reconcile_return(self.id, &self.result_dtype, expected_len, values) + } + + /// Scatter `values` (one per set bit of `valid`, in order) back to the positions of the set + /// bits, producing an array of length `valid.len()` that is null at every unset position. + fn scatter_valid(&self, values: ArrayRef, valid: &Mask) -> VortexResult { + vortex_ensure_eq!( + values.len(), + valid.true_count(), + "the {} kernel produced {} rows for {} filtered rows", + self.id, + values.len(), + valid.true_count(), + ); + + let AllOr::Some(slices) = valid.slices() else { + // The caller handles the all-true and all-false masks. + vortex_bail!("scatter_valid requires a mixed mask"); + }; + + // Gather indices: row i of the output reads values[rank(i)]. Rows behind nulls read index + // 0, and any in-bounds index would do since they are masked out below (values is non-empty + // here). + let mut indices = vec![0u64; valid.len()]; + let mut rank = 0u64; + for &(start, end) in slices { + for index in &mut indices[start..end] { + *index = rank; + rank += 1; + } + } + let indices = PrimitiveArray::new(indices, Validity::NonNullable).into_array(); + + let scattered = values.take(indices)?; + + // A kernel that produced nulls of its own (only `reduce_encoded` may) cannot be wrapped, + // since a `Masked` child must be all valid. Those nulls have to be unioned with the + // lifting's, which is what the general masking pass does. + if scattered.dtype().is_nullable() { + let mask = BoolArray::new(valid.to_bit_buffer(), Validity::NonNullable).into_array(); + return scattered.mask(mask); + } + + // Attaching the mask as validity rather than masking again: the gathered values are + // already all valid, so recording which rows survive is the whole job and a `Masked` + // wrapper says exactly that. Worth 1.13-1.53x here, growing with null density + // (`null_strategy_bytes`, 65536 rows, divan fastest and median of 100 samples, best of two + // runs, Apple M4 Max). The same substitution on the dense path measured no difference, so + // it is deliberately confined to the scatter. + Ok(MaskedArray::try_new( + scattered, + Validity::from_mask(valid.clone(), Nullability::Nullable), + )? + .into_array()) + } +} + +/// Validate the row count and reconcile nullability against a row function's declared dtype. +pub(super) fn reconcile_return( + id: ScalarFnId, + result_dtype: &DType, + expected_len: usize, + values: ArrayRef, +) -> VortexResult { + vortex_ensure_eq!( + values.len(), + expected_len, + "the {id} kernel produced {} rows for {expected_len} input rows", + values.len(), + ); + vortex_ensure!( + values.dtype().eq_ignore_nullability(result_dtype), + "the {id} kernel produced {} but the function declares {result_dtype}", + values.dtype(), + ); + + if values.dtype() == result_dtype { + Ok(values) + } else { + values.cast(result_dtype.clone()) + } +} + +/// The minimum surviving-row fraction (`true_count / len` of the conjoined mask) at which +/// branch-and-skip is still chosen for one filtered decode unit. +/// +/// From the branch-and-skip measurements (65536 rows, divan fastest of 100 samples, two runs on a +/// shared 4-vCPU VM). A kernel with a _bulk_ decode never lost under branch: `byte_length` over +/// a byte-string element ran 1.8-5.9x faster than filter at every null density from 1% to 90%, so +/// such kernels skip this check entirely. A kernel with a _per-row_ decode (geo `contains`, which +/// arrow-exports and parses one geometry per row) pays that decode over the full column under +/// branch but only over the survivors under filter, so filter wins once validity is sparse: +/// +/// - polygons CONTAINS constant point: branch won 1.07-1.18x at 1-50% nulls; filter won 1.38x at +/// 90% nulls (10% of rows surviving). +/// - polygons CONTAINS points, independent nulls on both: branch won up to ~10% null density +/// (~81% surviving); filter won 1.2x at ~56% surviving, 1.9x at ~25%, 11.3x at ~1%. +/// +/// A single nullable operand still favored branch at 50% surviving, while two independent nullable +/// operands favored filtering at 81% surviving. Keep those cases distinct instead of collapsing +/// every per-row decode into one boolean. There is not yet enough evidence to distinguish two from +/// three or more decode units, so they share the conservative multi-decode threshold. +pub(super) const ONE_DECODE_BRANCH_MIN_SURVIVING_FRACTION: f64 = 0.50; +pub(super) const MULTI_DECODE_BRANCH_MIN_SURVIVING_FRACTION: f64 = 0.85; + +/// Whether the branch-and-skip strategy should be preferred over filtering for the mixed mask +/// `valid`. A zero cost always branches; otherwise the survivor threshold grows when filtering +/// avoids more than one unit of per-row decode work. +pub(super) fn branch_beats_filter(filtered_decode_cost: usize, valid: &Mask) -> bool { + if filtered_decode_cost == 0 { + return true; + } + + let minimum = if filtered_decode_cost == 1 { + ONE_DECODE_BRANCH_MIN_SURVIVING_FRACTION + } else { + MULTI_DECODE_BRANCH_MIN_SURVIVING_FRACTION + }; + valid.true_count() as f64 >= valid.len() as f64 * minimum +} + +/// Which null strategy a forced execution takes for a mixed validity mask. +/// +/// A test and benchmark seam: pinning a strategy is how the two are compared and how their +/// agreement is asserted. Production execution selects per batch inside the lifting and never +/// names one. See [`execute_row_fn_with_strategy`](super::execute_row_fn_with_strategy). +#[cfg(any(test, feature = "_test-harness"))] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum NullStrategy { + /// Filter the inputs down to the conjoined-valid rows, run the kernel, and scatter back. + Filter, + + /// Decode the unfiltered inputs null-tolerantly, compute only the conjoined-valid rows, and + /// mask the full-length result. + BranchAndSkip, +} + +#[cfg(any(test, feature = "_test-harness"))] +impl Batch<'_> { + /// Execute this batch with a forced null strategy, bypassing the per-batch selection. + /// + /// A test and benchmark seam only. It mirrors [`execute_filtered`](Self::execute_filtered) + /// (conjoined validity, the all-true and all-false shortcuts, output dtype reconciliation) but + /// takes the strategy from the caller instead of the selection rule, and it skips the + /// null-constant and all-constant folds, so do not pass such inputs. `Ok(None)` means + /// [`NullStrategy::BranchAndSkip`] was forced on a kernel with no branch execution, which the + /// caller reports rather than silently falling back. + pub(super) fn execute_with_strategy( + &self, + kernel: impl Fn(KernelArgs<'_>, &mut ExecutionCtx) -> VortexResult, + branch: impl FnOnce( + KernelArgs<'_>, + &Mask, + &mut ExecutionCtx, + ) -> VortexResult>, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let valid = match self.resolve_validity(&kernel, ctx)? { + ResolvedMask::Decided(result) => return Ok(Some(result)), + ResolvedMask::Mixed(valid) => valid, + }; + + match strategy { + NullStrategy::Filter => self.filter_and_scatter(kernel, &valid, ctx).map(Some), + NullStrategy::BranchAndSkip => self.execute_branched(branch, &valid, ctx), + } + } +} diff --git a/vortex-array/src/scalar_fn/row/mod.rs b/vortex-array/src/scalar_fn/row/mod.rs new file mode 100644 index 00000000000..d9799fff678 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/mod.rs @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Defining scalar functions one row at a time. +//! +//! This is the derived way to write a scalar function, and the right default for a kernel that has +//! to read every row anyway. See [choosing a trait](crate::scalar_fn#choosing-a-trait) for when to +//! drop to [`ScalarFnVTable`](crate::scalar_fn::ScalarFnVTable) instead. +//! +//! [`RowFn`] names its arguments and provides a [`dispatch`](RowFn::dispatch) that picks the +//! concrete element and sink types for a batch. Everything else (dtype checks, output dtype, null +//! handling, constants, and validity) is derived from that dispatch. +//! +//! When the element types are fixed, `dispatch` is a single visit at those types. When one function +//! ID has to cover several (`l2_norm` accepts `f16`, `f32` and `f64` columns), `dispatch` matches on +//! the input dtypes and visits at the chosen width. Kernel fallibility is declared separately +//! because callers need it before dispatch. +//! +//! [`RowFn`] does not say how a row is _stored_, which is the element's job: `vortex-tensor` adds a +//! `TensorRow` [`InputElement`] and writes ordinary kernels over it. +//! +//! Output has two capabilities. [`RowVisitor::visit_prepared_deferred`] returns one independent +//! [`OutputElement`] and failure word per row, letting shared execution own the stores and choose an +//! indexed dense source. [`RowVisitor::visit_prepared_into`] writes through an [`OutputSink`] for +//! runtime-shaped rows, shared builders, skip-capable output, and values requiring drop glue. Both +//! prepare closures see every batch-constant input and return shared state for the row loop. Pass +//! `|_| ()` when there is nothing to prepare. +//! +//! A kernel that can safely write a provisional value uses [`DeferredError`] instead of returning +//! a per-row result. The executor vector-reduces those bits and hands one batch-wide error to the +//! sink. With nullable fixed-width inputs it runs densely and retries only valid rows on the cold +//! error path. +//! +//! Null handling is derived and executed by the [lifting](lift), never by the row closure, which +//! only ever computes rows valid in every argument. A batch with a mixed validity mask executes by +//! one of two strategies, selected per batch: _branch-and-skip_ (decode the unfiltered columns +//! null-tolerantly via [`InputElement::decode_null_tolerant`], compute only the valid rows a word +//! of the mask at a time, mask the result) whenever it can, and _filter_ (shrink every input to +//! the surviving rows, compute, scatter back) when an argument has no null-tolerant decode for its +//! array or when a per-row decode makes filtering cheaper at sparse validity. Authors do nothing; +//! an element whose decode does expensive per-row work reports that work through +//! [`InputElement::FILTERED_DECODE_COST`]. The costs of all arguments are added together so the +//! batch selector can distinguish one expensive decode from several. A sink opts into +//! branch-and-skip with [`OutputSink::SUPPORTS_SKIPPED_ROWS`]. + +mod element; +pub use element::ElementTuple; +pub use element::IndexedElementTuple; +pub use element::InputElement; +pub use element::OutputElement; +#[cfg(any(test, feature = "_test-harness"))] +pub use element::assert_element_conforms; + +mod result; +pub use result::DeferredError; +pub use result::SinkResult; + +mod sink; +pub use sink::ElementSink; +pub use sink::OutputSink; + +mod execute; + +mod lift; +#[cfg(any(test, feature = "_test-harness"))] +pub use lift::NullStrategy; + +mod row_fn; +pub use row_fn::RowFn; +pub use row_fn::RowVisitor; + +mod vtable; +#[cfg(any(test, feature = "_test-harness"))] +pub use vtable::execute_row_fn_with_strategy; + +#[cfg(test)] +mod tests; diff --git a/vortex-array/src/scalar_fn/row/result.rs b/vortex-array/src/scalar_fn/row/result.rs new file mode 100644 index 00000000000..86b90db41f5 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/result.rs @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What a sink-writing row closure may return. + +use std::ops::BitOrAssign; + +use vortex_error::VortexResult; + +mod private { + pub trait Sealed {} +} + +/// A value-dependent failure bit reduced across the whole row loop and handed to the output sink. +/// +/// Unlike [`VortexResult`], this never exits the loop. It is for kernels such as checked addition +/// that can safely write a provisional value for every row and report any failure once at the end. +/// +/// **The reduction is one byte wide on purpose.** It is OR-reduced once per row alongside the +/// kernel's own arithmetic, so a wider accumulator caps how many rows a vector of the reduction +/// covers, whatever the element width. Carrying the bit in an `i64` instead cost the primitive +/// `Mul` kernel 3.1x at `i8`, 1.9x at `i16` and 1.2x at `i32`, and nothing at `i64` where the two +/// widths already agree (`binary_ops`, 65536 rows, divan fastest of 100 samples, best of two runs, +/// Apple M4 Max). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct DeferredError(bool); + +impl DeferredError { + /// Record whether this row encountered an error. + pub const fn new(failed: bool) -> Self { + Self(failed) + } + + /// Whether any row accumulated into this value failed. + pub const fn occurred(self) -> bool { + self.0 + } +} + +impl BitOrAssign for DeferredError { + fn bitor_assign(&mut self, rhs: Self) { + self.0 |= rhs.0; + } +} + +/// What a row computation that _writes_ into an [`OutputSink`](crate::scalar_fn::OutputSink) may +/// produce: nothing, an early [`VortexResult`] error, or non-branching failure evidence. +/// +/// The value is already in the sink by the time the closure returns, so the only thing left to +/// report is failure. +/// +/// [`Accumulated`](Self::Accumulated) is the word the executor OR-reduces in a **local**, which is +/// what keeps the reduction in a register and the row loop vectorizable. It exists so that evidence +/// can be wider than one bit when narrowing it per row would cost more than carrying it: unsigned +/// multiplication hands back the discarded high half of its product, because comparing that half +/// against zero per row is what LLVM folds into `llvm.umul.with.overflow`, which has no vector form. +/// **The word must be no wider than the element**, or the reduction, rather than the arithmetic, +/// bounds how many rows a vector covers. +/// +/// The sink never sees this. It is handed a plain [`DeferredError`] once, after the loop. +/// +/// This trait is framework-only. Row functions choose one of the supplied return forms; custom +/// output representation belongs in [`OutputSink`](crate::scalar_fn::OutputSink). +pub trait SinkResult: 'static + private::Sealed { + /// The word this result reduces into, kept in a loop-local by the executor. + type Accumulated: 'static + Copy + Default; + + /// Whether this return type can carry an error. + const FALLIBLE: bool; + + /// Whether this result carries non-branching failure evidence for the sink. + const DEFERRED: bool; + + /// Merge this row's outcome into the batch-wide reduction. + fn accumulate(self, accumulated: &mut Self::Accumulated) -> VortexResult<()>; + + /// Whether the finished reduction means some row failed. + fn occurred(accumulated: Self::Accumulated) -> bool; +} + +impl private::Sealed for () {} + +impl SinkResult for () { + type Accumulated = (); + + const FALLIBLE: bool = false; + const DEFERRED: bool = false; + + fn accumulate(self, _accumulated: &mut ()) -> VortexResult<()> { + Ok(()) + } + + fn occurred(_accumulated: ()) -> bool { + false + } +} + +impl private::Sealed for VortexResult<()> {} + +impl SinkResult for VortexResult<()> { + type Accumulated = (); + + const FALLIBLE: bool = true; + const DEFERRED: bool = false; + + fn accumulate(self, _accumulated: &mut ()) -> VortexResult<()> { + self + } + + fn occurred(_accumulated: ()) -> bool { + false + } +} + +/// The evidence widths a row closure may reduce. `bool` is the ordinary answer; the unsigned +/// integers exist for a kernel whose per-row comparison would cost it its vectorization. +macro_rules! impl_sink_result_word { + ($($word:ty),+ $(,)?) => { + $( + impl private::Sealed for $word {} + + impl SinkResult for $word { + type Accumulated = $word; + + const FALLIBLE: bool = false; + const DEFERRED: bool = true; + + fn accumulate(self, accumulated: &mut $word) -> VortexResult<()> { + *accumulated |= self; + Ok(()) + } + + fn occurred(accumulated: $word) -> bool { + accumulated != <$word>::default() + } + } + )+ + }; +} + +impl_sink_result_word!(bool, u8, u16, u32, u64); + +#[cfg(test)] +mod tests { + use super::DeferredError; + + #[test] + fn one_failing_row_is_enough() { + let mut error = DeferredError::default(); + assert!(!error.occurred()); + + error |= DeferredError::new(false); + assert!(!error.occurred()); + + error |= DeferredError::new(true); + assert!(error.occurred()); + + error |= DeferredError::new(false); + assert!(error.occurred()); + } +} diff --git a/vortex-array/src/scalar_fn/row/row_fn.rs b/vortex-array/src/scalar_fn/row/row_fn.rs new file mode 100644 index 00000000000..f790d21d8a0 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/row_fn.rs @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Scalar functions computed one row at a time. + +use std::fmt::Debug; +use std::fmt::Display; +use std::hash::Hash; +use std::ops::BitOrAssign; + +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_session::VortexSession; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::IndexedElementTuple; +use crate::scalar_fn::OutputElement; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::SinkResult; + +/// A scalar function computed one row at a time. +/// +/// An implementor declares its argument names, then [`dispatch`](Self::dispatch) picks the concrete +/// element and sink types for a batch. The planning visit reads dense safety, decode fallibility, +/// and decode cost from that concrete choice; no representative element types are needed. +/// +/// A function whose kernel is columnar rather than row-at-a-time (negating a whole bit buffer, a +/// zero-copy unwrap) is not a `RowFn`, and implements +/// [`ScalarFnVTable`](crate::scalar_fn::ScalarFnVTable) directly. +pub trait RowFn: 'static + Sized + Clone + Send + Sync { + /// Options for this function, if any. Use [`EmptyOptions`](crate::scalar_fn::EmptyOptions) + /// for none. + type Options: 'static + Send + Sync + Clone + Debug + Display + PartialEq + Eq + Hash; + + /// The arguments in display order. Its length is the function's exact arity. + const ARG_NAMES: &'static [&'static str]; + + /// Whether any legal dispatch can fail while decoding or computing a row. + /// + /// The framework verifies that every fallible dispatched element or result implies this value. + /// A conservative `true` is allowed when only some dtype choices are fallible. + const FALLIBLE: bool = false; + + /// Returns the ID of the scalar function. + fn id(&self) -> ScalarFnId; + + /// Serialize this function's options, or return `None` when the function is not serializable. + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + _ = options; + Ok(None) + } + + /// Restore options written by [`serialize`](Self::serialize). + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + vortex_bail!("Expression {} is not deserializable", self.id()) + } + + /// Choose element types for these input dtypes and visit the framework with them. + /// + /// This is where a per-batch width match lives (`match_each_float_ptype!` and friends panic + /// outside their width class, so check the class first), and where cross-argument dtype + /// constraints belong, since per-argument validation runs inside the visit. Plan time and run + /// time both come through here, so the choice **must** be a pure function of `options` and + /// `args`. + fn dispatch( + &self, + options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult; + + /// An encoding-aware rewrite, tried on the input arrays before the row loop. + /// + /// `Some` skips the row loop entirely, which makes this the escape hatch for a function that is + /// row-shaped in general but has a bulk answer for some encodings: reading stored values back out + /// of a wrapper encoding, or handing back a child array whole. The result may be lazy and + /// nullable, but its nulls **must** be a subset of the rows the lifting will mask, and it + /// **must** have one row per row of `args`, which on the filter strategy is the _filtered_ count + /// rather than the original one. Size the result from `args`, which are filtered to match, and + /// never from a length captured elsewhere. + /// + /// Whether the arrays still carry their original encoding depends on the execution path. + /// Dense execution always passes them through untouched. Valid-only execution does too when + /// no row is null; for a mixed mask, branch-and-skip also passes them through untouched (full + /// length, with the result masked afterwards), while filtering hands over filtered copies, + /// which are canonical and so match no encoding fast path. + /// + /// A non-nullable operand therefore reaches an encoding fast path under either. Note also that + /// filtering a constant yields a constant, so a fast path keyed on + /// [`as_constant`](ArrayRef::as_constant) still fires even for a filtered batch. + fn reduce_encoded( + &self, + options: &Self::Options, + args: &[ArrayRef], + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + _ = (options, args, ctx); + Ok(None) + } +} + +/// One use of a [`RowFn`] at concrete element types. +/// +/// The framework hands a visitor to [`RowFn::dispatch`], which calls one of the visit methods with +/// the element types it chose: at plan time the visit validates dtypes, at run time it executes the row +/// loop. Only the framework implements this trait, and a function only ever _calls_ a visit. +/// +/// The function names one output sink and one preparation step. Passing `|_| ()` is the no-prepare +/// case. +pub trait RowVisitor: private::Sealed { + /// What this visit produces. + type Out; + + /// Visit at indexed argument tuple `A`, returning one independently owned output and one + /// deferred failure word per row. + /// + /// The executor allocates and writes the output column. It reads through + /// [`IndexedElementTuple`] when every argument varies; batches containing a constant use + /// ordinary row access selected once outside the loop. `F::default()` **must** mean success, + /// including for empty execution, and `|=` must combine the evidence from independent rows. + /// `finish_failure` runs once after the loop: it must return `Ok(())` for successful evidence + /// and may report only the operation's row error for failed evidence. That error is deferred, + /// so nullable lifting may retry over only valid rows. + /// + /// `A` **must** have the arity declared by [`RowFn::ARG_NAMES`]. This method requires + /// [`RowFn::FALLIBLE`] to be `true`, and `O` must be no narrower than `F` so failure reduction + /// does not constrain vector width. `O` must not require drop glue. Use + /// [`visit_prepared_into`](Self::visit_prepared_into) for non-indexed tuples, runtime-shaped + /// output, shared builders, and output that requires drop. + fn visit_prepared_deferred( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, F), + finish_failure: impl FnOnce(F) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + F: 'static + Copy + Default + BitOrAssign; + + /// Visit at argument tuple `A`, preparing shared state once and writing every output row into + /// sink `S`. + /// + /// `prepare` receives [`A::ConstElems`](ElementTuple::ConstElems): the element value of every + /// argument whose operand is constant for the batch, and `None` for each one that varies by + /// row. Whatever it returns is handed to every `apply` call by shared reference. + /// + /// `A` **must** have the arity declared by [`RowFn::ARG_NAMES`]. A fallible element or result + /// also requires [`RowFn::FALLIBLE`] to be `true`; the reverse is not required. A deferred result + /// must be paired with a sink whose [`OutputSink::ERRORS_ARE_DEFERRED`] is `true`. + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult; +} + +pub(super) mod private { + pub trait Sealed {} +} diff --git a/vortex-array/src/scalar_fn/row/sink.rs b/vortex-array/src/scalar_fn/row/sink.rs new file mode 100644 index 00000000000..886f9f5f9e1 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/sink.rs @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The column builders a row function can write its output into. + +use vortex_error::VortexResult; + +use crate::ArrayRef; +use crate::dtype::DType; +use crate::scalar_fn::DeferredError; +use crate::scalar_fn::OutputElement; + +/// A column allocated once per batch that a row closure writes into, one row at a time. +/// +/// Every [`RowFn`](crate::scalar_fn::RowFn) writes through one. [`ElementSink`] covers an ordinary +/// owned value per row. A custom sink covers output whose width is runtime data or whose rows append +/// into one batch-wide builder. +/// +/// Two properties of the contract are worth stating, since both are load-bearing: +/// +/// - **The row loop, not the closure, holds the sink.** [`row`](Self::row) is called by the framework +/// and its result passed in, so a writing closure stays [`Fn`] and captures nothing mutable. +/// Relaxing the row closure to `FnMut` instead was measured at 8 to 11%, because a captured `&mut` +/// inhibits vectorization of the loop. +/// - **[`sink_dtype`](Self::sink_dtype) sees the input dtypes**, unlike +/// [`OutputElement::element_dtype`], which takes +/// none. That is the whole reason a runtime-shaped output fits here: the width comes out of the +/// arguments. +/// +/// Rows arrive in increasing index order. Ordinary execution visits `0..row_count` exactly once; +/// branch-and-skip may omit null rows when [`SUPPORTS_SKIPPED_ROWS`](Self::SUPPORTS_SKIPPED_ROWS) +/// is `true`. +pub trait OutputSink: 'static + Sized { + /// Whether this sink accepts [`DeferredError`] from its row closure instead of requiring a + /// per-row [`VortexResult`]. + /// + /// The executor OR-reduces the row error words and passes the result to + /// [`finish`](Self::finish). When the arguments are safe to read behind nulls, this lets the + /// lifting optimistically run a dense loop. If `finish` reports the deferred error for a + /// nullable batch, the lifting retries over only the valid rows: success means the error came + /// exclusively from null rows, while another deferred error is real. + /// + /// A supporting sink must return an error from `finish` when its `error` argument occurred. + const ERRORS_ARE_DEFERRED: bool = false; + + /// Whether this sink can finish a full-length output when some rows were never visited. + /// + /// A supporting sink must leave a legal arbitrary value at every skipped row. The lifting masks + /// those rows before the result escapes, so that value is never observable. + const SUPPORTS_SKIPPED_ROWS: bool = false; + + /// A loop-local view of all output rows. + /// + /// Borrowed once before execution so the sink's buffer descriptor and shape become loop + /// invariants rather than being re-read through `&mut Self` for every row. + type Rows<'a> + where + Self: 'a; + + /// The place a row closure writes one row through, borrowed from the sink. + type Row<'a> + where + Self: 'a; + + /// The dtype of the column this sink builds, given the function's input dtypes. + /// + /// Must be non-nullable: nullability is derived from the inputs by the lifting, which + /// widens the result and masks the null rows itself. + fn sink_dtype(args: &[DType]) -> VortexResult; + + /// Allocate a sink for `rows` rows of `dtype`, which is this sink's own + /// [`sink_dtype`](Self::sink_dtype). Called once per batch. + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult; + + /// Borrow all output rows for the hot loop. + fn rows(&mut self) -> Self::Rows<'_>; + + /// Whether every index in `0..row_count` is addressable through [`row`](Self::row). + /// + /// Called once before the hot loop. Besides validating the sink contract, this gives the + /// optimizer the output bounds it needs to remove the bounds check hidden in each row accessor. + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool; + + /// Hand out the place to write row `index`. Must be `O(1)`: it is called in the row loop. + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a>; + + /// Finish into the built column, whose dtype **must** be this sink's + /// [`sink_dtype`](Self::sink_dtype). Called once per batch with the OR of every row's deferred + /// error bit. + fn finish(self, error: DeferredError) -> VortexResult; +} + +/// The standard output sink for one owned [`OutputElement`] per row. +pub struct ElementSink { + values: Vec, +} + +impl OutputSink for ElementSink { + const SUPPORTS_SKIPPED_ROWS: bool = true; + + type Rows<'a> = &'a mut [T]; + type Row<'a> = &'a mut T; + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(T::element_dtype()) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + // `vec![placeholder; rows]` rather than `resize_with(rows, placeholder)`: the former hands + // a zeroable placeholder (every primitive, `false`) straight to `alloc_zeroed`, while the + // latter always writes one element at a time. Only branch-and-skip ever reads a + // placeholder back, so on the dense and filter paths that write is pure waste. + Ok(Self { + values: vec![T::placeholder(); rows], + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + &mut self.values + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + &mut rows[index] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(T::build(self.values)) + } +} diff --git a/vortex-array/src/scalar_fn/row/tests/conformance.rs b/vortex-array/src/scalar_fn/row/tests/conformance.rs new file mode 100644 index 00000000000..8df438a734c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/conformance.rs @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Conformance tests for every [`InputElement`](crate::scalar_fn::InputElement) in this crate. + +use std::sync::Arc; + +use vortex_buffer::BitBuffer; +use vortex_buffer::ByteBuffer; +use vortex_buffer::buffer; +use vortex_error::VortexResult; + +use crate::IntoArray; +use crate::VortexSessionExecute; +use crate::array_session; +use crate::arrays::BoolArray; +use crate::arrays::PrimitiveArray; +use crate::arrays::VarBinViewArray; +use crate::arrays::varbinview::BinaryView; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::scalar_fn::assert_element_conforms; +use crate::scalar_fn::row::tests::TestBytes; +use crate::validity::Validity; + +/// A `Utf8` column whose single null row carries a view naming a buffer that does not exist, at +/// an offset far past the end of the data. Reading its _bytes_ densely panics; reading its +/// _length_ does not, which is exactly the distinction `DENSE_SAFE` encodes. +fn hostile_views() -> VortexResult { + let views = buffer![ + BinaryView::make_view(b"a longer string here", 0, 0), + BinaryView::new_ref(64, *b"junk", 9, 4096), + ]; + Ok(VarBinViewArray::try_new( + views, + Arc::from([ByteBuffer::copy_from(b"a longer string here")]), + DType::Utf8(Nullability::Nullable), + Validity::from_iter([true, false]), + )? + .into_array()) +} + +#[test] +fn primitive_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + // The extremes sit at the rows that are then marked null. + let array = PrimitiveArray::new( + buffer![i32::MAX, 1, i32::MIN, 2], + Validity::from_iter([false, true, false, true]), + ) + .into_array(); + + assert_element_conforms::(array, &DType::Utf8(Nullability::NonNullable), &mut ctx) +} + +#[test] +fn bool_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let array = BoolArray::new( + BitBuffer::from(vec![true, true, false, true]), + Validity::from_iter([false, true, true, false]), + ) + .into_array(); + + assert_element_conforms::(array, &DType::Utf8(Nullability::NonNullable), &mut ctx) +} + +#[test] +fn test_bytes_element_conforms() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + assert_element_conforms::( + hostile_views()?, + &DType::Bool(Nullability::NonNullable), + &mut ctx, + ) +} diff --git a/vortex-array/src/scalar_fn/row/tests/constant_operands.rs b/vortex-array/src/scalar_fn/row/tests/constant_operands.rs new file mode 100644 index 00000000000..5e86b925587 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/constant_operands.rs @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests that constant operands are decoded once and broadcast across the batch. + +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; + +use vortex_buffer::Buffer; + +use super::*; + +/// Total rows handed to [`CountedI64::decode`] across one execution. Sound as a global because +/// each test binary runs one test per process. +static DECODED_ROWS: AtomicUsize = AtomicUsize::new(0); + +/// Stands in for an element whose decode is expensive per row, recording how wide a column each +/// decode was actually given. +struct CountedI64; + +impl InputElement for CountedI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + DECODED_ROWS.fetch_add(array.len(), Ordering::Relaxed); + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +#[derive(Clone)] +struct AddCounted; + +impl RowFn for AddCounted { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.add_counted"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(CountedI64, CountedI64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs + rhs, + ) + } +} + +/// An element whose decode drops the last row, standing in for a buggy element implementation. +struct ShortDecodeI64; + +impl InputElement for ShortDecodeI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let column = ::decode(array, ctx)?; + Ok(column.slice(0..column.len().saturating_sub(1))) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +/// Pairs a short-decoding argument with an ordinary one, so a batch-constant second operand takes +/// the mixed constant-and-varying read path rather than the all-varying one. +#[derive(Clone)] +struct AddShort; + +impl RowFn for AddShort { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.add_short"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(ShortDecodeI64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = lhs + rhs, + ) + } +} + +/// A constant operand makes the whole tuple decline the all-varying read path, so the row loop +/// indexes each [`ArgColumn`](crate::scalar_fn::ArgColumn) directly. The decoded length still has to +/// be checked there, or a short column reaches an out-of-bounds row read. +#[test] +fn a_short_decode_beside_a_constant_operand_is_rejected() { + let mut ctx = array_session().create_execution_ctx(); + let column = PrimitiveArray::from_iter(0..64i64).into_array(); + let constant = ConstantArray::new(Scalar::from(10i64), 64).into_array(); + + let error = apply(AddShort, [column, constant], &mut ctx).unwrap_err(); + + assert!( + error + .to_string() + .contains("does not address exactly 64 rows"), + "{error}" + ); +} + +#[test] +fn a_constant_operand_is_decoded_once() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let column = PrimitiveArray::from_iter(0..64i64).into_array(); + let constant = ConstantArray::new(Scalar::from(10i64), 64).into_array(); + + let result = apply(AddCounted, [column, constant], &mut ctx)?; + + // 64 rows for the real column, plus exactly one for the constant. + assert_eq!(DECODED_ROWS.load(Ordering::Relaxed), 65); + assert_arrays_eq!( + result, + PrimitiveArray::from_iter((0..64i64).map(|value| value + 10)), + &mut ctx + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs b/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs new file mode 100644 index 00000000000..e3ade44f680 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/decode_fallibility.rs @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for fallible input decoding and its effect on execution strategy. + +use super::*; + +/// Stands in for an element that _parses_ its bytes, like a WKB geometry: malformed bytes in a +/// valid row are a domain error, so decoding can fail on otherwise legal input. +struct ParsedBytes; + +impl InputElement for ParsedBytes { + type Column = VarBinViewArray; + type Varying<'a> = &'a VarBinViewArray; + type Elem<'a> = usize; + + const DENSE_SAFE: bool = true; + const DECODE_FALLIBLE: bool = true; + + fn validate(_dtype: &DType) -> VortexResult<()> { + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + array.execute::(ctx) + } + + fn get(column: &Self::Column, index: usize) -> usize { + column.views()[index].len() as usize + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> usize + where + Self: 'a, + { + Self::get(column, index) + } +} + +/// Its row computation is total; only the decode can fail. +#[derive(Clone)] +struct TotalKernelOverParsedInput; + +impl RowFn for TotalKernelOverParsedInput { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.total_over_parsed"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(ParsedBytes,), ElementSink, _, _>( + |_| (), + |&(), (len,), output| *output = len as u64, + ) + } +} + +/// The row closure is infallible, so reading only the kernel declaration would report +/// `false` and let dict pushdown speculatively evaluate the parse over unreferenced values. +#[test] +fn a_fallible_decode_makes_the_function_fallible() { + assert!(ScalarFnVTable::is_fallible( + &TotalKernelOverParsedInput, + &EmptyOptions + )); +} + +/// And it must not run densely: rows behind nulls would be parsed too. +#[test] +fn a_fallible_decode_forces_filtering() { + assert_eq!( + policy( + &TotalKernelOverParsedInput, + &[DType::Binary(Nullability::Nullable)] + ), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/dispatched.rs b/vortex-array/src/scalar_fn/row/tests/dispatched.rs new file mode 100644 index 00000000000..ca377bea4b1 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/dispatched.rs @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for row functions that choose their element types per batch. + +use vortex_error::vortex_ensure; + +use super::*; +use crate::match_each_integer_ptype; + +#[derive(Clone)] +struct Max; + +impl RowFn for Max { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.int_max"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + let DType::Primitive(ptype, _) = args[0] else { + vortex_bail!("int_max requires primitive inputs, got {}", args[0]); + }; + vortex_ensure!( + ptype.is_int(), + "int_max requires integer inputs, got {ptype}" + ); + + match_each_integer_ptype!(ptype, |T| { + visitor.visit_prepared_into::<(T, T), ElementSink, _, _>( + |_| (), + |&(), (a, b), output| *output = a.max(b), + ) + }) + } +} + +#[rstest] +#[case::i16(buffer![1i16, 9, 3].into_array(), buffer![4i16, 2, 3].into_array(), buffer![4i16, 9, 3].into_array())] +#[case::i64(buffer![1i64, 9, 3].into_array(), buffer![4i64, 2, 3].into_array(), buffer![4i64, 9, 3].into_array())] +#[case::u8(buffer![1u8, 9, 3].into_array(), buffer![4u8, 2, 3].into_array(), buffer![4u8, 9, 3].into_array())] +fn dispatches_at_each_integer_width( + #[case] lhs: ArrayRef, + #[case] rhs: ArrayRef, + #[case] expected: ArrayRef, +) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let result = apply(Max, [lhs, rhs], &mut ctx)?; + + assert_arrays_eq!(result, expected, &mut ctx); + Ok(()) +} + +#[test] +fn rejects_a_float_width() { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![1.0f64].into_array(); + let rhs = buffer![2.0f64].into_array(); + + let error = apply(Max, [lhs, rhs], &mut ctx) + .expect_err("a float width must be rejected at construction"); + + assert!( + error.to_string().contains("integer inputs"), + "unexpected error: {error}" + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/lifting.rs b/vortex-array/src/scalar_fn/row/tests/lifting.rs new file mode 100644 index 00000000000..f7d2fba4d04 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/lifting.rs @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for null propagation, constant folding, nullability widening, and options serde. + +use super::*; +use crate::dtype::Nullability; +use crate::dtype::PType; + +/// An `i32` element that is [dense-safe] iff `DENSE`, and otherwise the plain `i32` element in +/// every respect. Dense-safety is what decides the null-handling path, so a pair of these is +/// how one kernel gets run under both. +/// +/// [dense-safe]: InputElement::DENSE_SAFE +struct MaybeDenseI32; + +impl InputElement for MaybeDenseI32 { + type Column = ::Column; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i32; + + const DENSE_SAFE: bool = DENSE; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i32 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i32 + where + Self: 'a, + { + ::get_varying(column, index) + } +} + +/// Wrapping addition over two [`MaybeDenseI32`] columns. +#[derive(Clone)] +struct Add; + +impl RowFn for Add { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + + fn id(&self) -> ScalarFnId { + if DENSE { + static ID: CachedId = CachedId::new("vortex.test.add.dense"); + *ID + } else { + static ID: CachedId = CachedId::new("vortex.test.add.filter"); + *ID + } + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::< + (MaybeDenseI32, MaybeDenseI32), + ElementSink, + _, + _, + >( + |_| (), + |&(), (lhs, rhs), output| *output = lhs.wrapping_add(rhs), + ) + } +} + +/// Adds `lhs` to `rhs` under both null-handling paths and asserts each result equals +/// `expected`, which is what every case below does. +/// +/// Forcing a _strategy_ within the filter contract is a separate axis, covered in +/// [`null_strategies`](super::null_strategies). +fn assert_add(lhs: ArrayRef, rhs: ArrayRef, expected: ArrayRef) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let dense = apply(Add::, [lhs.clone(), rhs.clone()], &mut ctx)?; + let filtered = apply(Add::, [lhs, rhs], &mut ctx)?; + + let args = [ + DType::Primitive(PType::I32, Nullability::NonNullable), + DType::Primitive(PType::I32, Nullability::NonNullable), + ]; + assert_eq!(policy(&Add::, &args), RowPolicy::Dense); + assert_eq!( + policy(&Add::, &args), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + assert_arrays_eq!(dense, expected, &mut ctx); + assert_arrays_eq!(filtered, expected, &mut ctx); + Ok(()) +} + +#[test] +fn no_nulls() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), + PrimitiveArray::from_iter([10i32, 20, 30]).into_array(), + PrimitiveArray::from_iter([11i32, 22, 33]).into_array(), + ) +} + +#[test] +fn nulls_propagate() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_option_iter([Some(1i32), None, Some(3), None]).into_array(), + PrimitiveArray::from_option_iter([Some(10i32), Some(20), None, None]).into_array(), + PrimitiveArray::from_option_iter([Some(11i32), None, None, None]).into_array(), + ) +} + +/// Strictness: a null constant makes the whole output null without the kernel running at all. +#[test] +fn null_constant_short_circuits() -> VortexResult<()> { + let null = Scalar::null(DType::Primitive(PType::I32, Nullability::Nullable)); + + assert_add( + PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), + ConstantArray::new(null, 3).into_array(), + PrimitiveArray::from_option_iter([Option::::None, None, None]).into_array(), + ) +} + +/// All-constant inputs evaluate one row and broadcast it. +#[test] +fn all_constants_broadcast() -> VortexResult<()> { + assert_add( + ConstantArray::new(Scalar::from(2i32), 4).into_array(), + ConstantArray::new(Scalar::from(40i32), 4).into_array(), + PrimitiveArray::from_iter([42i32, 42, 42, 42]).into_array(), + ) +} + +#[test] +fn mixed_constant_and_column() -> VortexResult<()> { + assert_add( + PrimitiveArray::from_option_iter([Some(1i32), None, Some(3)]).into_array(), + ConstantArray::new(Scalar::from(10i32), 3).into_array(), + PrimitiveArray::from_option_iter([Some(11i32), None, Some(13)]).into_array(), + ) +} + +/// An empty batch is neither all-valid nor all-null, and a zero-length non-nullable execution +/// keeps its non-nullable dtype. +#[test] +fn empty_input_keeps_dtype() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let empty = || PrimitiveArray::from_iter(Vec::::new()).into_array(); + + let result = apply(Add::, [empty(), empty()], &mut ctx)?; + + assert_eq!(result.len(), 0); + assert!(!result.dtype().is_nullable()); + Ok(()) +} + +/// The output element dtype is non-nullable, and the lifting widens it iff an input is +/// nullable, which is what makes strictness's dtype contract hold by construction. +#[test] +fn return_dtype_unions_nullability() -> VortexResult<()> { + let non_nullable = DType::Primitive(PType::I32, Nullability::NonNullable); + let nullable = non_nullable.as_nullable(); + + assert_eq!( + ScalarFnVTable::return_dtype( + &Add::, + &EmptyOptions, + &[non_nullable.clone(), non_nullable.clone()] + )?, + non_nullable + ); + assert_eq!( + ScalarFnVTable::return_dtype( + &Add::, + &EmptyOptions, + &[non_nullable, nullable.clone()] + )?, + nullable + ); + Ok(()) +} + +#[test] +fn a_row_fn_is_strict() { + assert!(ScalarFnVTable::is_strict(&Add::, &EmptyOptions)); +} + +/// Output sinks build an all-valid column, so the output validity is exactly the child +/// conjunction and the planner never has to execute the function to learn which rows are null. +#[test] +fn validity_is_the_child_conjunction() -> VortexResult<()> { + let expr = Add::.new_expr(EmptyOptions, [root(), root()]); + + assert!(ScalarFnVTable::validity(&Add::, &EmptyOptions, &expr)?.is_some()); + Ok(()) +} + +/// A row function is not serializable until the function opts into a wire representation. +#[test] +fn options_are_not_serializable_by_default() -> VortexResult<()> { + assert_eq!( + ScalarFnVTable::serialize(&Add::, &EmptyOptions)?, + None + ); + assert!(ScalarFnVTable::deserialize(&Add::, &[], &array_session()).is_err()); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/mod.rs b/vortex-array/src/scalar_fn/row/tests/mod.rs new file mode 100644 index 00000000000..c8f2ab77aee --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/mod.rs @@ -0,0 +1,509 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! End-to-end tests for row function execution. + +use rstest::rstest; +use vortex_buffer::ByteBuffer; +use vortex_buffer::buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_session::registry::CachedId; + +use super::lift::RowPolicy; +use super::vtable::row_policy; +use crate::ArrayRef; +use crate::Canonical; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::VortexSessionExecute; +use crate::array_session; +use crate::arrays::ConstantArray; +use crate::arrays::MaskedArray; +use crate::arrays::PrimitiveArray; +use crate::arrays::VarBinViewArray; +use crate::arrays::scalar_fn::ScalarFnFactoryExt; +use crate::arrays::varbinview::BinaryView; +use crate::assert_arrays_eq; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::expr::root; +use crate::scalar::Scalar; +use crate::scalar_fn::*; + +mod conformance; +mod constant_operands; +mod decode_fallibility; +mod dispatched; +mod lifting; +mod null_strategies; +mod nullable_outputs; +mod prepared; +mod sink; + +/// Builds `scalar_fn` over `args` and executes it end to end, which is what every test below does. +fn apply>( + scalar_fn: F, + args: impl IntoIterator, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = args.into_iter().collect::>(); + let rows = args.first().map_or(0, |arg| arg.len()); + + Ok(scalar_fn + .try_new_array(rows, EmptyOptions, args)? + .execute::(ctx)? + .into_array()) +} + +/// A binary row function over fixed primitive types: `hypot(x, y)`. +#[derive(Clone)] +struct Hypot; + +impl RowFn for Hypot { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["x", "y"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.hypot"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(f64, f64), ElementSink, _, _>( + |_| (), + |&(), (x, y), output| *output = x.hypot(y), + ) + } +} + +/// A byte-string element that resolves each row's view into a data buffer, which is only +/// meaningful for a valid row. +/// +/// This is the crate's only non-dense-safe element, so it exercises valid-only execution, +/// branch-and-skip, and the agreement between the two strategies. It lives here rather than beside +/// the framework because no production row function reads bytes yet. +struct TestBytes; + +/// The canonical views array plus its resolved data buffers. +struct TestBytesColumn { + array: VarBinViewArray, + buffers: Vec, +} + +/// Resolve one view, which is either inlined or an offset into `buffers`. +fn read_view<'a>(view: &'a BinaryView, buffers: &'a [ByteBuffer]) -> &'a [u8] { + if view.is_inlined() { + view.as_inlined().value() + } else { + let view = view.as_view(); + &buffers[view.buffer_index as usize].as_slice()[view.as_range()] + } +} + +impl InputElement for TestBytes { + type Column = TestBytesColumn; + // The views slice, not the array: `VarBinViewArray::views` resolves a host buffer and its + // `vortex_expect` is a side effect the optimizer cannot hoist out of the row loop. + type Varying<'a> = (&'a [BinaryView], &'a [ByteBuffer]); + type Elem<'a> = &'a [u8]; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + matches!(dtype, DType::Utf8(_) | DType::Binary(_)), + "expected a Utf8 or Binary column, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let array = array.execute::(ctx)?; + let buffers = (0..array.data_buffers().len()) + .map(|idx| array.buffer(idx).clone()) + .collect(); + Ok(TestBytesColumn { array, buffers }) + } + + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Self::decode(array, ctx).map(Some) + } + + fn get(column: &Self::Column, index: usize) -> &[u8] { + read_view(&column.array.views()[index], &column.buffers) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + (column.array.views(), &column.buffers) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.0.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a [u8] + where + Self: 'a, + { + read_view(&column.0[index], column.1) + } +} + +impl OutputElement for String { + fn element_dtype() -> DType { + DType::Utf8(Nullability::NonNullable) + } + + fn build(values: Vec) -> ArrayRef { + VarBinViewArray::from_iter_str(values).into_array() + } + + fn placeholder() -> Self { + String::new() + } +} + +/// A unary row function over strings: uppercased text, exercising [`TestBytes`] input and +/// [`String`] output. +#[derive(Clone)] +struct Shout; + +impl RowFn for Shout { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.shout"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(TestBytes,), ElementSink, _, _>( + |_| (), + |&(), (text,), output| { + *output = String::from_utf8_lossy(text).to_uppercase(); + }, + ) + } +} + +/// A fallible row function: integer division, undefined at a zero divisor. +#[derive(Clone)] +struct CheckedDiv; + +impl RowFn for CheckedDiv { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.checked_div"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| { + if rhs == 0 { + vortex_bail!("division by zero"); + } + *output = lhs / rhs; + Ok(()) + }, + ) + } +} + +#[test] +fn hypot_columns() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0].into_array(); + let y = buffer![4.0f64, 12.0].into_array(); + + let result = apply(Hypot, [x, y], &mut ctx)?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([5.0f64, 13.0]), &mut ctx); + Ok(()) +} + +#[test] +fn hypot_propagates_nulls_and_constants() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = PrimitiveArray::from_option_iter([Some(3.0f64), None, Some(8.0)]).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + + let result = apply(Hypot, [x, y], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} + +#[test] +fn shout_strings() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = + VarBinViewArray::from_iter_nullable_str([Some("hello"), None, Some("Vortex")]).into_array(); + + let result = apply(Shout, [input], &mut ctx)?; + + let expected = + VarBinViewArray::from_iter_nullable_str([Some("HELLO"), None, Some("VORTEX")]).into_array(); + assert_arrays_eq!(result, expected, &mut ctx); + Ok(()) +} + +#[test] +fn display_names_the_function_id() { + let expr = Hypot.new_expr(EmptyOptions, [root(), root()]); + assert_eq!(expr.to_string(), "vortex.test.hypot($, $)"); +} + +#[derive(Clone)] +struct WrongLength; + +impl RowFn for WrongLength { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.wrong_length"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = value, + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + _args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + Ok(Some(PrimitiveArray::from_iter([0i64]).into_array())) + } +} + +#[test] +fn kernel_result_length_is_validated() { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, 2, 3].into_array(); + + let error = apply(WrongLength, [input], &mut ctx).unwrap_err(); + + assert!( + error + .to_string() + .contains("produced 1 rows for 3 input rows"), + "{error}" + ); +} + +#[derive(Clone)] +struct FortyTwo; + +impl RowFn for FortyTwo { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &[]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.forty_two"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(), ElementSink, _, _>( + |()| (), + |&(), (), output| *output = 42, + ) + } +} + +#[test] +fn nullary_row_fn_executes_requested_rows() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let result = FortyTwo + .try_new_array(3, EmptyOptions, [])? + .execute::(&mut ctx)?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([42i64; 3]), &mut ctx); + Ok(()) +} + +#[derive(Clone)] +struct SumFour; + +impl RowFn for SumFour { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["a", "b", "c", "d"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.sum_four"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64, i64, i64, i64), ElementSink, _, _>( + |_| (), + |&(), (a, b, c, d), output| *output = a + b + c + d, + ) + } +} + +#[test] +fn four_argument_row_fn_executes() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let result = apply( + SumFour, + [ + buffer![1i64, 2].into_array(), + buffer![10i64, 20].into_array(), + buffer![100i64, 200].into_array(), + buffer![1000i64, 2000].into_array(), + ], + &mut ctx, + )?; + + assert_arrays_eq!(result, PrimitiveArray::from_iter([1111i64, 2222]), &mut ctx); + Ok(()) +} + +#[test] +fn tuples_are_supported_through_arity_twelve() { + type TwelveI64s = (i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64); + + assert_eq!(<() as ElementTuple>::ARITY, 0); + assert_eq!(::ARITY, 12); +} + +#[test] +fn kernel_flag_decides_fallibility() { + assert!(!ScalarFnVTable::is_fallible(&Hypot, &EmptyOptions)); + assert!(ScalarFnVTable::is_fallible(&CheckedDiv, &EmptyOptions)); +} + +#[test] +fn fallible_apply_propagates_its_error() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10].into_array(); + let rhs = buffer![2i64, 0].into_array(); + + let error = apply(CheckedDiv, [lhs, rhs], &mut ctx) + .expect_err("a zero divisor must fail the execution"); + + assert!( + error.to_string().contains("division by zero"), + "unexpected error: {error}" + ); + Ok(()) +} + +/// The divisor's null slot holds a zero, which a dense pass would divide by. Filtering keeps the +/// fallible kernel away from it. +#[test] +fn fallible_apply_never_sees_rows_behind_nulls() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10].into_array(); + let rhs = PrimitiveArray::from_option_iter([Some(2i64), None]).into_array(); + + let result = apply(CheckedDiv, [lhs, rhs], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5i64), None]), + &mut ctx + ); + Ok(()) +} + +/// The internal nullable execution policy selected by a concrete dispatch. +fn policy>(row_fn: &F, args: &[DType]) -> RowPolicy { + row_policy(row_fn, &EmptyOptions, args).expect("test dispatch must produce a policy") +} + +/// The function never declares a policy: the dispatched arguments and result decide it. +#[test] +fn null_handling_follows_from_args_and_fallibility() { + // Primitive arguments, infallible: nothing behind a null row can fault. + assert_eq!( + policy( + &Hypot, + &[ + DType::Primitive(PType::F64, Nullability::NonNullable), + DType::Primitive(PType::F64, Nullability::NonNullable), + ] + ), + RowPolicy::Dense + ); + // `TestBytes` resolves a view into a data buffer, which is only meaningful for valid rows. + assert_eq!( + policy(&Shout, &[DType::Utf8(Nullability::Nullable)]), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + // Fallible: a garbage row could raise an error of its own. + assert_eq!( + policy( + &CheckedDiv, + &[ + DType::Primitive(PType::I64, Nullability::NonNullable), + DType::Primitive(PType::I64, Nullability::NonNullable), + ] + ), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); +} diff --git a/vortex-array/src/scalar_fn/row/tests/null_strategies.rs b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs new file mode 100644 index 00000000000..487cb4786f6 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/null_strategies.rs @@ -0,0 +1,502 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Equivalence and selection tests for filtering and branch-and-skip null execution. + +use std::sync::Arc; + +use vortex_buffer::ByteBuffer; + +use super::*; +use crate::arrays::varbinview::BinaryView; +use crate::dtype::Nullability; +use crate::validity::Validity; + +/// Executes `scalar_fn` over `args` with `strategy` forced, canonicalized like [`apply`]. +fn apply_forced>( + scalar_fn: &F, + args: &[ArrayRef], + strategy: NullStrategy, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let rows = args.first().map_or(0, |arg| arg.len()); + + Ok( + execute_row_fn_with_strategy(scalar_fn, &EmptyOptions, args.to_vec(), rows, strategy, ctx)? + .execute::(ctx)? + .into_array(), + ) +} + +/// Runs `scalar_fn` under forced filter, forced branch-and-skip, and the automatic per-batch +/// selection, and asserts all three produce identical arrays. +fn assert_strategies_agree>( + scalar_fn: F, + args: Vec, +) -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + + let filtered = apply_forced(&scalar_fn, &args, NullStrategy::Filter, &mut ctx)?; + let branched = apply_forced(&scalar_fn, &args, NullStrategy::BranchAndSkip, &mut ctx)?; + let auto = apply(scalar_fn, args, &mut ctx)?; + + assert_arrays_eq!(branched, filtered, &mut ctx); + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) +} + +/// A `Utf8` column whose null rows carry views naming a buffer that does not exist, at +/// offsets far out of bounds. Resolving such a row's bytes panics, so strategy agreement +/// proves the branch loop never calls `get` behind a null. +fn hostile_nullable_strings() -> VortexResult { + let views = buffer![ + BinaryView::make_view(b"a longer string here", 0, 0), + BinaryView::new_ref(64, *b"junk", 9, 4096), + BinaryView::make_view(b"another non-inlined string", 1, 0), + BinaryView::new_ref(64, *b"junk", 7, 1 << 20), + ]; + + Ok(VarBinViewArray::try_new( + views, + Arc::from([ + ByteBuffer::copy_from(b"a longer string here"), + ByteBuffer::copy_from(b"another non-inlined string"), + ]), + DType::Utf8(Nullability::Nullable), + Validity::from_iter([true, false, true, false]), + )? + .into_array()) +} + +/// `Bytes` is not dense-safe, so `Shout` uses valid-only execution; both strategies must produce +/// the same array without resolving the hostile views behind the nulls. +#[test] +fn branch_matches_filter_for_bytes() -> VortexResult<()> { + assert_strategies_agree(Shout, vec![hostile_nullable_strings()?]) +} + +/// A fallible kernel with a poison value (zero divisor) behind every null: the branch loop +/// must skip those rows rather than spuriously failing on them. +#[test] +fn branch_never_applies_a_fallible_kernel_behind_nulls() -> VortexResult<()> { + let lhs = buffer![10i64, 10, 12, 9].into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 0, 3, 0], + Validity::from_iter([true, false, true, false]), + ) + .into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// Nulls in both operands: the branch loop must honor the _conjoined_ mask, not either +/// input's own validity. +#[test] +fn branch_conjoins_validities() -> VortexResult<()> { + let lhs = + PrimitiveArray::from_option_iter([Some(10i64), None, Some(12), Some(9), None]).into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 0, 3, 0, 0], + Validity::from_iter([true, true, true, false, false]), + ) + .into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// A constant operand under the branch strategy still hoists through the stride-0 decode. +#[test] +fn branch_handles_constant_operands() -> VortexResult<()> { + let lhs = PrimitiveArray::from_option_iter([Some(10i64), None, Some(12)]).into_array(); + let rhs = ConstantArray::new(Scalar::from(2i64), 3).into_array(); + + assert_strategies_agree(CheckedDiv, vec![lhs, rhs]) +} + +/// An error from a _valid_ row still propagates under the branch strategy. +#[test] +fn branch_propagates_real_errors() { + let mut ctx = array_session().create_execution_ctx(); + let lhs = buffer![10i64, 10, 12].into_array(); + let rhs = PrimitiveArray::new( + buffer![2i64, 3, 0], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let error = apply_forced( + &CheckedDiv, + &[lhs, rhs], + NullStrategy::BranchAndSkip, + &mut ctx, + ) + .expect_err("a zero divisor in a valid row must fail"); + + assert!( + error.to_string().contains("division by zero"), + "unexpected error: {error}" + ); +} + +/// The automatic per-batch selection, observed through elements that record which decode ran +/// on how many rows: the branch strategy decodes null-tolerantly at full length, the filter +/// strategy decodes ordinarily over the survivors. +mod selection { + use std::cell::Cell; + use std::cell::RefCell; + + use vortex_buffer::Buffer; + use vortex_error::vortex_err; + use vortex_mask::Mask; + + use super::*; + use crate::scalar_fn::row::lift::branch_beats_filter; + + thread_local! { + /// What the last varying-column decode did: `(null_tolerant, rows)`. Thread-local so + /// concurrent tests in one process cannot race it; execution runs on the calling + /// thread. + static LAST_DECODE: Cell> = const { Cell::new(None) }; + } + + /// An i64 element that records its decodes and reports `COST` units of filtered decode work. + /// It is not dense-safe, so strategy selection actually happens. + struct TrackedI64; + + impl InputElement for TrackedI64 { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + const FILTERED_DECODE_COST: usize = COST; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + LAST_DECODE.set(Some((false, array.len()))); + ::decode(array, ctx) + } + + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + LAST_DECODE.set(Some((true, array.len()))); + ::decode(array, ctx).map(Some) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } + } + + /// Negation over one tracked column. + #[derive(Clone)] + struct TrackedNegate; + + impl RowFn for TrackedNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + if COST == 0 { + static ID: CachedId = CachedId::new("vortex.test.tracked_negate.bulk"); + *ID + } else { + static ID: CachedId = CachedId::new("vortex.test.tracked_negate.per_row"); + *ID + } + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(TrackedI64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + } + + /// A 32-row nullable i64 column whose first `valid_count` rows are valid. + fn column_with_survivors(valid_count: usize) -> ArrayRef { + PrimitiveArray::from_option_iter( + (0..32u16).map(|i| (usize::from(i) < valid_count).then_some(i64::from(i))), + ) + .into_array() + } + + /// Executes the tracked function through the full pipeline and returns what the decode + /// recorded: whether it was null-tolerant, and how many rows it saw. + fn run(valid_count: usize) -> VortexResult<(bool, usize)> { + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + + apply( + TrackedNegate::, + [column_with_survivors(valid_count)], + &mut ctx, + )?; + + LAST_DECODE + .get() + .ok_or_else(|| vortex_err!("no decode ran")) + } + + /// A bulk-decoded element takes branch-and-skip on a mixed mask however sparse the + /// survivors: the decode is null-tolerant and full length. + #[test] + fn bulk_decode_branches_at_any_density() -> VortexResult<()> { + assert_eq!(run::<0>(31)?, (true, 32)); + assert_eq!(run::<0>(4)?, (true, 32)); + Ok(()) + } + + /// One per-row decode still branches when half the rows survive, matching the measured + /// single-nullable-input crossover. + #[test] + fn per_row_decode_filters_when_sparse() -> VortexResult<()> { + // 30/32 surviving: branch, full-length null-tolerant decode. + assert_eq!(run::<1>(30)?, (true, 32)); + // 16/32 = 50% surviving sits exactly on the threshold: still branch. + assert_eq!(run::<1>(16)?, (true, 32)); + // Below 50% surviving: filter, ordinary decode over the survivors. + assert_eq!(run::<1>(15)?, (false, 15)); + Ok(()) + } + + /// An all-true mask short-circuits to the plain kernel and an all-false mask to an + /// all-null constant, before any strategy is selected. + #[test] + fn degenerate_masks_bypass_the_selection() -> VortexResult<()> { + assert_eq!(run::<1>(32)?, (false, 32)); + + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + apply(TrackedNegate::<1>, [column_with_survivors(0)], &mut ctx)?; + assert_eq!(LAST_DECODE.get(), None); + Ok(()) + } + + /// An i64 element that omits `decode_null_tolerant`: the conservative default refuses, so + /// the batch must fall back to the filter strategy even though the selection preferred + /// branch. + struct RefusesNullTolerant; + + impl InputElement for RefusesNullTolerant { + type Column = Buffer; + type Varying<'a> = ::Varying<'a>; + type Elem<'a> = i64; + + const DENSE_SAFE: bool = false; + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + ::validate(dtype) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + LAST_DECODE.set(Some((false, array.len()))); + ::decode(array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> i64 { + ::get(column, index) + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + ::varying(column) + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + ::varying_len(column) + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> i64 + where + Self: 'a, + { + ::get_varying(column, index) + } + } + + #[derive(Clone)] + struct RefusingNegate; + + impl RowFn for RefusingNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.refusing_negate"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(RefusesNullTolerant,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + } + + /// The fallback is silent and correct: the ordinary decode runs over the survivors and + /// the result matches the expected negation. + #[test] + fn missing_null_tolerant_decode_falls_back_to_filter() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + LAST_DECODE.set(None); + + let result = apply( + RefusingNegate, + [PrimitiveArray::from_option_iter([Some(3i64), None, Some(5)]).into_array()], + &mut ctx, + )?; + + assert_eq!(LAST_DECODE.get(), Some((false, 2))); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(-3i64), None, Some(-5)]), + &mut ctx + ); + Ok(()) + } + + thread_local! { + /// Every `row_count` `reduce_encoded` was handed, in call order. + static REDUCE_ROW_COUNTS: RefCell> = const { RefCell::new(Vec::new()) }; + } + + /// [`RefusingNegate`] with an encoding-aware rewrite that declines, recording the row count it + /// was offered. + #[derive(Clone)] + struct ProbingNegate; + + impl RowFn for ProbingNegate { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.probing_negate"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(RefusesNullTolerant,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = -value, + ) + } + + fn reduce_encoded( + &self, + _options: &Self::Options, + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + REDUCE_ROW_COUNTS.with_borrow_mut(|counts| counts.push(args[0].len())); + Ok(None) + } + } + + /// A `reduce_encoded` rewrite must be sized from the arrays it was handed, which under the + /// filter strategy hold the surviving rows rather than the whole batch. This is the only + /// mixed-mask path where the two differ, so nothing else would catch a rewrite sized from a + /// length captured elsewhere. + /// + /// This also pins the double probe as deliberate. The first call sees the original arrays at + /// full length, and is the only one that does; the second sees filtered, canonical copies. An + /// "optimization" that skipped the first because the batch will end up filtering would take an + /// encoding-aware rewrite away from every function whose sink cannot skip rows. + #[test] + fn reduce_encoded_is_probed_before_and_after_filtering() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + REDUCE_ROW_COUNTS.with_borrow_mut(Vec::clear); + + let result = apply( + ProbingNegate, + [PrimitiveArray::from_option_iter([Some(3i64), None, Some(5)]).into_array()], + &mut ctx, + )?; + + assert_eq!( + REDUCE_ROW_COUNTS.with_borrow(|counts| counts.clone()), + vec![3, 2], + "expected an unfiltered probe at the batch length, then a filtered one at the \ + surviving count", + ); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(-3i64), None, Some(-5)]), + &mut ctx + ); + Ok(()) + } + + /// The rule itself, at and around the threshold, without going through an execution. + #[rstest] + #[case::bulk_dense_mask(0, 99, 100, true)] + #[case::bulk_sparse_mask(0, 1, 100, true)] + #[case::one_decode_dense_mask(1, 99, 100, true)] + #[case::one_decode_at_threshold(1, 50, 100, true)] + #[case::one_decode_below_threshold(1, 49, 100, false)] + #[case::two_decodes_at_old_boolean_choice(2, 81, 100, false)] + #[case::two_decodes_dense_mask(2, 90, 100, true)] + fn selects_branch_per_the_measured_rule( + #[case] filtered_decode_cost: usize, + #[case] true_count: usize, + #[case] len: usize, + #[case] expect_branch: bool, + ) { + let valid = Mask::from_indices(len, 0..true_count); + assert_eq!( + branch_beats_filter(filtered_decode_cost, &valid), + expect_branch, + ); + } + + #[test] + fn planning_adds_decode_cost_across_arguments() { + assert_eq!( + RowPolicy::for_sink::<(TrackedI64<1>, TrackedI64<1>), ()>(), + RowPolicy::ValidOnly { + filtered_decode_cost: 2 + } + ); + } +} diff --git a/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs b/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs new file mode 100644 index 00000000000..50ea77ef015 --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/nullable_outputs.rs @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests that row output dtypes cannot introduce their own nulls. + +use super::*; +use crate::dtype::Nullability; +use crate::dtype::PType; + +#[derive(Clone)] +struct NullableI64(i64); + +impl OutputElement for NullableI64 { + fn element_dtype() -> DType { + DType::Primitive(PType::I64, Nullability::Nullable) + } + + fn build(values: Vec) -> ArrayRef { + PrimitiveArray::from_option_iter(values.into_iter().map(|value| Some(value.0))).into_array() + } + + fn placeholder() -> Self { + Self(0) + } +} + +struct NullableSink(usize); + +impl OutputSink for NullableSink { + type Rows<'a> = usize; + type Row<'a> = (); + + fn sink_dtype(_args: &[DType]) -> VortexResult { + Ok(DType::Primitive(PType::I64, Nullability::Nullable)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self(rows)) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.0 + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + *rows == row_count + } + + fn row<'a>(_rows: &'a mut Self::Rows<'_>, _index: usize) -> Self::Row<'a> {} + + fn finish(self, _error: DeferredError) -> VortexResult { + Ok(PrimitiveArray::from_option_iter(Vec::>::new()).into_array()) + } +} + +#[derive(Clone)] +struct NullableElementFn; + +impl RowFn for NullableElementFn { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.nullable_element"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), ElementSink, _, _>( + |_| (), + |&(), (value,), output| *output = NullableI64(value), + ) + } +} + +#[derive(Clone)] +struct NullableSinkFn; + +impl RowFn for NullableSinkFn { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.nullable_sink"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), NullableSink, _, _>(|_| (), |&(), _, ()| {}) + } +} + +#[test] +fn nullable_element_dtype_is_rejected() { + let input = DType::Primitive(PType::I64, Nullability::NonNullable); + let error = + ScalarFnVTable::return_dtype(&NullableElementFn, &EmptyOptions, &[input]).unwrap_err(); + + assert!(error.to_string().contains("non-nullable dtype"), "{error}"); +} + +#[test] +fn nullable_sink_dtype_is_rejected() { + let input = DType::Primitive(PType::I64, Nullability::NonNullable); + let error = ScalarFnVTable::return_dtype(&NullableSinkFn, &EmptyOptions, &[input]).unwrap_err(); + + assert!(error.to_string().contains("non-nullable dtype"), "{error}"); +} diff --git a/vortex-array/src/scalar_fn/row/tests/prepared.rs b/vortex-array/src/scalar_fn/row/tests/prepared.rs new file mode 100644 index 00000000000..64ed82c8d2c --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/prepared.rs @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for preparing batch-constant state once before the row loop. + +use std::cell::Cell; + +use super::*; +use crate::validity::Validity; + +thread_local! { + /// Which operands the last `prepare` saw as constant, as a bitmask (bit 0 for `x`, bit 1 + /// for `y`). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; +} + +/// `sqrt(x^2 + y^2)` through [`RowVisitor::visit_prepared_into`]: the square of any constant +/// operand is hoisted out of the row loop, and recorded in [`SEEN_CONSTANTS`]. +#[derive(Clone)] +struct PreparedHypot; + +impl RowFn for PreparedHypot { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["x", "y"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.prepared_hypot"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(f64, f64), ElementSink, _, _>( + |(x, y)| { + SEEN_CONSTANTS.set(u8::from(x.is_some()) | (u8::from(y.is_some()) << 1)); + (x.map(|x| x * x), y.map(|y| y * y)) + }, + |&(x_sq, y_sq), (x, y), output| { + *output = (x_sq.unwrap_or(x * x) + y_sq.unwrap_or(y * y)).sqrt(); + }, + ) + } +} + +/// A constant operand reaches `prepare` as `Some`, and the result is identical to the same +/// value expanded into a full column, which reaches `prepare` as `None`. +#[test] +fn a_constant_operand_matches_its_expanded_column() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0, 8.0].into_array(); + + let constant = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + let from_constant = apply(PreparedHypot, [x.clone(), constant], &mut ctx)?; + assert_eq!(SEEN_CONSTANTS.get(), 0b10); + + let expanded = buffer![4.0f64, 4.0, 4.0].into_array(); + let from_expanded = apply(PreparedHypot, [x, expanded], &mut ctx)?; + assert_eq!(SEEN_CONSTANTS.get(), 0b00); + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// A masked constant (the same value in every row, some rows null, how the compressor spells +/// an all-same-with-nulls chunk) is a batch constant too: the wrapper carries only validity, +/// which the lifting owns, so `prepare` sees the child's value and the null rows stay +/// null in the result. +#[test] +fn a_masked_constant_operand_is_seen_as_constant() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0, 8.0].into_array(); + + let masked_constant = MaskedArray::try_new( + ConstantArray::new(Scalar::from(4.0f64), 3).into_array(), + Validity::from_iter([true, false, true]), + )? + .into_array(); + let result = apply(PreparedHypot, [x, masked_constant], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b10); + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} + +/// With no constant operand every `ConstElems` slot is `None` and the loop computes exactly +/// what unit preparation would. +#[test] +fn all_varying_operands_prepare_nothing() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = buffer![3.0f64, 5.0].into_array(); + let y = buffer![4.0f64, 12.0].into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b00); + assert_arrays_eq!(result, PrimitiveArray::from_iter([5.0f64, 13.0]), &mut ctx); + Ok(()) +} + +/// Two constant operands are folded to a single-row execution by the lifting, and that +/// row still goes through `prepare`, seeing both constants. +#[test] +fn all_constant_operands_fold_and_still_prepare() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = ConstantArray::new(Scalar::from(3.0f64), 4).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 4).into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_eq!(SEEN_CONSTANTS.get(), 0b11); + assert_arrays_eq!( + result, + PrimitiveArray::from_iter([5.0f64, 5.0, 5.0, 5.0]), + &mut ctx + ); + Ok(()) +} + +/// Null rows pass through the prepared path exactly as through unit preparation. +#[test] +fn nulls_propagate_through_the_prepared_path() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let x = PrimitiveArray::from_option_iter([Some(3.0f64), None, Some(8.0)]).into_array(); + let y = ConstantArray::new(Scalar::from(4.0f64), 3).into_array(); + + let result = apply(PreparedHypot, [x, y], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(5.0f64), None, Some((80.0f64).sqrt())]), + &mut ctx + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/tests/sink.rs b/vortex-array/src/scalar_fn/row/tests/sink.rs new file mode 100644 index 00000000000..45b91ed191d --- /dev/null +++ b/vortex-array/src/scalar_fn/row/tests/sink.rs @@ -0,0 +1,375 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for row functions that write into a batch-wide output sink. + +use std::sync::Arc; + +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; + +use super::*; +use crate::arrays::FixedSizeListArray; +use crate::dtype::NativePType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::validity::Validity; + +/// Builds a `FixedSizeList` column, presenting each row as the `&mut [T]` slice to fill. +/// +/// Its element dtype comes from the input rather than from `T` alone, so it exercises +/// [`OutputSink::sink_dtype`] actually reading `args`. +struct SpreadSink { + dtype: DType, + rows: usize, + elements: BufferMut, +} + +impl OutputSink for SpreadSink { + type Rows<'a> = (&'a mut [T], usize); + type Row<'a> = &'a mut [T]; + + fn sink_dtype(args: &[DType]) -> VortexResult { + let element = args + .first() + .ok_or_else(|| vortex_err!("a spread sink takes its element dtype from its input"))?; + ::validate(element)?; + Ok(DType::FixedSizeList( + Arc::new(element.as_nonnullable()), + u32::try_from(W).vortex_expect("test width fits in u32"), + Nullability::NonNullable, + )) + } + + fn with_capacity(rows: usize, dtype: &DType) -> VortexResult { + Ok(Self { + dtype: dtype.clone(), + rows, + elements: BufferMut::zeroed(rows * W), + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + (self.elements.as_mut_slice(), self.rows) + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.1 == row_count && row_count.checked_mul(W) == Some(rows.0.len()) + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> &'a mut [T] { + &mut rows.0[index * W..][..W] + } + + fn finish(self, _error: DeferredError) -> VortexResult { + vortex_ensure_eq!( + self.dtype, + Self::sink_dtype(&[DType::Primitive(T::PTYPE, self.dtype.nullability())])?, + "the sink must build the dtype it named", + ); + Ok(FixedSizeListArray::try_new( + PrimitiveArray::new(self.elements.freeze(), Validity::NonNullable).into_array(), + u32::try_from(W).vortex_expect("test width fits in u32"), + Validity::NonNullable, + self.rows, + )? + .into_array()) + } +} + +/// A sink that reports a data-dependent error only after every row has been written. +struct NonNegativeSink { + values: BufferMut, +} + +struct NonNegativeRow<'a> { + value: &'a mut i64, +} + +impl NonNegativeRow<'_> { + fn write(self, value: i64) -> bool { + *self.value = value; + value < 0 + } +} + +impl OutputSink for NonNegativeSink { + const ERRORS_ARE_DEFERRED: bool = true; + + type Rows<'a> = &'a mut [i64]; + type Row<'a> = NonNegativeRow<'a>; + + fn sink_dtype(args: &[DType]) -> VortexResult { + let dtype = args + .first() + .ok_or_else(|| vortex_err!("a non-negative sink requires one input"))?; + ::validate(dtype)?; + Ok(DType::Primitive(PType::I64, Nullability::NonNullable)) + } + + fn with_capacity(rows: usize, _dtype: &DType) -> VortexResult { + Ok(Self { + values: BufferMut::zeroed(rows), + }) + } + + fn rows(&mut self) -> Self::Rows<'_> { + self.values.as_mut_slice() + } + + fn row_count_matches(rows: &Self::Rows<'_>, row_count: usize) -> bool { + rows.len() == row_count + } + + fn row<'a>(rows: &'a mut Self::Rows<'_>, index: usize) -> Self::Row<'a> { + NonNegativeRow { + value: &mut rows[index], + } + } + + fn finish(self, error: DeferredError) -> VortexResult { + if error.occurred() { + vortex_bail!("negative output"); + } + Ok(PrimitiveArray::new(self.values.freeze(), Validity::NonNullable).into_array()) + } +} + +/// Broadcasts each input value across a fixed-size list row: `spread(x) == [x, x, x]`. +#[derive(Clone)] +struct Spread; + +impl RowFn for Spread { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.spread"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), SpreadSink, _, _>( + |_| (), + |&(), (x,), out| out.fill(x), + ) + } +} + +/// The same, but refusing negative inputs, so its row closure returns `VortexResult<()>`. +#[derive(Clone)] +struct SpreadNonNegative; + +impl RowFn for SpreadNonNegative { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.spread_non_negative"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), SpreadSink, _, _>( + |_| (), + |&(), (x,), out| { + if x < 0 { + vortex_bail!("negative input {x}"); + } + out.fill(x); + Ok(()) + }, + ) + } +} + +/// Writes infallibly and lets its output sink report the error after the loop. +#[derive(Clone)] +struct DeferredNonNegative; + +impl RowFn for DeferredNonNegative { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + const FALLIBLE: bool = true; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.deferred_non_negative"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(i64,), NonNegativeSink, _, _>( + |_| (), + |&(), (value,), output| output.write(value), + ) + } +} + +/// `SpreadSink`'s three-element rows, built from `values`. +fn spread_rows(values: impl IntoIterator>) -> VortexResult { + let values = values.into_iter().collect::>(); + let rows = values.len(); + let flat = values + .iter() + .flat_map(|value| [value.unwrap_or(0); 3]) + .collect::>(); + let validity = if values.iter().all(Option::is_some) { + Validity::NonNullable + } else { + Validity::from_iter(values.iter().map(Option::is_some)) + }; + + Ok(FixedSizeListArray::try_new( + PrimitiveArray::new(flat, Validity::NonNullable).into_array(), + 3, + validity, + rows, + )? + .into_array()) +} + +/// The output dtype is the sink's, with its width, and every row holds the written slice. +#[test] +fn writes_one_row_at_a_time() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![7i64, -2, 0].into_array(); + + let result = apply(Spread, [input], &mut ctx)?; + + assert_arrays_eq!(result, spread_rows([Some(7), Some(-2), Some(0)])?, &mut ctx); + Ok(()) +} + +/// A null input row is written densely and masked away afterwards, exactly as on the value path. +#[test] +fn nulls_are_masked_after_the_sink() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::from_option_iter([Some(7i64), None, Some(4)]).into_array(); + + let result = apply(Spread, [input], &mut ctx)?; + + assert!(result.dtype().is_nullable()); + assert_arrays_eq!(result, spread_rows([Some(7), None, Some(4)])?, &mut ctx); + Ok(()) +} + +/// A sink whose closure cannot fail is dense, while one whose closure can fail is valid-only. +#[test] +fn null_handling_follows_from_declared_fallibility() { + let args = [DType::Primitive(PType::I64, Nullability::NonNullable)]; + assert_eq!(policy(&Spread, &args), RowPolicy::Dense); + assert!(!ScalarFnVTable::is_fallible(&Spread, &EmptyOptions)); + + assert_eq!( + policy(&SpreadNonNegative, &args), + RowPolicy::ValidOnly { + filtered_decode_cost: 0 + } + ); + assert!(ScalarFnVTable::is_fallible( + &SpreadNonNegative, + &EmptyOptions + )); +} + +/// An error from a writing closure aborts the batch rather than being written into the sink. +#[test] +fn a_failing_row_propagates() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, -5, 3].into_array(); + + let error = apply(SpreadNonNegative, [input], &mut ctx).unwrap_err(); + + assert!(error.to_string().contains("negative input -5"), "{error}"); + Ok(()) +} + +/// A sink may accumulate a failure while its row closure remains infallible. +#[test] +fn a_sink_can_defer_its_error_until_finish() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = buffer![1i64, -5, 3].into_array(); + + let error = apply(DeferredNonNegative, [input], &mut ctx).unwrap_err(); + + assert!(error.to_string().contains("negative output"), "{error}"); + Ok(()) +} + +/// A deferred failure behind a null triggers a valid-row retry and is then discarded. +#[test] +fn a_deferred_error_behind_a_null_is_ignored() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::new( + buffer![1i64, -5, 3], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let result = apply(DeferredNonNegative, [input], &mut ctx)?; + + assert_arrays_eq!( + result, + PrimitiveArray::from_option_iter([Some(1i64), None, Some(3)]), + &mut ctx + ); + Ok(()) +} + +/// Being fallible, `SpreadNonNegative` is filtered, so its closure never sees the value behind a +/// null row. A negative payload there must therefore not raise. +#[test] +fn a_failing_row_is_never_reached_behind_a_null() -> VortexResult<()> { + let mut ctx = array_session().create_execution_ctx(); + let input = PrimitiveArray::new( + buffer![1i64, -5, 3], + Validity::from_iter([true, false, true]), + ) + .into_array(); + + let result = apply(SpreadNonNegative, [input], &mut ctx)?; + + assert_arrays_eq!(result, spread_rows([Some(1), None, Some(3)])?, &mut ctx); + Ok(()) +} + +/// The sink names its output dtype from the input, so a wrong input dtype is rejected at plan +/// time rather than producing a mistyped column. +#[test] +fn the_sink_dtype_validates_its_input() { + let dtype = DType::Primitive(PType::F64, Nullability::NonNullable); + assert!(ScalarFnVTable::return_dtype(&Spread, &EmptyOptions, &[dtype]).is_err()); +} + +/// The width the sink declares is the width it builds, over the element dtype it read off the +/// input. +#[test] +fn the_return_dtype_is_the_sinks() -> VortexResult<()> { + let dtype = DType::Primitive(PType::I64, Nullability::NonNullable); + assert_eq!( + ScalarFnVTable::return_dtype(&Spread, &EmptyOptions, std::slice::from_ref(&dtype))?, + DType::FixedSizeList(Arc::new(dtype), 3, Nullability::NonNullable), + ); + Ok(()) +} diff --git a/vortex-array/src/scalar_fn/row/vtable.rs b/vortex-array/src/scalar_fn/row/vtable.rs new file mode 100644 index 00000000000..26c83e9c11b --- /dev/null +++ b/vortex-array/src/scalar_fn/row/vtable.rs @@ -0,0 +1,484 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Blanket scalar-function implementation and execution visitors for row functions. + +use std::marker::PhantomData; +use std::mem::needs_drop; +use std::ops::BitOrAssign; + +use vortex_error::VortexResult; +#[cfg(any(test, feature = "_test-harness"))] +use vortex_error::vortex_err; +use vortex_mask::Mask; +use vortex_session::VortexSession; + +use super::row_fn::RowFn; +use super::row_fn::RowVisitor; +use super::row_fn::private; +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::expr::Expression; +use crate::expr::union_child_validities; +use crate::scalar_fn::Arity; +use crate::scalar_fn::ChildName; +use crate::scalar_fn::ElementTuple; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::IndexedElementTuple; +#[cfg(any(test, feature = "_test-harness"))] +use crate::scalar_fn::NullStrategy; +use crate::scalar_fn::OutputElement; +use crate::scalar_fn::OutputSink; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::SinkResult; +#[cfg(any(test, feature = "_test-harness"))] +use crate::scalar_fn::VecExecutionArgs; +use crate::scalar_fn::row::execute::RowExecution; +use crate::scalar_fn::row::execute::execute_row_output_prepared; +use crate::scalar_fn::row::execute::execute_row_sink_branch; +use crate::scalar_fn::row::execute::execute_row_sink_prepared; +use crate::scalar_fn::row::execute::validate_row_output; +use crate::scalar_fn::row::execute::validate_row_sink; +use crate::scalar_fn::row::lift::Batch; +use crate::scalar_fn::row::lift::BatchPlan; +use crate::scalar_fn::row::lift::KernelArgs; +use crate::scalar_fn::row::lift::RowPolicy; +use crate::scalar_fn::row::lift::reconcile_return; + +/// Compile-time checks shared by both output capabilities. +const fn assert_input_dispatch_agrees() { + assert!( + A::ARITY == F::ARG_NAMES.len(), + "dispatch visited a tuple whose arity differs from RowFn::ARG_NAMES", + ); + // Dictionary pushdown treats an infallible function as safe to evaluate over values no code + // references, so every dispatch must fit the function-wide declaration. + assert!( + !A::DECODE_FALLIBLE || F::FALLIBLE, + "dispatch decoded fallibly without declaring RowFn::FALLIBLE", + ); +} + +/// Compile-time check for a sink-writing dispatch. +const fn assert_sink_dispatch_agrees() { + assert_input_dispatch_agrees::(); + assert!( + !R::FALLIBLE || F::FALLIBLE, + "dispatch returned an error without declaring RowFn::FALLIBLE", + ); + assert!( + !R::DEFERRED || F::FALLIBLE, + "dispatch deferred an error without declaring RowFn::FALLIBLE", + ); + assert!( + S::ERRORS_ARE_DEFERRED == R::DEFERRED, + "a deferred-error sink and row closure must be used together", + ); +} + +/// Compile-time check for an owned output with deferred failure evidence. +const fn assert_deferred_dispatch_agrees() +where + F: RowFn, + A: IndexedElementTuple, + O: OutputElement, + Failure: Copy + Default + BitOrAssign, +{ + assert_input_dispatch_agrees::(); + assert!( + F::FALLIBLE, + "dispatch deferred an error without declaring RowFn::FALLIBLE", + ); + assert!( + size_of::() <= size_of::(), + "failure evidence must be no wider than the value, or it bounds the vector width", + ); + assert!( + !needs_drop::(), + "owned deferred outputs must not require drop glue", + ); +} + +/// The plan-time visit: validate the dtypes and derive execution from the output capability and row +/// closure selected by dispatch. +struct PlanRows<'a, F> { + args: &'a [DType], + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for PlanRows<'_, F> {} + +impl RowVisitor for PlanRows<'_, F> { + type Out = BatchPlan; + + fn visit_prepared_deferred( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + _finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + + Ok(BatchPlan { + output_dtype: validate_row_output::(self.args)?, + policy: RowPolicy::for_deferred_output::(), + }) + } + + fn visit_prepared_into( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult { + const { assert_sink_dispatch_agrees::() }; + + Ok(BatchPlan { + output_dtype: validate_row_sink::(self.args)?, + policy: RowPolicy::for_sink::(), + }) + } +} + +/// The run-time visit: decode every column once and run the row loop. +struct ExecuteRows<'a, 'b, F> { + args: &'a dyn ExecutionArgs, + + /// The output dtype computed by the planning visit. + output_dtype: &'a DType, + + ctx: &'b mut ExecutionCtx, + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for ExecuteRows<'_, '_, F> {} + +impl RowVisitor for ExecuteRows<'_, '_, F> { + type Out = RowExecution; + + fn visit_prepared_deferred( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + execute_row_output_prepared::( + self.args, + self.ctx, + prepare, + apply, + finish_failure, + ) + } + + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult { + const { assert_sink_dispatch_agrees::() }; + execute_row_sink_prepared::( + self.args, + self.output_dtype, + self.ctx, + prepare, + apply, + ) + } +} + +/// The run-time visit for the branch-and-skip null strategy: compute only the conjoined-valid +/// rows over unfiltered columns. +/// +/// `Ok(None)` means the visit requires filtering, a sink cannot skip rows, or an argument has no +/// null-tolerant decode. The lifting then falls back to the filter strategy. +struct ExecuteRowsBranch<'a, 'b, F> { + args: &'a dyn ExecutionArgs, + + /// The output dtype computed by the planning visit. + output_dtype: &'a DType, + + /// The conjoined validity, materialized by the lifting and guaranteed mixed. + valid: &'a Mask, + + ctx: &'b mut ExecutionCtx, + + /// The visited function, carried only so the dispatch check can name its contract. + row_fn: PhantomData, +} + +impl private::Sealed for ExecuteRowsBranch<'_, '_, F> {} + +impl RowVisitor for ExecuteRowsBranch<'_, '_, F> { + type Out = Option; + + fn visit_prepared_deferred( + self, + _prepare: impl FnOnce(A::ConstElems<'_>) -> P, + _apply: impl Fn(&P, A::Elems<'_>) -> (O, Failure), + _finish_failure: impl FnOnce(Failure) -> VortexResult<()>, + ) -> VortexResult> + where + A: IndexedElementTuple, + O: OutputElement, + Failure: 'static + Copy + Default + BitOrAssign, + { + const { assert_deferred_dispatch_agrees::() }; + Ok(None) + } + + fn visit_prepared_into( + self, + prepare: impl FnOnce(A::ConstElems<'_>) -> P, + apply: impl Fn(&P, A::Elems<'_>, S::Row<'_>) -> R, + ) -> VortexResult> { + const { assert_sink_dispatch_agrees::() }; + execute_row_sink_branch::( + self.args, + self.output_dtype, + self.valid, + self.ctx, + prepare, + apply, + ) + } +} + +/// The kernel the lifting runs: the encoding-aware rewrite if it answers, otherwise the row loop +/// over whichever arguments the lifting hands over. +fn execute_rows( + row_fn: &F, + options: &F::Options, + args: KernelArgs<'_>, + ctx: &mut ExecutionCtx, +) -> VortexResult { + if let Some(reduced) = row_fn.reduce_encoded(options, args.arrays, ctx)? { + return Ok(RowExecution::Output(reduced)); + } + + row_fn.dispatch( + options, + args.dtypes, + ExecuteRows:: { + args: args.execution, + output_dtype: args.output_dtype, + ctx, + row_fn: PhantomData, + }, + ) +} + +/// The branch-and-skip kernel: compute only the rows set in `valid`, over the unfiltered `args`. +/// +/// `Ok(None)` sends the batch to the filter strategy instead. +fn execute_rows_branch( + row_fn: &F, + options: &F::Options, + args: KernelArgs<'_>, + valid: &Mask, + ctx: &mut ExecutionCtx, +) -> VortexResult> { + // The encoding-aware rewrite runs before the row loop exactly as in [`execute_rows`]. Here it + // sees the original (unfiltered) encodings, and its full-length result is masked by the caller + // like any other branch result. + if let Some(reduced) = row_fn.reduce_encoded(options, args.arrays, ctx)? { + return Ok(Some(RowExecution::Output(reduced))); + } + + row_fn.dispatch( + options, + args.dtypes, + ExecuteRowsBranch:: { + args: args.execution, + output_dtype: args.output_dtype, + valid, + ctx, + row_fn: PhantomData, + }, + ) +} + +/// The batch facts for `row_fn` over `args`, derived from its selected output capability. +fn lift_batch<'a, F: RowFn>( + row_fn: &F, + options: &F::Options, + args: &'a dyn ExecutionArgs, +) -> VortexResult> { + Batch::new(RowFn::id(row_fn), args, |arg_dtypes| { + let plan = row_fn.dispatch( + options, + arg_dtypes, + PlanRows:: { + args: arg_dtypes, + row_fn: PhantomData, + }, + )?; + Ok(plan) + }) +} + +/// The nullable execution policy selected by one concrete dispatch. +#[cfg(test)] +pub(super) fn row_policy( + row_fn: &F, + options: &F::Options, + args: &[DType], +) -> VortexResult { + row_fn + .dispatch( + options, + args, + PlanRows:: { + args, + row_fn: PhantomData, + }, + ) + .map(|plan| plan.policy) +} + +/// Every [`RowFn`] is a [`ScalarFnVTable`], the row loop lifted by `Batch`. +/// +/// This impl is why a [`RowFn`] cannot also implement [`ScalarFnVTable`] itself: coherence forbids +/// the second impl. Nothing in tree needs to, since everything a row function can vary lives on +/// [`RowFn`]; mirror another [`ScalarFnVTable`] method onto it when something actually does. +impl ScalarFnVTable for F { + type Options = F::Options; + + fn id(&self) -> ScalarFnId { + RowFn::id(self) + } + + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + RowFn::serialize(self, options) + } + + fn deserialize(&self, metadata: &[u8], session: &VortexSession) -> VortexResult { + RowFn::deserialize(self, metadata, session) + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(F::ARG_NAMES.len()) + } + + fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { + ChildName::from(F::ARG_NAMES[child_idx]) + } + + /// The visited output element's dtype, widened to nullable iff any input is nullable, which is + /// what makes the strictness dtype contract hold by construction. + fn return_dtype(&self, options: &Self::Options, args: &[DType]) -> VortexResult { + let plan = self.dispatch( + options, + args, + PlanRows:: { + args, + row_fn: PhantomData, + }, + )?; + + let nullability = plan.output_dtype.nullability() + | Nullability::from(args.iter().any(DType::is_nullable)); + Ok(plan.output_dtype.with_nullability(nullability)) + } + + fn execute( + &self, + options: &Self::Options, + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + // Nullary functions have no input values that could be null, so there is nothing to lift. + if args.num_inputs() == 0 { + let result_dtype = ScalarFnVTable::return_dtype(self, options, &[])?; + let values = execute_rows( + self, + options, + KernelArgs { + execution: args, + arrays: &[], + dtypes: &[], + output_dtype: &result_dtype, + }, + ctx, + )? + .into_result()?; + return reconcile_return(RowFn::id(self), &result_dtype, args.row_count(), values); + } + + lift_batch(self, options, args)?.execute( + |args, ctx| execute_rows(self, options, args, ctx), + |args, valid, ctx| execute_rows_branch(self, options, args, valid, ctx), + ctx, + ) + } + + /// Row output capabilities build an all-valid column, so a kernel cannot turn a wholly non-null + /// row into a null and the output validity is exactly the conjunction of the inputs'. Letting an + /// output capability produce nulls would invalidate this. + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + union_child_validities(expression) + } + + /// A row kernel maps a null input row to a null output row, and computes non-null outputs from + /// non-null inputs alone, which is exactly strictness. The lifting is what makes it true. + fn is_strict(&self, _options: &Self::Options) -> bool { + true + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + F::FALLIBLE + } +} + +/// Execute `row_fn` over `inputs` with a forced null strategy, bypassing the per-batch selection. +/// +/// A test and benchmark seam only, and the only way to name a strategy from outside: it is how the +/// two are compared and how their agreement is asserted. It skips the null-constant and +/// all-constant folds, so do not pass such inputs. Forcing [`NullStrategy::BranchAndSkip`] on a +/// dispatch with no branch execution is an error rather than a silent fallback to filtering. +#[cfg(any(test, feature = "_test-harness"))] +pub fn execute_row_fn_with_strategy( + row_fn: &F, + options: &F::Options, + inputs: Vec, + row_count: usize, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let args = VecExecutionArgs::new(inputs, row_count); + + lift_batch(row_fn, options, &args)? + .execute_with_strategy( + |args, ctx| execute_rows(row_fn, options, args, ctx), + |args, valid, ctx| execute_rows_branch(row_fn, options, args, valid, ctx), + strategy, + ctx, + )? + .ok_or_else(|| { + vortex_err!( + "{} has no branch-and-skip execution for these inputs", + RowFn::id(row_fn), + ) + }) +} diff --git a/vortex-array/src/scalar_fn/vtable.rs b/vortex-array/src/scalar_fn/vtable.rs index 5d3561ff039..e5b074e9cf3 100644 --- a/vortex-array/src/scalar_fn/vtable.rs +++ b/vortex-array/src/scalar_fn/vtable.rs @@ -361,7 +361,7 @@ impl ExecutionArgs for VecExecutionArgs { } } -#[derive(Clone, Debug, PartialEq, Eq, Hash)] +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] pub struct EmptyOptions; impl Display for EmptyOptions { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { diff --git a/vortex-compute/src/lane_kernels/map_into.rs b/vortex-compute/src/lane_kernels/map_into.rs index c1e1107b1b9..9ede2df69e5 100644 --- a/vortex-compute/src/lane_kernels/map_into.rs +++ b/vortex-compute/src/lane_kernels/map_into.rs @@ -157,6 +157,50 @@ pub trait IndexedSourceExt: IndexedSource + Sized { } } + /// Write each mapped value and OR-reduce independent failure evidence across the batch. + /// + /// The failure stays local to this method so the optimizer can keep it in a register. The + /// caller receives only whether the batch failed and can attribute errors on a cold retry. + /// **`Failure` must be no wider than `Output`**, or its reduction can limit vector width. + /// + /// # Panics + /// + /// Panics if `out.len() != self.len()`. + #[inline] + fn map_checked_into( + self, + out: &mut [MaybeUninit], + mut apply: Apply, + ) -> Failure + where + Failure: Copy + Default + BitOrAssign, + Apply: FnMut(Self::Item) -> (Output, Failure), + { + const { + assert!( + size_of::() <= size_of::(), + "failure evidence must be no wider than the value, or it bounds the vector width" + ) + }; + + let values = self; + let len = values.len(); + assert_eq!(out.len(), len, "out must have the same length as values"); + + let mut failed = Failure::default(); + for index in 0..len { + // SAFETY: `index < len` by the loop bound. + let value = unsafe { values.get_unchecked(index) }; + let (output, failure) = apply(value); + failed |= failure; + + // SAFETY: `index < len == out.len()`. + unsafe { out.get_unchecked_mut(index).write(output) }; + } + + failed + } + /// Apply the predicate `f(value)` lane-by-lane and bit-pack the results into /// `words`, LSB-first, 64 lanes per `u64`. /// @@ -218,58 +262,6 @@ pub trait IndexedSourceExt: IndexedSource + Sized { } } - /// Split value/failure map with **no validity awareness at all**: write every lane's value - /// unconditionally and OR-reduce its failure evidence into the return. - /// - /// The fastest checked shape, running at the speed of the unchecked [`map_into`] in exchange - /// for reporting only _that_ some lane failed and never exiting early. Re-run the now known - /// cold input through [`try_map_into`] or [`try_map_masked_into`] to attribute the failure or - /// to drop the null-lane ones. The evidence reduces inside the kernel because a captured `&mut` - /// becomes a loop-carried memory dependence that blocks vectorization. - /// - /// Anything other than [`Default`] means failure, and `bool` is the ordinary `Fail`. Wider - /// words exist for operations where deriving a `bool` costs the vectorization it guards. - /// **`Fail` must be no wider than `R`**, asserted below, or the reduction rather than the - /// operation decides how many lanes fit in a vector. - /// - /// [`map_into`]: IndexedSourceExt::map_into - /// [`try_map_into`]: IndexedSourceExt::try_map_into - /// [`try_map_masked_into`]: IndexedSourceExt::try_map_masked_into - /// - /// # Panics - /// - /// Panics if `out.len() != self.len()`. - #[inline] - fn map_checked_into(self, out: &mut [MaybeUninit], mut apply: Apply) -> Fail - where - Fail: Copy + Default + BitOrAssign, - Apply: FnMut(Self::Item) -> (R, Fail), - { - const { - assert!( - size_of::() <= size_of::(), - "failure evidence must be no wider than the value, or it bounds the vector width" - ) - }; - - let values = self; - let len = values.len(); - assert_eq!(out.len(), len, "out must have the same length as values"); - - let mut failed = Fail::default(); - for idx in 0..len { - // SAFETY: idx < len by the loop bound, and out.len() == len. - let val = unsafe { values.get_unchecked(idx) }; - - let (result, failure) = apply(val); - failed |= failure; - - // SAFETY: idx < len == out.len(). - unsafe { out.get_unchecked_mut(idx).write(result) }; - } - failed - } - /// Fallible map with **no validity awareness at all** — every `None` returned /// by the closure is treated as a failure, even at null lanes. /// @@ -600,23 +592,22 @@ mod tests { } #[test] - fn map_checked_into_writes_all_lanes_and_reduces_flag() { + fn map_checked_into_writes_all_lanes_and_reduces_failure() { let mut values: Vec = (0..130).collect(); - let mut out = vec![MaybeUninit::::uninit(); 130]; + let mut output = vec![MaybeUninit::::uninit(); 130]; let failed = values .as_slice() - .map_checked_into(&mut out, |v| (v as u32, v > u32::MAX as u64)); + .map_checked_into(&mut output, |value| (value as u32, value > u32::MAX as u64)); assert!(!failed); - assert_eq!(write_t(out), (0..130u32).collect::>()); + assert_eq!(write_t(output), (0..130u32).collect::>()); values[77] = (u32::MAX as u64) + 1; - let mut out = vec![MaybeUninit::::uninit(); 130]; + let mut output = vec![MaybeUninit::::uninit(); 130]; let failed = values .as_slice() - .map_checked_into(&mut out, |v| (v as u32, v > u32::MAX as u64)); + .map_checked_into(&mut output, |value| (value as u32, value > u32::MAX as u64)); assert!(failed); - // Failing lanes still write their (wrapped) value. - assert_eq!(write_t(out)[76], 76); + assert_eq!(write_t(output)[76], 76); } #[test] diff --git a/vortex-spatial/Cargo.toml b/vortex-spatial/Cargo.toml index 8b790da3c05..139f28181d6 100644 --- a/vortex-spatial/Cargo.toml +++ b/vortex-spatial/Cargo.toml @@ -47,11 +47,15 @@ name = "envelope" harness = false [[bench]] -name = "predicate_bbox" +name = "binary_predicates" harness = false [[bench]] -name = "binary_predicates" +name = "null_strategies" +harness = false + +[[bench]] +name = "predicate_bbox" harness = false [[bench]] diff --git a/vortex-spatial/benches/null_strategies.rs b/vortex-spatial/benches/null_strategies.rs new file mode 100644 index 00000000000..89f43f97429 --- /dev/null +++ b/vortex-spatial/benches/null_strategies.rs @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Null-strategy comparison for the geo `contains` kernel, whose per-row geometry decode is what +//! the selection threshold exists for. +//! +//! Arms: `filter` and `branch` force one strategy through the test-harness seam +//! ([`execute_row_fn_with_strategy`]); `auto` executes the full pipeline and lets the per-batch +//! selection choose, which should track the faster forced arm on both sides of the crossover +//! (branch at dense validity, filter at sparse). +//! +//! Workloads: a column of small polygons CONTAINS a constant point, and polygon column CONTAINS +//! point column with independent nulls on both, each at null densities 0/1/5/10/25/50/90 percent +//! over 65536 rows, nulls placed by a seeded splitmix hash. +//! +//! Run with `cargo bench -p vortex-spatial --bench null_strategies`. + +#![expect(clippy::unwrap_used)] + +use std::sync::LazyLock; + +use divan::Bencher; +use divan::counter::ItemsCount; +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::NullStrategy; +use vortex_array::scalar_fn::execute_row_fn_with_strategy; +use vortex_array::validity::Validity; +use vortex_session::VortexSession; +use vortex_spatial::scalar_fn::contains::SpatialContains; +use vortex_spatial::test_harness::point_column; +use vortex_spatial::test_harness::polygon_column; +use vortex_spatial::test_harness::spatial_session; + +static SESSION: LazyLock = LazyLock::new(spatial_session); + +fn main() { + LazyLock::force(&SESSION); + divan::main(); +} + +const ROWS: usize = 65536; + +/// Null densities in percent. +const DENSITIES: &[usize] = &[0, 1, 5, 10, 25, 50, 90]; + +/// Deterministic pseudo-random value in `[0, 1)` (same generator as `binary_predicates`). +fn unit(i: usize) -> f64 { + ((i.wrapping_mul(2654435761) >> 8) % 10_000) as f64 / 10_000.0 +} + +/// splitmix64, for seeded random null placement. +fn splitmix64(mut x: u64) -> u64 { + x = x.wrapping_add(0x9E3779B97F4A7C15); + x = (x ^ (x >> 30)).wrapping_mul(0xBF58476D1CE4E5B9); + x = (x ^ (x >> 27)).wrapping_mul(0x94D049BB133111EB); + x ^ (x >> 31) +} + +/// A small square (side 2) centered at `(cx, cy)`. +fn square(cx: f64, cy: f64) -> Vec> { + vec![vec![ + (cx - 1.0, cy - 1.0), + (cx + 1.0, cy - 1.0), + (cx + 1.0, cy + 1.0), + (cx - 1.0, cy + 1.0), + (cx - 1.0, cy - 1.0), + ]] +} + +/// [`ROWS`] small squares spread over roughly `[-150, 150)^2`; a handful contain the origin, and +/// each row's verdict is a direct point-in-polygon test. +fn squares() -> ArrayRef { + let rows = (0..ROWS) + .map(|i| square(300.0 * unit(i) - 150.0, 300.0 * unit(i + 1) - 150.0)) + .collect(); + polygon_column(rows).unwrap() +} + +/// [`ROWS`] points over the same region. +fn points() -> ArrayRef { + let xs = (0..ROWS).map(|i| 300.0 * unit(i + 7) - 150.0).collect(); + let ys = (0..ROWS).map(|i| 300.0 * unit(i + 8) - 150.0).collect(); + point_column(xs, ys).unwrap() +} + +/// The constant point operand, at the origin so some squares contain it. +fn constant_point(ctx: &mut ExecutionCtx) -> ArrayRef { + let scalar = point_column(vec![0.0], vec![0.0]) + .unwrap() + .execute_scalar(0, ctx) + .unwrap(); + ConstantArray::new(scalar, ROWS).into_array() +} + +/// Wrap `array` with seeded random nulls at `density` percent. Zero density stays unwrapped, as a +/// non-nullable column would. +fn with_nulls(array: ArrayRef, seed: u64, density: usize) -> ArrayRef { + if density == 0 { + return array; + } + + let valid = (0..ROWS).map(|i| (splitmix64(seed ^ i as u64) % 100) >= density as u64); + MaskedArray::try_new(array, Validity::from_iter(valid)) + .unwrap() + .into_array() +} + +/// One arm over the operand pair: `Some` forces a strategy through the harness seam, `None` runs +/// the full pipeline with the per-batch selection. +fn bench_contains(bencher: Bencher, a: ArrayRef, b: ArrayRef, strategy: Option) { + let mut ctx = SESSION.create_execution_ctx(); + + bencher + .counter(ItemsCount::new(ROWS)) + .bench_local(|| match strategy { + None => SpatialContains::try_new_array(a.clone(), b.clone()) + .unwrap() + .into_array() + .execute::(&mut ctx) + .unwrap(), + Some(strategy) => execute_row_fn_with_strategy( + &SpatialContains, + &EmptyOptions, + vec![a.clone(), b.clone()], + ROWS, + strategy, + &mut ctx, + ) + .unwrap() + .execute::(&mut ctx) + .unwrap(), + }); +} + +/// Column of polygons CONTAINS constant point, nulls on the polygon column. +mod polygons_x_constant_point { + use super::*; + + fn operands(density: usize) -> (ArrayRef, ArrayRef) { + let mut ctx = SESSION.create_execution_ctx(); + (with_nulls(squares(), 1, density), constant_point(&mut ctx)) + } + + #[divan::bench(args = DENSITIES)] + fn filter(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::Filter)); + } + + #[divan::bench(args = DENSITIES)] + fn branch(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::BranchAndSkip)); + } + + #[divan::bench(args = DENSITIES)] + fn auto(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, None); + } +} + +/// Column of polygons CONTAINS column of points, independent nulls on both, so the conjoined +/// valid fraction is roughly `(1 - d)^2`. +mod polygons_x_points { + use super::*; + + fn operands(density: usize) -> (ArrayRef, ArrayRef) { + ( + with_nulls(squares(), 1, density), + with_nulls(points(), 2, density), + ) + } + + #[divan::bench(args = DENSITIES)] + fn filter(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::Filter)); + } + + #[divan::bench(args = DENSITIES)] + fn branch(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, Some(NullStrategy::BranchAndSkip)); + } + + #[divan::bench(args = DENSITIES)] + fn auto(bencher: Bencher, density: usize) { + let (a, b) = operands(density); + bench_contains(bencher, a, b, None); + } +} diff --git a/vortex-spatial/src/extension/mod.rs b/vortex-spatial/src/extension/mod.rs index d1e2c37ebf4..e89ec15b500 100644 --- a/vortex-spatial/src/extension/mod.rs +++ b/vortex-spatial/src/extension/mod.rs @@ -178,6 +178,47 @@ pub(crate) fn geometries( } } +/// The geometry a null row decodes to under [`geometries_null_tolerant`]. Arbitrary: the caller +/// guarantees null rows are never read. +pub(crate) fn placeholder_geometry() -> Geometry { + Geometry::Point(geo_types::Point::new(0.0, 0.0)) +} + +/// Decode a native geometry column that may contain null rows, writing [`placeholder_geometry`] +/// into their slots. The caller guarantees null rows are never read. +/// +/// `Ok(None)` means this geometry type has no null-tolerant decode yet (`Point` and `Polygon` are +/// covered), and the caller falls back to the filter strategy, which never decodes a null row. A +/// column with definitely no nulls delegates to the ordinary [`geometries`] for any type. +pub(crate) fn geometries_null_tolerant( + array: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>>> { + if array.validity()?.definitely_no_nulls() { + return geometries(array, ctx).map(Some); + } + + let Some(ext) = array.dtype().as_extension_opt() else { + vortex_bail!( + "spatial: operand is not a geometry extension type, was {}", + array.dtype() + ); + }; + let storage = array + .clone() + .execute::(ctx)? + .storage_array() + .clone(); + + if ext.is::() { + point_geometries_null_tolerant(&storage, ctx).map(Some) + } else if ext.is::() { + polygon_geometries_null_tolerant(&storage, ctx).map(Some) + } else { + Ok(None) + } +} + /// Decode a constant operand scalar to one geometry, a constant of any /// supported geometry type is decoded exactly like a column. pub(crate) fn single_geometry( diff --git a/vortex-spatial/src/extension/point.rs b/vortex-spatial/src/extension/point.rs index e8f3ad3c169..8189fcba7bf 100644 --- a/vortex-spatial/src/extension/point.rs +++ b/vortex-spatial/src/extension/point.rs @@ -52,6 +52,7 @@ use super::coordinate::coordinate_from_struct; use super::coordinate::coordinate_storage_dtype; use super::geoarrow_metadata; use super::geoarrow_to_wkb; +use super::placeholder_geometry; use super::spatial_metadata_from_arrow; /// A single location: `geoarrow.point`, stored as `Struct` of non-nullable `f64`. @@ -149,6 +150,23 @@ pub(crate) fn point_geometries( .collect() } +/// Like [`point_geometries`], but a null row decodes to the placeholder geometry instead of +/// failing. The caller guarantees null rows are never read. +pub(crate) fn point_geometries_null_tolerant( + storage: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>> { + point_array(storage, ctx)? + .iter() + .map(|geometry| match geometry { + None => Ok(placeholder_geometry()), + Some(geometry) => Ok(geometry + .map_err(|e| vortex_err!("spatial: geometry access failed: {e}"))? + .to_geometry()), + }) + .collect() +} + impl ArrowExportVTable for Point { fn arrow_ext_id(&self) -> Id { *ARROW_POINT diff --git a/vortex-spatial/src/extension/polygon.rs b/vortex-spatial/src/extension/polygon.rs index dcfa8514ff3..362dfe311e9 100644 --- a/vortex-spatial/src/extension/polygon.rs +++ b/vortex-spatial/src/extension/polygon.rs @@ -52,6 +52,7 @@ use super::coordinate::coordinate_dimension; use super::coordinate::coordinate_storage_dtype; use super::geoarrow_metadata; use super::geoarrow_to_wkb; +use super::placeholder_geometry; use super::spatial_metadata_from_arrow; /// A polygon: `geoarrow.polygon`, stored as `List>>` (rings of vertices). @@ -131,6 +132,23 @@ pub(crate) fn polygon_geometries( .collect() } +/// Like [`polygon_geometries`], but a null row decodes to the placeholder geometry instead of +/// failing. The caller guarantees null rows are never read. +pub(crate) fn polygon_geometries_null_tolerant( + storage: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult>> { + polygon_array(storage, ctx)? + .iter() + .map(|geometry| match geometry { + None => Ok(placeholder_geometry()), + Some(geometry) => Ok(geometry + .map_err(|e| vortex_err!("spatial: geometry access failed: {e}"))? + .to_geometry()), + }) + .collect() +} + /// Build a geoarrow `PolygonArray` from a `Polygon`'s `List>` storage. fn polygon_array(storage: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { let polygon_type = polygon_type( diff --git a/vortex-spatial/src/scalar_fn/contains.rs b/vortex-spatial/src/scalar_fn/contains.rs index 599c0eee2be..54316678f5f 100644 --- a/vortex-spatial/src/scalar_fn/contains.rs +++ b/vortex-spatial/src/scalar_fn/contains.rs @@ -3,44 +3,30 @@ //! `ST_Contains`: OGC containment test between two native geometries. +use std::cell::OnceCell; + +use geo::BoundingRect; use geo::Contains; +use geo::PreparedGeometry; +use geo::Relate; +use geo_types::Geometry; +use geo_types::Rect; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Contains`. -fn validate_contains_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: contains requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: contains operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; +#[cfg(test)] +use crate::scalar_fn::row::probe; /// OGC `ST_Contains` between two native geometry operands, each a column or a constant /// literal: true where operand `b` lies completely inside operand `a` (boundary contact alone @@ -59,83 +45,297 @@ impl SpatialContains { } } -impl ScalarFnVTable for SpatialContains { +impl RowFn for SpatialContains { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.contains"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { Ok(EmptyOptions) } - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) + /// Containment is not symmetric, so `a` is always the container and `b` the contained. + fn dispatch( + &self, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |(a, b)| { + #[cfg(test)] + probe::record(a.is_some(), b.is_some()); + ConstOperands { + a: a.map(PreparedOperand::new), + b: b.map(PreparedOperand::new), + } + }, + |operands, (a, b), output| *output = contains_row_prepared(operands, a, b), + ) } +} + +/// Per-batch state for the contains row kernel: the prepared form of whichever operand is +/// constant for the batch. `None` marks an operand that varies by row. +struct ConstOperands { + /// Operand `a` (the container) when it is batch-constant. + a: Option, + + /// Operand `b` (the contained) when it is batch-constant. + b: Option, +} + +/// One batch-constant operand: the geometry cloned out of its decoded column (the state must not +/// borrow from the columns), plus its [`PreparedGeometry`], built on the first row whose pairing +/// routes through relate. +/// +/// The build is lazy because preparation (self-noding the topology graph plus an R*-tree over the +/// edges) costs `O(edges log edges)` and pays off only on relate-routed pairings; a batch of +/// point rows against a constant polygon never touches it, and preparing a large constant eagerly +/// would charge such a batch for nothing. +struct PreparedOperand { + /// The constant's decoded geometry, owned so [`prepared`](Self::prepared) can be `'static`. + geometry: Geometry, + + /// The constant's bounding rectangle, folded once for conservative row rejection. + bbox: Option>, + + /// The lazily built prepared form of [`geometry`](Self::geometry). + prepared: OnceCell, f64>>, +} - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("contains has exactly two children"), +impl PreparedOperand { + fn new(geometry: &Geometry) -> Self { + Self { + geometry: geometry.clone(), + bbox: geometry.bounding_rect(), + prepared: OnceCell::new(), } } - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_contains_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Bool(nullability)) + /// The prepared geometry, built on first use. + fn get(&self) -> &PreparedGeometry<'static, Geometry, f64> { + self.prepared + .get_or_init(|| PreparedGeometry::from(self.geometry.clone())) } +} - fn execute( - &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Containment is not symmetric: `a` is always the container and `b` the contained. A - // container's rect must cover the contained's rect (`Rect::contains` is the closed - // test), so a contained rect poking outside proves the row false. - execute_binary_geo_types( - &a, - &b, - |a, b| a.contains(b), - Some(|ra, rb| (!ra.contains(rb)).then_some(false)), - ctx, - ) - } +/// How geo's `a.contains(b)` computes its verdict for a pairing. +enum ContainsRoute { + /// `a.relate(b).is_contains()`. + ForwardRelate, - fn validity( - &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) + /// `b.relate(a).is_within()`, how geo phrases relate for `MultiPolygon` containers. + ReversedRelate, + + /// A direct algorithm (coordinate position, point arithmetic); nothing to prepare. + Direct, +} + +/// The route geo 0.31's `Contains` dispatch takes for `a.contains(b)`. +/// +/// The prepared substitution in [`contains_row_prepared`] **must** run relate exactly where geo +/// runs relate, with the same argument order, because geo's direct algorithms are not everywhere +/// bit-identical to a relate matrix query (they resolve degenerate and boundary cases with +/// different arithmetic). The relate rows below transcribe geo's `impl_contains_from_relate!` +/// lists per container type; everything else, notably every `Point`/`MultiPoint` contained side +/// and every `Point` container, is direct. +/// +/// **This table is coupled to the geo version.** It transcribes a dispatch that geo is free to +/// reshuffle in any release, and a wrong row is a silently wrong verdict rather than a build error. +/// The workspace therefore pins `geo = "=0.31.0"`: taking any new geo, patch releases included, is +/// a deliberate edit of that line, and the edit must re-verify this table against +/// `impl_contains_from_relate!`. +/// +/// `constant_operands_agree_with_columns` is the mechanical check, and it is **not** complete: it +/// compares the prepared route against plain `a.contains(b)` only for the container types it has +/// cases for. `routes_agree_with_geo_for_every_container` covers the rest, one representative +/// pairing per container variant, and is the one to extend when geo grows a geometry type. Both +/// stay green wherever relate and the direct algorithm agree, so neither replaces the pin. +fn contains_route(a: &Geometry, b: &Geometry) -> ContainsRoute { + use Geometry as G; + + match (a, b) { + // Line contains [Polygon, MultiLineString, MultiPolygon, GeometryCollection, Rect, + // Triangle]. + ( + G::Line(_), + G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // LineString contains [Polygon, MultiPoint, MultiLineString, MultiPolygon, + // GeometryCollection, Rect, Triangle]. + | ( + G::LineString(_), + G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // MultiLineString contains everything except Point. + | ( + G::MultiLineString(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // MultiPoint contains [Line, LineString, Polygon, MultiLineString, MultiPolygon, + // GeometryCollection, Rect, Triangle]. + | ( + G::MultiPoint(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // Polygon contains everything except Point and MultiPoint. + | ( + G::Polygon(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) + // Rect contains [Line, LineString, MultiPoint, MultiLineString, MultiPolygon, + // GeometryCollection, Triangle]; Rect contains Rect and Polygon are direct. + | ( + G::Rect(_), + G::Line(_) + | G::LineString(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Triangle(_), + ) + // Triangle and GeometryCollection contain everything except Point. + | ( + G::Triangle(_) | G::GeometryCollection(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiPoint(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) => ContainsRoute::ForwardRelate, + + // MultiPolygon contains everything except Point and MultiPoint, phrased reversed. + ( + G::MultiPolygon(_), + G::Line(_) + | G::LineString(_) + | G::Polygon(_) + | G::MultiLineString(_) + | G::MultiPolygon(_) + | G::GeometryCollection(_) + | G::Rect(_) + | G::Triangle(_), + ) => ContainsRoute::ReversedRelate, + + _ => ContainsRoute::Direct, } +} - fn is_strict(&self, _: &Self::Options) -> bool { - true +/// Computes one row of contains, substituting a prepared graph for a constant operand on the +/// pairings geo itself answers through relate. +/// +/// [`PreparedGeometry`] carries the operand's self-noded topology graph and edge R*-tree, so a +/// relate against it skips rebuilding both and reads its bounding rect from cache; geo asserts +/// the cached graph equal to a freshly built one (its `swap_arg_index` test), which is what makes +/// the substitution result-preserving. Before dispatch, a disjoint constant-side bounding rect +/// conservatively rejects the row, matching the columnar implementation's #9076 optimization. +/// All other rows delegate to the same direct or relate route as `a.contains(b)`. +fn contains_row_prepared(operands: &ConstOperands, a: &Geometry, b: &Geometry) -> bool { + let rejected = match (&operands.a, &operands.b) { + (None, None) => false, + (Some(const_a), Some(const_b)) => const_a + .bbox + .zip(const_b.bbox) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + (Some(const_a), None) => const_a + .bbox + .zip(b.bounding_rect()) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + (None, Some(const_b)) => a + .bounding_rect() + .zip(const_b.bbox) + .is_some_and(|(bbox_a, bbox_b)| !bbox_a.contains(&bbox_b)), + }; + + if rejected { + return false; } - fn is_fallible(&self, _: &Self::Options) -> bool { - false + match contains_route(a, b) { + ContainsRoute::Direct => a.contains(b), + ContainsRoute::ForwardRelate => match (&operands.a, &operands.b) { + (Some(const_a), Some(const_b)) => const_a.get().relate(const_b.get()).is_contains(), + (Some(const_a), None) => const_a.get().relate(b).is_contains(), + (None, Some(const_b)) => a.relate(const_b.get()).is_contains(), + (None, None) => a.contains(b), + }, + ContainsRoute::ReversedRelate => match (&operands.a, &operands.b) { + (Some(const_a), Some(const_b)) => const_b.get().relate(const_a.get()).is_within(), + (Some(const_a), None) => b.relate(const_a.get()).is_within(), + (None, Some(const_b)) => const_b.get().relate(a).is_within(), + (None, None) => a.contains(b), + }, } } #[cfg(test)] mod tests { + use geo::Contains; + use geo_types::Coord; use geo_types::Geometry; + use geo_types::GeometryCollection; + use geo_types::Line; use geo_types::LineString; + use geo_types::MultiLineString; + use geo_types::MultiPoint; + use geo_types::MultiPolygon; use geo_types::Point; use geo_types::Polygon; + use geo_types::Rect; + use geo_types::Triangle; use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::Canonical; @@ -144,23 +344,31 @@ mod tests { use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; + use vortex_array::arrays::MaskedArray; use vortex_array::assert_arrays_eq; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::EmptyOptions; + use vortex_array::scalar_fn::NullStrategy; use vortex_array::scalar_fn::ScalarFnVTable; + use vortex_array::scalar_fn::execute_row_fn_with_strategy; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; use vortex_error::VortexResult; use vortex_error::vortex_err; use wkb::writer::WriteOptions; + use super::ConstOperands; + use super::PreparedOperand; use super::SpatialContains; + use super::contains_row_prepared; + use crate::scalar_fn::row::probe::assert_prepared_agrees_with_columns; use crate::test_harness::linestring_column; use crate::test_harness::nullable_point_column; use crate::test_harness::point_column; + use crate::test_harness::polygon_column; /// A rectangle polygon with corners `(x0, y0)` and `(x1, y1)`, no holes. fn rect_polygon(x0: f64, y0: f64, x1: f64, y1: f64) -> Polygon { @@ -244,6 +452,20 @@ mod tests { assert_contains(container, points, [true, false, false]) } + /// Constant container vs a linestring column: a row whose bounding rect pokes outside the + /// container's is not contained, while one wholly inside is. Carried over from the columnar + /// bounding-rect rejection in #9076, since it constrains the verdict rather than the mechanism. + #[test] + fn constant_container_vs_row_rect_poking_outside() -> VortexResult<()> { + let container = geometry_constant(&Geometry::Polygon(rect_polygon(0.0, 0.0, 4.0, 4.0)), 3)?; + let lines = linestring_column(vec![ + vec![(1.0, 1.0), (3.0, 3.0)], + vec![(1.0, 1.0), (9.0, 1.0)], + vec![(5.0, 5.0), (9.0, 9.0)], + ])?; + assert_contains(container, lines, [true, false, false]) + } + /// Polygon column vs constant point: only the polygon around the point contains it. #[test] fn polygon_column_vs_constant_point() -> VortexResult<()> { @@ -264,20 +486,6 @@ mod tests { assert_contains(away, point, [false; 2]) } - /// Constant container vs a linestring column: a row whose bounding rect pokes outside the - /// container's rect is proven false by the rect pre-check alone; a fully inside row still - /// needs (and passes) the exact test. - #[test] - fn constant_container_vs_row_rect_poking_outside() -> VortexResult<()> { - let container = geometry_constant(&Geometry::Polygon(rect_polygon(0.0, 0.0, 4.0, 4.0)), 3)?; - let lines = linestring_column(vec![ - vec![(1.0, 1.0), (3.0, 3.0)], - vec![(1.0, 1.0), (9.0, 1.0)], - vec![(5.0, 5.0), (9.0, 9.0)], - ])?; - assert_contains(container, lines, [true, false, false]) - } - /// Column vs column pairs rows: each polygon row is tested against the point row at the /// same position. #[test] @@ -408,6 +616,117 @@ mod tests { Ok(()) } + /// A nullable polygon column: unit squares at `centers`, the rows where `nulls` is true + /// masked out, spelled as `Masked` over non-nullable storage. + fn nullable_squares(centers: &[(f64, f64)], nulls: &[bool]) -> VortexResult { + let squares = centers + .iter() + .map(|&(x, y)| { + vec![vec![ + (x - 1.0, y - 1.0), + (x + 1.0, y - 1.0), + (x + 1.0, y + 1.0), + (x - 1.0, y + 1.0), + (x - 1.0, y - 1.0), + ]] + }) + .collect(); + let polygons = polygon_column(squares)?; + + Ok( + MaskedArray::try_new(polygons, Validity::from_iter(nulls.iter().map(|n| !n)))? + .into_array(), + ) + } + + /// Executes `SpatialContains(a, b)` with a forced null strategy, canonicalized. + fn contains_forced( + a: &ArrayRef, + b: &ArrayRef, + strategy: NullStrategy, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + Ok(execute_row_fn_with_strategy( + &SpatialContains, + &EmptyOptions, + vec![a.clone(), b.clone()], + a.len(), + strategy, + ctx, + )? + .execute::(ctx)? + .into_array()) + } + + /// The branch-and-skip and filter strategies, plus the automatic per-batch selection, must + /// return identical arrays for nullable geometry operands: `Masked` polygons against nullable + /// points, with independent nulls conjoined. + #[test] + fn branch_matches_filter_for_nullable_geometries() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + let centers = [(0.0, 0.0), (5.0, 5.0), (0.5, -0.2), (9.0, 9.0), (0.0, 1.0)]; + let nulls = [false, true, false, false, true]; + let polygons = nullable_squares(¢ers, &nulls)?; + let points = nullable_point_column(vec![ + Some((0.0, 0.0)), + Some((5.0, 5.0)), + None, + Some((0.0, 0.0)), + Some((0.0, 1.0)), + ])?; + + let filtered = contains_forced(&polygons, &points, NullStrategy::Filter, &mut ctx)?; + let branched = contains_forced(&polygons, &points, NullStrategy::BranchAndSkip, &mut ctx)?; + let auto = SpatialContains::try_new_array(polygons, points)? + .into_array() + .execute::(&mut ctx)? + .into_array(); + + assert_arrays_eq!(branched, filtered, &mut ctx); + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) + } + + /// Geometry types without a null-tolerant decode refuse the branch strategy: forcing it is an + /// error, and the automatic selection (which prefers branch at this density) silently falls + /// back to filtering with the correct result. + #[test] + fn unsupported_geometry_falls_back_to_filter() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + // Four rows with one null: 75% surviving, so the selection prefers branch. + let lines = MaskedArray::try_new( + linestring_column(vec![ + vec![(0.0, 0.0), (4.0, 4.0)], + vec![(0.0, 0.0), (1.0, 1.0)], + vec![(2.0, 2.0), (3.0, 3.0)], + vec![(0.0, 4.0), (4.0, 0.0)], + ])?, + Validity::from_iter([true, false, true, true]), + )? + .into_array(); + let point = geometry_constant(&Geometry::Point(Point::new(2.0, 2.0)), 4)?; + + let error = contains_forced(&lines, &point, NullStrategy::BranchAndSkip, &mut ctx) + .expect_err("a linestring column with nulls has no branch decode"); + assert!( + error.to_string().contains("branch-and-skip"), + "unexpected error: {error}" + ); + + let filtered = contains_forced(&lines, &point, NullStrategy::Filter, &mut ctx)?; + let auto = SpatialContains::try_new_array(lines, point)? + .into_array() + .execute::(&mut ctx)? + .into_array(); + + assert_arrays_eq!(auto, filtered, &mut ctx); + Ok(()) + } + /// A non-geometry operand dtype is rejected up front, before execution. #[test] fn non_geometry_operand_is_rejected() -> VortexResult<()> { @@ -417,4 +736,166 @@ mod tests { assert!(result.is_err()); Ok(()) } + + // The prepared-vs-expanded agreement grid: every constant arrangement of a pairing must + // return exactly what the fully expanded columns return. + + /// A point geometry. + fn point(x: f64, y: f64) -> Geometry { + Geometry::Point(Point::new(x, y)) + } + + /// A linestring geometry through `coords`. + fn line(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::LineString(LineString::from(coords)) + } + + /// A multipoint geometry over `coords`. + fn multipoint(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::MultiPoint(MultiPoint::from(coords)) + } + + /// A two-point line segment geometry, the `Line` container variant. + fn line_geometry(start: (f64, f64), end: (f64, f64)) -> Geometry { + Geometry::Line(Line::new( + Coord { + x: start.0, + y: start.1, + }, + Coord { x: end.0, y: end.1 }, + )) + } + + /// A multilinestring geometry over one linestring per entry of `parts`. + fn multilinestring(parts: Vec>) -> Geometry { + Geometry::MultiLineString(MultiLineString::new( + parts.into_iter().map(LineString::from).collect(), + )) + } + + /// A geometry collection wrapping `parts`. + fn collection(parts: Vec) -> Geometry { + Geometry::GeometryCollection(GeometryCollection::from(parts)) + } + + /// An axis-aligned rectangle geometry, the `Rect` container variant. + fn rect_geometry(x0: f64, y0: f64, x1: f64, y1: f64) -> Geometry { + Geometry::Rect(Rect::new(Coord { x: x0, y: y0 }, Coord { x: x1, y: y1 })) + } + + /// A triangle geometry large enough to contain the small test polygons. + fn triangle_geometry() -> Geometry { + Geometry::Triangle(Triangle::new( + Coord { x: 0.0, y: 0.0 }, + Coord { x: 8.0, y: 0.0 }, + Coord { x: 0.0, y: 8.0 }, + )) + } + + /// A two-part multipolygon: `4x4` squares at the origin and at `(10, 10)`. + fn two_part_multipolygon() -> Geometry { + Geometry::MultiPolygon(MultiPolygon::new(vec![ + rect_polygon(0.0, 0.0, 4.0, 4.0), + rect_polygon(10.0, 10.0, 14.0, 14.0), + ])) + } + + /// Every container variant `contains_route` distinguishes, checked against plain + /// `a.contains(b)` in all four constant arrangements. + /// + /// Every case is a containment geo answers `true`, which the test asserts: a pairing that is + /// false regardless of route (a lower-dimensional container, say) also agrees regardless of + /// route, and pins nothing. A true case fails when the prepared substitution diverges from + /// geo — a table row whose relate phrasing disagrees with geo's dispatch on this input, or a + /// bounding-rect prescreen that wrongly rejects a contained row. It is **not** a version + /// tripwire: a geo release that reshuffles its dispatch stays green wherever relate and the + /// direct algorithm agree, which is why the workspace pins `geo` exactly. + /// + /// This is the table's own regression, and the one to extend when geo grows a geometry type: + /// `constant_operands_agree_with_columns` below goes through real arrays and so is the better + /// end-to-end check, but it only covers the container types it has cases for, and WKB decoding + /// limits which types those can be. The MultiPoint and Line containers route relate only for + /// contained types a MultiPoint or Line can rarely contain, so their true cases lean on + /// `GeometryCollection` membership and collinear `MultiLineString` parts respectively. + #[rstest] + #[case::point(point(1.0, 1.0), point(1.0, 1.0))] + #[case::line(line_geometry((0.0, 0.0), (4.0, 4.0)), point(2.0, 2.0))] + #[case::line_x_multilinestring(line_geometry((0.0, 0.0), (4.0, 4.0)), multilinestring(vec![vec![(1.0, 1.0), (2.0, 2.0)]]))] + #[case::linestring(line(vec![(0.0, 0.0), (4.0, 4.0)]), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::multipoint(multipoint(vec![(0.0, 0.0), (2.0, 2.0), (4.0, 4.0)]), collection(vec![point(2.0, 2.0)]))] + #[case::multilinestring(multilinestring(vec![vec![(0.0, 0.0), (4.0, 4.0)]]), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipolygon(two_part_multipolygon(), rect_polygon(1.0, 1.0, 3.0, 3.0).into())] + #[case::geometrycollection(collection(vec![rect_polygon(0.0, 0.0, 8.0, 8.0).into()]), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::rect(rect_geometry(0.0, 0.0, 8.0, 8.0), line(vec![(2.0, 2.0), (4.0, 4.0)]))] + #[case::triangle(triangle_geometry(), rect_polygon(1.0, 1.0, 2.0, 2.0).into())] + fn routes_agree_with_geo_for_every_container(#[case] a: Geometry, #[case] b: Geometry) { + let expected = a.contains(&b); + assert!( + expected, + "route cases must be containments geo answers true, or every route agrees vacuously", + ); + + let arrangements = [ + (None, None), + (Some(PreparedOperand::new(&a)), None), + (None, Some(PreparedOperand::new(&b))), + ( + Some(PreparedOperand::new(&a)), + Some(PreparedOperand::new(&b)), + ), + ]; + + for (index, (const_a, const_b)) in arrangements.into_iter().enumerate() { + let operands = ConstOperands { + a: const_a, + b: const_b, + }; + assert_eq!( + contains_row_prepared(&operands, &a, &b), + expected, + "arrangement {index} disagrees with geo's own contains", + ); + } + } + + /// Constant arrangements agree with expanded columns across the routes the prepared kernel + /// distinguishes: forward relate (polygon, linestring and multipoint containers), reversed + /// relate (multipolygon containers), and the direct pairings (a point on either side, + /// multipoint over multipoint, polygon over multipoint), including boundary contact, + /// crossing, disjoint and empty cases. + #[rstest] + #[case::polygon_nested_polygon(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::polygon_touching_from_inside(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(0.0, 2.0, 2.0, 4.0).into())] + #[case::polygon_overlapping_polygon(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(2.0, 2.0, 6.0, 6.0).into())] + #[case::polygon_disjoint_polygon(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::polygon_x_point_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(2.0, 2.0))] + #[case::polygon_x_point_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(0.0, 2.0))] + #[case::polygon_x_point_outside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(20.0, 20.0))] + #[case::polygon_x_nan_point(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), point(f64::NAN, 2.0))] + #[case::polygon_x_linestring_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon_x_linestring_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(0.0, 1.0), (0.0, 3.0)]))] + #[case::polygon_x_linestring_crossing(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![(-2.0, 2.0), (2.0, 2.0)]))] + #[case::polygon_x_empty_linestring(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), line(vec![]))] + #[case::polygon_x_multipoint_inside(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::polygon_x_multipoint_on_boundary(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), multipoint(vec![(0.0, 1.0), (0.0, 3.0)]))] + #[case::linestring_x_multipoint_on_line(line(vec![(0.0, 0.0), (4.0, 4.0)]), multipoint(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipoint_x_multipoint_subset(multipoint(vec![(0.0, 0.0), (2.0, 2.0), (4.0, 4.0)]), multipoint(vec![(2.0, 2.0)]))] + #[case::multipoint_x_linestring_between_points(multipoint(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(1.0, 1.0), (2.0, 2.0)]))] + #[case::multipolygon_x_polygon_in_one_part(two_part_multipolygon(), rect_polygon(1.0, 1.0, 3.0, 3.0).into())] + #[case::multipolygon_x_polygon_straddling(two_part_multipolygon(), rect_polygon(3.0, 3.0, 11.0, 11.0).into())] + #[case::multipolygon_x_polygon_disjoint(two_part_multipolygon(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::multipolygon_x_point_inside(two_part_multipolygon(), point(11.0, 11.0))] + #[case::point_x_point_equal(point(1.0, 1.0), point(1.0, 1.0))] + #[case::point_x_polygon(point(2.0, 2.0), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + fn constant_operands_agree_with_columns( + #[case] a: Geometry, + #[case] b: Geometry, + ) -> VortexResult<()> { + assert_prepared_agrees_with_columns( + SpatialContains::try_new_array, + geometry_constant(&a, 3)?, + geometry_constant(&b, 3)?, + ) + } } diff --git a/vortex-spatial/src/scalar_fn/distance.rs b/vortex-spatial/src/scalar_fn/distance.rs index dfd3d09ed23..e8de808bd32 100644 --- a/vortex-spatial/src/scalar_fn/distance.rs +++ b/vortex-spatial/src/scalar_fn/distance.rs @@ -6,43 +6,19 @@ use geo::Distance; use geo::Euclidean; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Distance`. -fn validate_distance_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: distance requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: distance operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; /// Planar (Euclidean) `ST_Distance` (no geodesic correction) between two native geometry /// operands, each a column or a constant literal. @@ -60,66 +36,45 @@ impl SpatialDistance { } } -impl ScalarFnVTable for SpatialDistance { +impl RowFn for SpatialDistance { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.distance"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { - Ok(EmptyOptions) - } - - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) - } - - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("distance has exactly two children"), - } - } - - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_distance_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Primitive(PType::F64, nullability)) - } - - fn execute( + fn deserialize( &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Distance is a value, not a verdict: no bounding-rect test can decide it. - execute_binary_geo_types(&a, &b, |x, y| Euclidean.distance(x, y), None, ctx) + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn validity( + /// Deliberately uses unit preparation: a batch-constant operand offers nothing + /// sound to hoist. geo computes linestring and polygon distances through its private + /// `nearest_neighbour_distance`, which builds the R*-trees for *both* sides inside each call, + /// and the point pairings are single expressions; reusing a tree across rows would mean + /// reimplementing geo's internals. A batch where both operands are constant already folds to + /// a single-row execution before the row loop. + fn dispatch( &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) - } - - fn is_strict(&self, _: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _: &Self::Options) -> bool { - false + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |_| (), + |&(), (a, b), output| *output = Euclidean.distance(a, b), + ) } } @@ -196,8 +151,9 @@ mod tests { Ok(()) } - /// Distance passes no bounding-rect rejection: a point far outside a constant polygon's - /// bounding rect still gets its true distance, alongside an inside point at distance zero. + /// Distance is a value rather than a verdict, so no bounding-rect rejection may fire for it: a + /// point far outside a constant polygon's rect still gets its true distance. Carried over from + /// #9076, which added the rejection to the predicates but deliberately not to this function. #[test] fn distance_to_constant_polygon_is_exact() -> VortexResult<()> { let session = vortex_array::array_session(); diff --git a/vortex-spatial/src/scalar_fn/execute.rs b/vortex-spatial/src/scalar_fn/execute.rs index ca5b4018249..836577ec26e 100644 --- a/vortex-spatial/src/scalar_fn/execute.rs +++ b/vortex-spatial/src/scalar_fn/execute.rs @@ -1,24 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Shared execution for native geometry scalar functions. -//! -//! [`dispatch_unary`] and the binary dispatcher handle constant/column operands and strict null -//! propagation without prescribing how a kernel represents geometries or builds its output. -//! Native columnar kernels such as `ST_Envelope` use the unary dispatcher directly. -//! -//! [`execute_binary_geo_types`] adapts row-oriented algorithms from the `geo` ecosystem. It decodes -//! valid inputs into `geo_types::Geometry`; the final output is still a Vortex [`ArrayRef`], such -//! as an `f64` or boolean array. +//! Shared unary execution for native geometry scalar functions. -mod binary; -mod geo_types; mod unary; -pub(crate) use binary::execute_binary_geo_types; pub(crate) use unary::dispatch_unary; use vortex_array::ArrayRef; -use vortex_array::dtype::Nullability; use vortex_array::scalar::Scalar; use vortex_mask::Mask; @@ -31,9 +19,6 @@ pub(crate) enum Operand { } /// Shared batch state presented to a null-propagating geometry kernel with `N` operands. -/// -/// Binary kernels use the default materialized [`Mask`]. Unary columnar kernels can instead -/// retain a lazy [`vortex_array::validity::Validity`] until they need row-wise access. pub(crate) struct Execution { /// Constant/column shape of each operand. pub(crate) operands: [Operand; N], @@ -41,6 +26,4 @@ pub(crate) struct Execution { pub(crate) valid: V, /// Number of output rows. pub(crate) len: usize, - /// Output nullability from the scalar function's return dtype. - pub(crate) nullability: Nullability, } diff --git a/vortex-spatial/src/scalar_fn/execute/binary.rs b/vortex-spatial/src/scalar_fn/execute/binary.rs deleted file mode 100644 index f2c03bd1beb..00000000000 --- a/vortex-spatial/src/scalar_fn/execute/binary.rs +++ /dev/null @@ -1,334 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Binary pairwise dispatch, plus an adapter for row-oriented `geo_types` kernels. - -use geo::BoundingRect; -use geo_types::Geometry; -use geo_types::Rect; -use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::Constant; -use vortex_array::arrays::ConstantArray; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::scalar::Scalar; -use vortex_error::VortexResult; -use vortex_mask::Mask; - -use super::Execution; -use super::Operand; -use super::geo_types::GeoTypesOutput; -use super::geo_types::eval_column; -use super::geo_types::eval_column_pair; -use crate::extension::single_geometry; - -/// Dispatch a binary strict geometry kernel over constants and columns. -/// -/// A null constant or an empty combined validity mask short-circuits to an all-null constant -/// output. Otherwise, `kernel` receives both operand shapes and the mask of rows where both are -/// valid. Two columns are always paired by row index. The kernel remains responsible for physical -/// input interpretation and Vortex output construction. -pub(crate) fn dispatch_binary( - left: &ArrayRef, - right: &ArrayRef, - output_dtype: DType, - kernel: K, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - K: FnOnce(Execution<2>, &mut ExecutionCtx) -> VortexResult, -{ - let len = left.len(); - for operand in [left, right] { - if operand - .as_opt::() - .is_some_and(|constant| constant.scalar().is_null()) - { - return Ok(ConstantArray::new(Scalar::null(output_dtype), len).into_array()); - } - } - - let (left, right, valid) = match (left.as_opt::(), right.as_opt::()) { - (Some(left), Some(right)) => ( - Operand::Constant(left.scalar().clone()), - Operand::Constant(right.scalar().clone()), - Mask::new_true(len), - ), - (Some(left), None) => ( - Operand::Constant(left.scalar().clone()), - Operand::Column(right.clone()), - right.validity()?.execute_mask(len, ctx)?, - ), - (None, Some(right)) => ( - Operand::Column(left.clone()), - Operand::Constant(right.scalar().clone()), - left.validity()?.execute_mask(len, ctx)?, - ), - (None, None) => { - let left_valid = left.validity()?.execute_mask(len, ctx)?; - let right_valid = right.validity()?.execute_mask(len, ctx)?; - ( - Operand::Column(left.clone()), - Operand::Column(right.clone()), - &left_valid & &right_valid, - ) - } - }; - - if len != 0 && valid.all_false() { - return Ok(ConstantArray::new(Scalar::null(output_dtype), len).into_array()); - } - kernel( - Execution { - operands: [left, right], - valid, - len, - nullability: output_dtype.nullability(), - }, - ctx, - ) -} - -/// A bounding-rectangle pre-check for [`execute_binary_geo_types`]'s one-constant paths. -/// -/// Called per row with rectangles in operand order, it returns `Some(result)` when they prove the -/// result and `None` when the exact kernel must run. -pub(crate) type BboxPrecheck = fn(&Rect, &Rect) -> Option; - -/// Run a binary row-oriented kernel whose inputs are decoded to `geo_types::Geometry`. -/// -/// The `geo_types` name describes the values passed to `compute`, not the output. `T` is converted -/// into a Vortex array before this function returns. Nulls propagate from either operand. With -/// exactly one constant operand, `bbox_precheck` may prove a result from the fixed constant -/// bounding rectangle and the current row's rectangle before the exact kernel runs. -pub(crate) fn execute_binary_geo_types( - left: &ArrayRef, - right: &ArrayRef, - compute: F, - bbox_precheck: Option>, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry, &Geometry) -> T + Copy, -{ - let nullability = Nullability::from(left.dtype().is_nullable() || right.dtype().is_nullable()); - dispatch_binary( - left, - right, - T::dtype(nullability), - |execution, ctx| match execution.operands { - [Operand::Constant(left), Operand::Constant(right)] => { - let left = single_geometry(&left, ctx)?; - let right = single_geometry(&right, ctx)?; - Ok(ConstantArray::new( - compute(&left, &right).into_scalar(execution.nullability), - execution.len, - ) - .into_array()) - } - [Operand::Constant(left), Operand::Column(right)] => { - let left = single_geometry(&left, ctx)?; - let prescreen = bbox_precheck.zip(left.bounding_rect()); - eval_column( - &right, - &execution.valid, - |right| { - prescreen - .and_then(|(precheck, fixed)| precheck(&fixed, &right.bounding_rect()?)) - .unwrap_or_else(|| compute(&left, right)) - }, - execution.nullability, - ctx, - ) - } - [Operand::Column(left), Operand::Constant(right)] => { - let right = single_geometry(&right, ctx)?; - let prescreen = bbox_precheck.zip(right.bounding_rect()); - eval_column( - &left, - &execution.valid, - |left| { - prescreen - .and_then(|(precheck, fixed)| precheck(&left.bounding_rect()?, &fixed)) - .unwrap_or_else(|| compute(left, &right)) - }, - execution.nullability, - ctx, - ) - } - [Operand::Column(left), Operand::Column(right)] => eval_column_pair( - &left, - &right, - &execution.valid, - compute, - execution.nullability, - ctx, - ), - }, - ctx, - ) -} - -#[cfg(test)] -mod tests { - use std::cell::Cell; - - use geo::Contains; - use geo::Intersects; - use geo_types::Geometry; - use vortex_array::ArrayRef; - use vortex_array::ExecutionCtx; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::BoolArray; - use vortex_array::arrays::ConstantArray; - use vortex_array::assert_arrays_eq; - use vortex_array::validity::Validity; - use vortex_buffer::BitBuffer; - use vortex_error::VortexResult; - - use super::BboxPrecheck; - use super::execute_binary_geo_types; - use crate::test_harness::linestring_column; - use crate::test_harness::nullable_point_column; - use crate::test_harness::point_column; - use crate::test_harness::polygon_column; - - const DISJOINT_PRECHECK: BboxPrecheck = - |left, right| (!left.intersects(right)).then_some(false); - - fn triangle_constant(len: usize, ctx: &mut ExecutionCtx) -> VortexResult { - let ring = vec![(0.0, 0.0), (10.0, 0.0), (0.0, 10.0), (0.0, 0.0)]; - let scalar = polygon_column(vec![vec![ring]])?.execute_scalar(0, ctx)?; - Ok(ConstantArray::new(scalar, len).into_array()) - } - - fn counting_intersects( - counter: &Cell, - ) -> impl Fn(&Geometry, &Geometry) -> bool + Copy { - move |left, right| { - counter.set(counter.get() + 1); - left.intersects(right) - } - } - - #[test] - fn bbox_precheck_skips_exact_test() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(3, &mut ctx)?; - let probes = point_column(vec![50.0, 8.0, 2.0], vec![50.0, 8.0, 2.0])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &triangle, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false, true]), &mut ctx); - assert_eq!(exact_runs.get(), 2); - Ok(()) - } - - #[test] - fn bbox_precheck_leaves_nulls_alone() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(3, &mut ctx)?; - let probes = nullable_point_column(vec![Some((50.0, 50.0)), None, Some((2.0, 2.0))])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &triangle, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - let expected = BoolArray::new( - BitBuffer::from_iter([false, false, true]), - Validity::from_iter([true, false, true]), - ) - .into_array(); - - assert_arrays_eq!(result, expected, &mut ctx); - assert_eq!(exact_runs.get(), 1); - Ok(()) - } - - #[test] - fn bbox_precheck_sees_rects_in_operand_order() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let probes = point_column(vec![2.0, 50.0], vec![2.0, 50.0])?; - let triangle = triangle_constant(2, &mut ctx)?; - let exact_runs = Cell::new(0); - let counted = |left: &Geometry, right: &Geometry| { - exact_runs.set(exact_runs.get() + 1); - left.contains(right) - }; - - let result = execute_binary_geo_types( - &probes, - &triangle, - counted, - Some(|left, right| (!left.contains(right)).then_some(false)), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false]), &mut ctx); - assert_eq!(exact_runs.get(), 0); - Ok(()) - } - - #[test] - fn empty_constant_falls_through_to_exact() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let scalar = linestring_column(vec![vec![]])?.execute_scalar(0, &mut ctx)?; - let empty = ConstantArray::new(scalar, 2).into_array(); - let probes = point_column(vec![2.0, 50.0], vec![2.0, 50.0])?; - let exact_runs = Cell::new(0); - - let result = execute_binary_geo_types( - &empty, - &probes, - counting_intersects(&exact_runs), - Some(DISJOINT_PRECHECK), - &mut ctx, - )?; - - assert_arrays_eq!(result, BoolArray::from_iter([false, false]), &mut ctx); - assert_eq!(exact_runs.get(), 2); - Ok(()) - } - - #[test] - fn bbox_precheck_matches_exact_results() -> VortexResult<()> { - let session = vortex_array::array_session(); - let mut ctx = session.create_execution_ctx(); - let triangle = triangle_constant(6, &mut ctx)?; - let probes = nullable_point_column(vec![ - Some((50.0, 50.0)), - Some((8.0, 8.0)), - Some((2.0, 2.0)), - None, - Some((0.0, 0.0)), - Some((10.0, 0.0)), - ])?; - let exact = |left: &Geometry, right: &Geometry| left.intersects(right); - - let with_precheck = - execute_binary_geo_types(&triangle, &probes, exact, Some(DISJOINT_PRECHECK), &mut ctx)?; - let exact_only = execute_binary_geo_types(&triangle, &probes, exact, None, &mut ctx)?; - - assert_arrays_eq!(with_precheck, exact_only, &mut ctx); - Ok(()) - } -} diff --git a/vortex-spatial/src/scalar_fn/execute/geo_types.rs b/vortex-spatial/src/scalar_fn/execute/geo_types.rs deleted file mode 100644 index 038aca46502..00000000000 --- a/vortex-spatial/src/scalar_fn/execute/geo_types.rs +++ /dev/null @@ -1,144 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Shared input decoding and Vortex output construction for `geo_types` kernels. -//! -//! `geo_types` is the row representation consumed by the kernel. These helpers always construct -//! and return Vortex arrays; they do not expose `geo_types` values as scalar-function outputs. - -use geo_types::Geometry; -use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::BoolArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::scalar::Scalar; -use vortex_array::validity::Validity; -use vortex_buffer::BitBuffer; -use vortex_error::VortexResult; -use vortex_mask::AllOr; -use vortex_mask::Mask; - -use crate::extension::geometries; - -/// A primitive result produced after kernel inputs are decoded to `geo_types`. -pub(crate) trait GeoTypesOutput: Copy { - /// The Vortex dtype used to represent this output. - fn dtype(nullability: Nullability) -> DType; - - /// Convert one computed value into a Vortex scalar for constant output. - fn into_scalar(self, nullability: Nullability) -> Scalar; - - /// Scatter values computed for valid rows into a full-length output array. - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef; -} - -impl GeoTypesOutput for f64 { - fn dtype(nullability: Nullability) -> DType { - DType::Primitive(PType::F64, nullability) - } - - fn into_scalar(self, nullability: Nullability) -> Scalar { - Scalar::primitive(self, nullability) - } - - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef { - let validity = Validity::from_mask(valid.clone(), nullability); - match valid.indices() { - AllOr::All => PrimitiveArray::new(values, validity).into_array(), - AllOr::None => PrimitiveArray::new(vec![0.0f64; len], validity).into_array(), - AllOr::Some(rows) => { - let mut data = vec![0.0f64; len]; - for (&row, value) in rows.iter().zip(values) { - data[row] = value; - } - PrimitiveArray::new(data, validity).into_array() - } - } - } -} - -impl GeoTypesOutput for bool { - fn dtype(nullability: Nullability) -> DType { - DType::Bool(nullability) - } - - fn into_scalar(self, nullability: Nullability) -> Scalar { - Scalar::bool(self, nullability) - } - - fn build_array( - len: usize, - valid: &Mask, - values: Vec, - nullability: Nullability, - ) -> ArrayRef { - let validity = Validity::from_mask(valid.clone(), nullability); - match valid.indices() { - AllOr::All => BoolArray::new(BitBuffer::from_iter(values), validity).into_array(), - AllOr::None => BoolArray::new(BitBuffer::new_unset(len), validity).into_array(), - AllOr::Some(rows) => { - let mut data = vec![false; len]; - for (&row, value) in rows.iter().zip(values) { - data[row] = value; - } - BoolArray::new(BitBuffer::from_iter(data), validity).into_array() - } - } - } -} - -/// Evaluate a decoded kernel over each valid row of one geometry column. -pub(super) fn eval_column( - column: &ArrayRef, - valid: &Mask, - compute: F, - nullability: Nullability, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry) -> T, -{ - let len = column.len(); - let decoded = geometries(&column.filter(valid.clone())?, ctx)?; - let values = decoded.iter().map(compute).collect(); - Ok(T::build_array(len, valid, values, nullability)) -} - -/// Evaluate a decoded kernel over rows where both geometry columns are valid. -pub(super) fn eval_column_pair( - left: &ArrayRef, - right: &ArrayRef, - valid: &Mask, - compute: F, - nullability: Nullability, - ctx: &mut ExecutionCtx, -) -> VortexResult -where - T: GeoTypesOutput, - F: Fn(&Geometry, &Geometry) -> T, -{ - let len = left.len(); - let left = geometries(&left.filter(valid.clone())?, ctx)?; - let right = geometries(&right.filter(valid.clone())?, ctx)?; - let values = left - .iter() - .zip(&right) - .map(|(left, right)| compute(left, right)) - .collect(); - Ok(T::build_array(len, valid, values, nullability)) -} diff --git a/vortex-spatial/src/scalar_fn/execute/unary.rs b/vortex-spatial/src/scalar_fn/execute/unary.rs index bdbbd0b33ac..478c62eef50 100644 --- a/vortex-spatial/src/scalar_fn/execute/unary.rs +++ b/vortex-spatial/src/scalar_fn/execute/unary.rs @@ -40,7 +40,6 @@ where operands: [Operand::Constant(constant.scalar().clone())], valid: Validity::AllValid, len, - nullability: output_dtype.nullability(), }, ctx, ); @@ -55,7 +54,6 @@ where operands: [Operand::Column(array.clone())], valid, len, - nullability: output_dtype.nullability(), }, ctx, ) diff --git a/vortex-spatial/src/scalar_fn/intersects.rs b/vortex-spatial/src/scalar_fn/intersects.rs index bdabd2b9967..3694303e954 100644 --- a/vortex-spatial/src/scalar_fn/intersects.rs +++ b/vortex-spatial/src/scalar_fn/intersects.rs @@ -3,44 +3,26 @@ //! `ST_Intersects`: OGC intersection test between two native geometries. +use geo::BoundingRect; use geo::Intersects; +use geo_types::Geometry; +use geo_types::Rect; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::arrays::ScalarFnArray; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; use vortex_session::VortexSession; use vortex_session::registry::CachedId; -use crate::extension::is_native_geometry; -use crate::scalar_fn::execute::execute_binary_geo_types; - -/// Validate the two native geometry operands accepted by `ST_Intersects`. -fn validate_intersects_operands(dtypes: &[DType]) -> VortexResult<()> { - vortex_ensure!( - dtypes.len() == 2, - "spatial: intersects requires exactly two geometry operands, got {}", - dtypes.len() - ); - for dtype in dtypes { - vortex_ensure!( - is_native_geometry(dtype), - "spatial: intersects operand {dtype} is not a native geometry type" - ); - } - Ok(()) -} +use crate::scalar_fn::row::GeometryRow; +#[cfg(test)] +use crate::scalar_fn::row::probe; /// OGC `ST_Intersects` (not disjoint; boundary contact counts) between two native geometry /// operands, each a column or a constant literal. @@ -58,74 +40,97 @@ impl SpatialIntersects { } } -impl ScalarFnVTable for SpatialIntersects { +impl RowFn for SpatialIntersects { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["a", "b"]; + const FALLIBLE: bool = true; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.st.intersects"); *ID } - fn serialize(&self, _: &Self::Options) -> VortexResult>> { + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { Ok(Some(vec![])) } - fn deserialize(&self, _: &[u8], _: &VortexSession) -> VortexResult { + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { Ok(EmptyOptions) } - fn arity(&self, _: &Self::Options) -> Arity { - Arity::Exact(2) - } - - fn child_name(&self, _: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("a"), - 1 => ChildName::from("b"), - _ => unreachable!("intersects has exactly two children"), - } - } - - fn return_dtype(&self, _: &Self::Options, dtypes: &[DType]) -> VortexResult { - validate_intersects_operands(dtypes)?; - let nullability = Nullability::from(dtypes.iter().any(DType::is_nullable)); - Ok(DType::Bool(nullability)) - } - - fn execute( + fn dispatch( &self, - _: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let a = args.get(0)?; - let b = args.get(1)?; - // Disjoint bounding rects prove the geometries disjoint; rect contact (closed test) - // falls through to the exact test. - execute_binary_geo_types( - &a, - &b, - |x, y| x.intersects(y), - Some(|ra, rb| (!ra.intersects(rb)).then_some(false)), - ctx, + _options: &Self::Options, + _args: &[DType], + visitor: V, + ) -> VortexResult { + visitor.visit_prepared_into::<(GeometryRow, GeometryRow), ElementSink, _, _>( + |(a, b)| { + #[cfg(test)] + probe::record(a.is_some(), b.is_some()); + ConstBboxes::new(a, b) + }, + |bboxes, (a, b), output| *output = intersects_row_prepared(bboxes, a, b), ) } +} - fn validity( - &self, - _: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - union_child_validities(expression) - } +/// Per-batch state for the intersects row kernel: the bounding rect of each operand that is +/// constant for the batch. +/// +/// geo opens many intersects pairings with `has_disjoint_bboxes`, an early-out that folds +/// [`bounding_rect`] over both operands. For a batch-constant operand that fold recomputes the +/// same rect every row, so it is hoisted here and [`intersects_row_prepared`] replays the +/// comparison with the hoisted value. `None` marks an operand that varies by row or has no +/// bounding rect (an empty geometry); both mean no early-out, exactly as `has_disjoint_bboxes` +/// treats a missing rect. +/// +/// [`bounding_rect`]: BoundingRect::bounding_rect +struct ConstBboxes { + /// The bounding rect of operand `a` when it is batch-constant. + a: Option>, + + /// The bounding rect of operand `b` when it is batch-constant. + b: Option>, +} - fn is_strict(&self, _: &Self::Options) -> bool { - true +impl ConstBboxes { + fn new(a: Option<&Geometry>, b: Option<&Geometry>) -> Self { + Self { + a: a.and_then(BoundingRect::bounding_rect), + b: b.and_then(BoundingRect::bounding_rect), + } } +} - fn is_fallible(&self, _: &Self::Options) -> bool { - false - } +/// Computes one row of intersects, spending any bounding rect hoisted into `bboxes`. +/// +/// Disjoint bounding rectangles conservatively prove that the geometries do not intersect. The +/// fall-through delegates to the unchanged `a.intersects(b)`, which refolds both rects internally, +/// so a batch where every row overlaps pays one extra `bounding_rect` fold over the row operand; +/// the win concentrates where most rows are disjoint, the usual spatial-filter shape. +fn intersects_row_prepared(bboxes: &ConstBboxes, a: &Geometry, b: &Geometry) -> bool { + let disjoint = match (bboxes.a, bboxes.b) { + (None, None) => false, + (Some(bbox_a), Some(bbox_b)) => !bbox_a.intersects(&bbox_b), + (Some(bbox_a), None) => b + .bounding_rect() + .is_some_and(|bbox_b| !bbox_a.intersects(&bbox_b)), + (None, Some(bbox_b)) => a + .bounding_rect() + .is_some_and(|bbox_a| !bbox_a.intersects(&bbox_b)), + }; + + if disjoint { + return false; + } + + a.intersects(b) } #[cfg(test)] @@ -133,7 +138,9 @@ mod tests { use geo_types::Coord; use geo_types::Geometry; use geo_types::LineString; + use geo_types::MultiPoint; use geo_types::MultiPolygon; + use geo_types::Point; use geo_types::Polygon; use rstest::rstest; use vortex_array::ArrayRef; @@ -157,8 +164,10 @@ mod tests { use wkb::writer::WriteOptions; use super::SpatialIntersects; + use crate::scalar_fn::row::probe::assert_prepared_agrees_with_columns; use crate::test_harness::nullable_point_column; use crate::test_harness::point_column; + use crate::test_harness::rect_column; /// A rectangle polygon with corners `(x0, y0)` and `(x1, y1)`, no holes. fn rect_polygon(x0: f64, y0: f64, x1: f64, y1: f64) -> Polygon { @@ -439,4 +448,85 @@ mod tests { assert!(result.is_err()); Ok(()) } + + // The prepared-vs-expanded agreement grid: every constant arrangement of a pairing must + // return exactly what the fully expanded columns return. + + /// A point geometry. + fn point(x: f64, y: f64) -> Geometry { + Geometry::Point(Point::new(x, y)) + } + + /// A linestring geometry through `coords`. + fn line(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::LineString(LineString::from(coords)) + } + + /// A multipoint geometry over `coords`. + fn multipoint(coords: Vec<(f64, f64)>) -> Geometry { + Geometry::MultiPoint(MultiPoint::from(coords)) + } + + /// Constant arrangements agree with expanded columns across the pairing classes the prepared + /// kernel treats differently: bbox-prechecked pairs (polygon x polygon, linestring x + /// anything, multipolygon blankets), direct pairs (points), the excluded `MultiPoint` route, + /// and an empty geometry whose bounding rect does not exist. + #[rstest] + #[case::polygons_overlapping(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(2.0, 2.0, 6.0, 6.0).into())] + #[case::polygons_touching_edge(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(4.0, 0.0, 8.0, 4.0).into())] + #[case::polygons_touching_corner(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(4.0, 4.0, 8.0, 8.0).into())] + #[case::polygons_disjoint(rect_polygon(0.0, 0.0, 4.0, 4.0).into(), rect_polygon(20.0, 20.0, 24.0, 24.0).into())] + #[case::polygons_nested(rect_polygon(0.0, 0.0, 8.0, 8.0).into(), rect_polygon(2.0, 2.0, 4.0, 4.0).into())] + #[case::polygon_x_point_inside(donut(), point(2.0, 2.0))] + #[case::polygon_x_point_on_boundary(donut(), point(0.0, 5.0))] + #[case::polygon_x_point_in_hole(donut(), point(5.0, 5.0))] + #[case::point_outside_x_polygon(point(20.0, 20.0), donut())] + #[case::nan_point_x_polygon(point(f64::NAN, 2.0), donut())] + #[case::polygon_x_nan_point(donut(), point(f64::NAN, 2.0))] + #[case::points_equal(point(1.0, 1.0), point(1.0, 1.0))] + #[case::points_distinct(point(1.0, 1.0), point(2.0, 1.0))] + #[case::linestring_crossing_polygon(line(vec![(-2.0, -2.0), (2.0, 2.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::linestring_disjoint_polygon(line(vec![(-2.0, -2.0), (-6.0, -6.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::linestring_in_polygon_hole(line(vec![(4.5, 4.5), (5.5, 5.5)]), donut())] + #[case::linestrings_crossing(line(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(0.0, 4.0), (4.0, 0.0)]))] + #[case::linestrings_disjoint(line(vec![(0.0, 0.0), (4.0, 4.0)]), line(vec![(10.0, 10.0), (14.0, 14.0)]))] + #[case::empty_linestring_x_polygon(line(vec![]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipoint_straddling_polygon(multipoint(vec![(2.0, 2.0), (20.0, 20.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipoint_outside_polygon(multipoint(vec![(20.0, 20.0), (30.0, 30.0)]), rect_polygon(0.0, 0.0, 4.0, 4.0).into())] + #[case::multipolygon_disjoint_polygon( + Geometry::MultiPolygon(MultiPolygon::new(vec![ + rect_polygon(0.0, 0.0, 2.0, 2.0), + rect_polygon(10.0, 10.0, 12.0, 12.0), + ])), + rect_polygon(20.0, 20.0, 24.0, 24.0).into() + )] + fn constant_operands_agree_with_columns( + #[case] a: Geometry, + #[case] b: Geometry, + ) -> VortexResult<()> { + assert_prepared_agrees_with_columns( + SpatialIntersects::try_new_array, + geometry_constant(&a, 3)?, + geometry_constant(&b, 3)?, + ) + } + + /// `Rect` has no WKB form, so its constant comes from a one-row rect column; its conservative + /// bbox early-out and exact fall-through must agree with the expanded form like the rest. + #[test] + fn rect_operand_agrees_with_columns() -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + + let rect_scalar = rect_column(vec![(0.0, 0.0, 4.0, 4.0)])?.execute_scalar(0, &mut ctx)?; + let rect_constant = ConstantArray::new(rect_scalar, 3).into_array(); + let polygon_constant = + geometry_constant(&Geometry::Polygon(rect_polygon(2.0, 2.0, 6.0, 6.0)), 3)?; + + assert_prepared_agrees_with_columns( + SpatialIntersects::try_new_array, + rect_constant, + polygon_constant, + ) + } } diff --git a/vortex-spatial/src/scalar_fn/mod.rs b/vortex-spatial/src/scalar_fn/mod.rs index e6770be4fff..bcdb15e51e6 100644 --- a/vortex-spatial/src/scalar_fn/mod.rs +++ b/vortex-spatial/src/scalar_fn/mod.rs @@ -8,3 +8,4 @@ pub mod distance; pub mod envelope; mod execute; pub mod intersects; +pub(crate) mod row; diff --git a/vortex-spatial/src/scalar_fn/row.rs b/vortex-spatial/src/scalar_fn/row.rs new file mode 100644 index 00000000000..ce7c34072ff --- /dev/null +++ b/vortex-spatial/src/scalar_fn/row.rs @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What the geo scalar functions add to the row-function machinery: an element type that decodes a +//! native geometry column into `geo_types` geometries. + +use geo_types::Geometry; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::dtype::DType; +use vortex_array::scalar_fn::InputElement; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +use crate::extension::geometries; +use crate::extension::geometries_null_tolerant; +use crate::extension::is_native_geometry; + +/// Marker for native geometry input elements: accepts any native geometry column and presents each +/// row as a decoded `geo_types` geometry. +/// +/// The two operands of a binary geo function need not share a geometry type, since distance, +/// containment and intersection across types are all meaningful, so this validates only that the +/// column is *some* native geometry. +pub struct GeometryRow; + +impl InputElement for GeometryRow { + type Column = Vec>; + type Varying<'a> = &'a [Geometry]; + type Elem<'a> = &'a Geometry; + + // A geometry row is decoded from its coordinate storage, which behind a null row holds arbitrary + // coordinates that need not describe a well-formed geometry. + const DENSE_SAFE: bool = false; + // Decoding builds a geometry from stored coordinates, and a malformed one in a *valid* row is a + // domain error rather than an infrastructural failure. + const DECODE_FALLIBLE: bool = true; + // Decoding arrow-exports the column and parses one geometry per row, so filtering the column + // first shrinks the decode itself, not just the row loop. + const FILTERED_DECODE_COST: usize = 1; + + fn validate(dtype: &DType) -> VortexResult<()> { + vortex_ensure!( + is_native_geometry(dtype), + "spatial: operand {dtype} is not a native geometry type" + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + geometries(&array, ctx) + } + + fn get(column: &Self::Column, index: usize) -> &Geometry { + &column[index] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column.as_slice() + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.len() + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a Geometry + where + Self: 'a, + { + &column[index] + } + + /// Null rows decode to a placeholder geometry that the branch-and-skip row loop never reads. + /// `Point` and `Polygon` columns are covered; other geometry types return `Ok(None)` and the + /// batch falls back to the filter strategy. + fn decode_null_tolerant( + array: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + geometries_null_tolerant(&array, ctx) + } +} + +/// Test-only support for the prepared geo row kernels: a probe recording which operands a +/// `prepare` step saw as batch-constant, and the shared prepared-vs-expanded agreement check +/// built on it. +#[cfg(test)] +pub(crate) mod probe { + use std::cell::Cell; + + use vortex_array::ArrayRef; + use vortex_array::Canonical; + use vortex_array::ExecutionCtx; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::MaskedArray; + use vortex_array::arrays::ScalarFnArray; + use vortex_array::assert_arrays_eq; + use vortex_array::validity::Validity; + use vortex_error::VortexResult; + + thread_local! { + /// Which operands the last `prepare` saw as constant, as a bitmask (bit 0 for `a`, bit 1 + /// for `b`). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + pub(crate) static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; + } + + /// Record which operands `prepare` saw as constant. + pub(crate) fn record(a_constant: bool, b_constant: bool) { + SEEN_CONSTANTS.set(u8::from(a_constant) | (u8::from(b_constant) << 1)); + } + + /// Execute `build(a, b)` and assert that `prepare` saw exactly `expect_seen` as its constant + /// operands, so the test knows which decode path the inputs took. + fn run_probed( + build: &impl Fn(ArrayRef, ArrayRef) -> VortexResult, + a: ArrayRef, + b: ArrayRef, + expect_seen: u8, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + SEEN_CONSTANTS.set(u8::MAX); + let result = build(a, b)? + .into_array() + .execute::(ctx)? + .into_array(); + + assert_eq!( + SEEN_CONSTANTS.get(), + expect_seen, + "prepare saw the wrong constant operands", + ); + Ok(result) + } + + /// Assert that every constant-operand arrangement of `build(a, b)` returns exactly what the + /// fully expanded columns return, and that each arrangement's constness really reached + /// `prepare` (so the constants exercised the stride-0 path rather than a decoded column). + /// + /// Arrangements: `a` constant, `b` constant, and both constant with `a` masked. A plain + /// constant pair folds to a single-row execution before the row loop, so masking one side is + /// what drives the both-hoisted arm across rows; that run is compared against the same mask + /// over the expanded column. + pub(crate) fn assert_prepared_agrees_with_columns( + build: impl Fn(ArrayRef, ArrayRef) -> VortexResult, + const_a: ArrayRef, + const_b: ArrayRef, + ) -> VortexResult<()> { + let session = vortex_array::array_session(); + let mut ctx = session.create_execution_ctx(); + let col_a = const_a.clone().execute::(&mut ctx)?.into_array(); + let col_b = const_b.clone().execute::(&mut ctx)?.into_array(); + + let baseline = run_probed(&build, col_a.clone(), col_b.clone(), 0b00, &mut ctx)?; + let a_hoisted = run_probed(&build, const_a.clone(), col_b.clone(), 0b01, &mut ctx)?; + let b_hoisted = run_probed(&build, col_a.clone(), const_b.clone(), 0b10, &mut ctx)?; + assert_arrays_eq!(a_hoisted, baseline, &mut ctx); + assert_arrays_eq!(b_hoisted, baseline, &mut ctx); + + let validity = Validity::from_iter((0..col_a.len()).map(|row| row != 1)); + let masked_const_a = MaskedArray::try_new(const_a, validity.clone())?.into_array(); + let masked_col_a = MaskedArray::try_new(col_a, validity)?.into_array(); + let both_hoisted = run_probed(&build, masked_const_a, const_b, 0b11, &mut ctx)?; + let masked_baseline = run_probed(&build, masked_col_a, col_b, 0b00, &mut ctx)?; + assert_arrays_eq!(both_hoisted, masked_baseline, &mut ctx); + + Ok(()) + } +} diff --git a/vortex-spatial/src/test_harness.rs b/vortex-spatial/src/test_harness.rs index 7b471bdf2c4..d8175d14f53 100644 --- a/vortex-spatial/src/test_harness.rs +++ b/vortex-spatial/src/test_harness.rs @@ -251,7 +251,7 @@ pub fn nullable_rect_column(boxes: Vec>) -> VortexR Ok(ExtensionArray::try_new(ext.erased(), storage)?.into_array()) } -/// Decode a [`Coordinate`] from an extension-typed point scalar (unwrapped to its coordinate +/// Decode a `Coordinate` from an extension-typed point scalar (unwrapped to its coordinate /// storage) or a bare coordinate `Struct` scalar — used to read back a single point in assertions. pub fn coordinate_from_scalar(scalar: &Scalar) -> VortexResult { match scalar.as_extension_opt() { diff --git a/vortex-tensor/benches/cosine_similarity.rs b/vortex-tensor/benches/cosine_similarity.rs index 6cc5eb867ef..fef94a0aa91 100644 --- a/vortex-tensor/benches/cosine_similarity.rs +++ b/vortex-tensor/benches/cosine_similarity.rs @@ -22,10 +22,12 @@ use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity; @@ -85,9 +87,9 @@ fn bench_cosine(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef) { bencher .with_inputs(|| { ( - CosineSimilarity::try_new_array(lhs.clone(), rhs.clone()) - .unwrap() - .into_array(), + CosineSimilarity + .try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/benches/inner_product.rs b/vortex-tensor/benches/inner_product.rs index 796e9b648d6..c0918f87ba2 100644 --- a/vortex-tensor/benches/inner_product.rs +++ b/vortex-tensor/benches/inner_product.rs @@ -20,6 +20,8 @@ use vortex_array::VortexSessionExecute; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::inner_product::InnerProduct; @@ -62,9 +64,9 @@ fn bench_inner_product(bencher: Bencher, lhs: ArrayRef, rhs: ArrayRef) { .counter(ItemsCount::new(lhs.len())) .with_inputs(|| { ( - InnerProduct::try_new_array(lhs.clone(), rhs.clone()) - .unwrap() - .into_array(), + InnerProduct + .try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/benches/l2_norm.rs b/vortex-tensor/benches/l2_norm.rs index 6f597084113..d96e4877af9 100644 --- a/vortex-tensor/benches/l2_norm.rs +++ b/vortex-tensor/benches/l2_norm.rs @@ -20,6 +20,8 @@ use vortex_array::VortexSessionExecute; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_tensor::scalar_fns::l2_norm::L2Norm; @@ -60,7 +62,9 @@ fn bench_l2_norm(bencher: Bencher, input: ArrayRef) { .counter(ItemsCount::new(input.len())) .with_inputs(|| { ( - L2Norm::try_new_array(input.clone()).unwrap().into_array(), + L2Norm + .try_new_array(input.len(), EmptyOptions, [input.clone()]) + .unwrap(), session.create_execution_ctx(), ) }) diff --git a/vortex-tensor/src/encodings/normalized/execute.rs b/vortex-tensor/src/encodings/normalized/execute.rs index 637c8c07117..b96a03113b7 100644 --- a/vortex-tensor/src/encodings/normalized/execute.rs +++ b/vortex-tensor/src/encodings/normalized/execute.rs @@ -14,7 +14,6 @@ use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; use vortex_array::arrays::fixed_size_list::FixedSizeListArraySlotsExt; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; -use vortex_array::dtype::NativePType; use vortex_array::match_each_float_ptype; use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::fns::operators::Operator; @@ -24,6 +23,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::matcher::AnyTensor; +use crate::utils::build_tensor_array; use crate::utils::extract_flat_elements; use crate::utils::unit_norm_tolerance; @@ -115,26 +115,6 @@ fn denormalize_constant_norms( Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) } -/// Rebuilds a tensor-like extension array from flat primitive elements. -fn build_tensor_array( - dtype: DType, - tensor_flat_size: usize, - row_count: usize, - validity: Validity, - elements: Buffer, -) -> VortexResult { - let list_size = - u32::try_from(tensor_flat_size).vortex_expect("tensor flat size must fit into `u32`"); - - // SAFETY: Tensor elements are always non-nullable, so the validity carries no length. - let elements = unsafe { PrimitiveArray::new_unchecked(elements, Validity::NonNullable) }; - - let storage = - FixedSizeListArray::try_new(elements.into_array(), list_size, validity, row_count)?; - - Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) -} - /// Returns the flattened element count of each row of a tensor-like extension dtype. fn tensor_flat_size(dtype: &DType) -> usize { dtype diff --git a/vortex-tensor/src/encodings/normalized/mod.rs b/vortex-tensor/src/encodings/normalized/mod.rs index 545236bba7d..2c9a72d2988 100644 --- a/vortex-tensor/src/encodings/normalized/mod.rs +++ b/vortex-tensor/src/encodings/normalized/mod.rs @@ -31,7 +31,6 @@ pub use array::NormalizedSlots; mod compress; pub use compress::NormalizedScheme; pub use compress::normalize; -pub(crate) use compress::try_build_constant_normalized; mod execute; diff --git a/vortex-tensor/src/scalar_fns/cosine_similarity.rs b/vortex-tensor/src/scalar_fns/cosine_similarity.rs index ca8fcf0efd4..e786d49b47c 100644 --- a/vortex-tensor/src/scalar_fns/cosine_similarity.rs +++ b/vortex-tensor/src/scalar_fns/cosine_similarity.rs @@ -1,48 +1,49 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Cosine similarity expression for tensor-like types. +//! Cosine similarity between two tensor columns. +use num_traits::Float; use num_traits::Zero; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::ScalarFnArray; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; +use vortex_array::dtype::NativePType; use vortex_array::match_each_float_ptype; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_array::serde::ArrayChildren; +use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::NormalizedOrientation; -use crate::encodings::normalized::try_build_constant_normalized; use crate::scalar_fns::inner_product::InnerProduct; use crate::scalar_fns::l2_norm::L2Norm; +use crate::scalar_fns::row::TensorRow; +#[cfg(test)] +use crate::scalar_fns::row::probe; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::BinaryTensorOpMetadata; use crate::utils::extract_normalized_children; -use crate::utils::validate_binary_tensor_float_inputs; +use crate::utils::l2_norm_row; /// Cosine similarity between two columns. /// /// Computes `dot(a, b) / (||a|| * ||b||)` over the flat backing buffer of each tensor or vector. /// The shape and permutation do not affect the result because cosine similarity only depends on the -/// element values, not their logical arrangement. +/// element values, not their logical arrangement. A zero norm on either side yields `0.0`. /// /// Both inputs must be tensor-like extension arrays ([`FixedShapeTensor`] or [`Vector`]) with the /// same dtype and a float element type. The output is a float column of the same float type. @@ -55,143 +56,79 @@ use crate::utils::validate_binary_tensor_float_inputs; /// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector /// [`Normalized`]: crate::encodings::normalized::Normalized -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct CosineSimilarity; -impl CosineSimilarity { - /// Creates a new [`TypedScalarFnInstance`] wrapping the cosine similarity operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(CosineSimilarity, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the cosine similarity between `lhs` and - /// `rhs`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(CosineSimilarity::new().erased(), vec![lhs, rhs]) - } -} - -impl ScalarFnVTable for CosineSimilarity { +impl RowFn for CosineSimilarity { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.cosine_similarity"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(2) + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("lhs"), - 1 => ChildName::from("rhs"), - _ => unreachable!("CosineSimilarity must have exactly two children"), - } + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let lhs = &arg_dtypes[0]; - let rhs = &arg_dtypes[1]; - - let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; - let ptype = tensor_match.element_ptype(); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow, TensorRow), ElementSink, _, _>( + |(lhs, rhs)| { + #[cfg(test)] + probe::record(lhs.is_some(), rhs.is_some()); + ConstNorms { + lhs: lhs.map(l2_norm_row), + rhs: rhs.map(l2_norm_row), + } + }, + |norms, (lhs, rhs), output| { + *output = cosine_similarity_row_prepared(norms, lhs, rhs); + }, + ) + }) } - fn execute( + /// [`Normalized`]-encoded operands make the *stored* norms and normalized children + /// authoritative: `cos(D(x, s), D(y, t)) = dot(x, y)` and `cos(D(x, s), y) = dot(x, y) / + /// ||y||`, in both cases forced to `0.0` on rows where any authoritative norm is `0.0` (even + /// for lossy children whose decoded coordinates are nonzero). + /// + /// [`Normalized`]: crate::encodings::normalized::Normalized + fn reduce_encoded( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, + args: &[ArrayRef], ctx: &mut ExecutionCtx, - ) -> VortexResult { - let mut lhs_ref = args.get(0)?; - let mut rhs_ref = args.get(1)?; - let len = args.row_count(); - - // If either side is a constant tensor-like extension array, eagerly normalize the single - // stored row and re-encode it as an `Normalized` whose children are both `ConstantArray`s. - // The `Normalized` fast path below then picks it up. - if let Some(normalized_array) = try_build_constant_normalized(&lhs_ref, len, ctx)? { - lhs_ref = normalized_array.into_array(); - } - if let Some(normalized_array) = try_build_constant_normalized(&rhs_ref, len, ctx)? { - rhs_ref = normalized_array.into_array(); - } + ) -> VortexResult> { + let lhs = args[0].clone(); + let rhs = args[1].clone(); - // Take any Normalized read-through fast path that applies. - match NormalizedOrientation::classify(&lhs_ref, &rhs_ref) { + match NormalizedOrientation::classify(&lhs, &rhs) { NormalizedOrientation::Both { lhs, rhs } => { - return self.execute_both_normalized(lhs, rhs, len, ctx); + cosine_both_normalized(lhs, rhs, ctx).map(Some) } NormalizedOrientation::One { normalized_array, plain, - } => { - return self.execute_one_normalized(normalized_array, plain, len, ctx); - } - NormalizedOrientation::Neither => {} + } => cosine_one_normalized(normalized_array, plain, ctx).map(Some), + NormalizedOrientation::Neither => Ok(None), } - - // Compute combined validity. - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - // Compute inner product and norms as columnar operations, and propagate the options. - let norm_lhs_arr = L2Norm::try_new_array(lhs_ref.clone())?; - let norm_rhs_arr = L2Norm::try_new_array(rhs_ref.clone())?; - let dot_arr = InnerProduct::try_new_array(lhs_ref, rhs_ref)?; - - // Execute to get the inner product and norms of the arrays. We only fully decompress - // because we need to perform special logic (guard against 0) during division. - let dot: PrimitiveArray = dot_arr.into_array().execute(ctx)?; - let norm_l: PrimitiveArray = norm_lhs_arr.into_array().execute(ctx)?; - let norm_r: PrimitiveArray = norm_rhs_arr.into_array().execute(ctx)?; - - // TODO(connor): Ideally we would have a `SafeDiv` binary numeric operation. - // TODO(connor): This can be written in a more SIMD-friendly manner. - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let norms_l = norm_l.as_slice::(); - let norms_r = norm_r.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - let denom = norms_l[i] * norms_r[i]; - - if denom == T::zero() { - T::zero() - } else { - dots[i] / denom - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - fn validity( - &self, - _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if either input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false } } @@ -221,578 +158,177 @@ impl ScalarFnArrayVTable for CosineSimilarity { } } -impl CosineSimilarity { - /// Both sides are [`Normalized`]-encoded: treat the normalized children as authoritative, so - /// `cosine_similarity = dot(n_l, n_r)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - fn execute_both_normalized( - &self, - lhs_ref: &ArrayRef, - rhs_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - let (normalized_l, norms_l) = extract_normalized_children(lhs_ref); - let (normalized_r, norms_r) = extract_normalized_children(rhs_ref); - - // `Normalized` makes the normalized children authoritative, so their dot product is the - // cosine similarity even for lossy storage wrappers, except that a zero stored norm still - // represents a zero vector. - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r)? - .into_array() - .execute(ctx)?; - let norms_l: PrimitiveArray = norms_l.execute(ctx)?; - let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let norms_l = norms_l.as_slice::(); - let norms_r = norms_r.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - if norms_l[i] == T::zero() || norms_r[i] == T::zero() { - T::zero() - } else { - dots[i] - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - /// One side is [`Normalized`]-encoded: treat the normalized child as authoritative, so - /// `cosine_similarity = dot(n, b) / ||b||`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - /// - /// The caller must pass the [`Normalized`] array as `normalized_ref` and the plain array as `plain_ref`. - fn execute_one_normalized( - &self, - normalized_ref: &ArrayRef, - plain_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = normalized_ref.validity()?.and(plain_ref.validity()?)?; - - let (normalized, normalized_norms) = extract_normalized_children(normalized_ref); - - let dot_arr = InnerProduct::try_new_array(normalized, plain_ref.clone())?; - let dot: PrimitiveArray = dot_arr.into_array().execute(ctx)?; - - let normalized_norms: PrimitiveArray = normalized_norms.execute(ctx)?; - - let norm_arr = L2Norm::try_new_array(plain_ref.clone())?; - let plain_norm: PrimitiveArray = norm_arr.into_array().execute(ctx)?; - - // TODO(connor): Ideally we would have a `SafeDiv` binary numeric operation. - // TODO(connor): This can be written in a more SIMD-friendly manner. - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let normalized_norms = normalized_norms.as_slice::(); - let plain_norms = plain_norm.as_slice::(); - let buffer: Buffer = (0..len) - .map(|i| { - if normalized_norms[i] == T::zero() || plain_norms[i] == T::zero() { - T::zero() - } else { - dots[i] / plain_norms[i] - } - }) - .collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } +/// Per-batch state for the cosine row kernel: the L2 norm of each operand that is constant for +/// the batch. +/// +/// A broadcast query vector holds the same elements in every row, so its norm is the same in +/// every row too. Computing it in the prepare step hoists an `O(width)` pass and a `sqrt` per row +/// out of the row loop. `None` marks an operand that varies by row, whose norm the row closure +/// computes exactly as it did before the hoist. +struct ConstNorms { + /// The norm of the lhs when it is batch-constant. + lhs: Option, + + /// The norm of the rhs when it is batch-constant. + rhs: Option, } -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::encodings::normalized::Normalized; - use crate::scalar_fns::cosine_similarity::CosineSimilarity; - use crate::tests::SESSION; - use crate::types::vector::Vector; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::constant_tensor_array; - use crate::utils::test_helpers::normalized_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates cosine similarity between two tensor arrays and returns the result as `Vec`. - fn eval_cosine_similarity(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - #[test] - fn unit_vectors_1d() -> VortexResult<()> { - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // Tensor 1 - 0.0, 1.0, 0.0, // Tensor 2 - ], - )?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // Tensor 1 - 1.0, 0.0, 0.0, // Tensor 2 - ], - )?; - - // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - /// Single-row cosine similarity for various vector pairs. - #[rstest] - // Antiparallel -> -1.0. - #[case::opposite(&[3], &[1.0, 0.0, 0.0], &[-1.0, 0.0, 0.0], &[-1.0])] - // dot=24, both magnitudes=5 -> 24/25 = 0.96. - #[case::non_unit(&[2], &[3.0, 4.0], &[4.0, 3.0], &[0.96])] - // Zero vector -> guarded to 0.0. - #[case::zero_norm(&[2], &[0.0, 0.0], &[1.0, 0.0], &[0.0])] - fn single_row( - #[case] shape: &[usize], - #[case] lhs_elems: &[f64], - #[case] rhs_elems: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let lhs = tensor_array(shape, lhs_elems)?; - let rhs = tensor_array(shape, rhs_elems)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, expected); - Ok(()) - } - - /// Self-similarity across various tensor shapes should always produce 1.0. - #[rstest] - // 2x3 matrix, flattened to 6 elements. - #[case::matrix_2d( - &[2, 3], - &[ - 1.0, 0.0, 0.0, // row 0 - 0.0, 0.0, 0.0, // row 1 - ], - )] - // 2x2x2 tensor, 8 elements. - #[case::tensor_3d(&[2, 2, 2], &[1.0; 8])] - fn self_similarity(#[case] shape: &[usize], #[case] elements: &[f64]) -> VortexResult<()> { - let lhs = tensor_array(shape, elements)?; - let rhs = tensor_array(shape, elements)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); - Ok(()) - } - - #[test] - fn scalar_0d() -> VortexResult<()> { - // 0-dimensional tensor: each "tensor" is a single scalar value. - let lhs = tensor_array(&[], &[5.0, 3.0])?; - let rhs = tensor_array(&[], &[5.0, -3.0])?; - - // Same sign -> 1.0, opposite sign -> -1.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, -1.0]); - Ok(()) - } - - #[test] - fn many_rows() -> VortexResult<()> { - // 5 tensors of shape [4] compared against themselves -> all 1.0. - let lhs = tensor_array( - &[4], - &[ - 1.0, 2.0, 3.0, 4.0, // tensor 0 - 0.0, 1.0, 0.0, 0.0, // tensor 1 - 5.0, 0.0, 5.0, 0.0, // tensor 2 - 1.0, 1.0, 1.0, 1.0, // tensor 3 - 0.0, 0.0, 0.0, 7.0, // tensor 4 - ], - )?; - let rhs = lhs.clone(); - - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0, 1.0, 1.0, 1.0, 1.0], - ); - Ok(()) - } - - #[test] - fn constant_query_tensor() -> VortexResult<()> { - // Compare 4 tensors of shape [3] against a single constant query tensor [1,0,0]. - let data = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // tensor 0 - 0.0, 1.0, 0.0, // tensor 1 - 0.0, 0.0, 1.0, // tensor 2 - 1.0, 0.0, 0.0, // tensor 3 - ], - )?; - let query = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 4)?; - - assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); - Ok(()) - } - - #[test] - fn vector_unit_vectors() -> VortexResult<()> { - let lhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 0.0, 1.0, 0.0, // vector 1 - ], - )?; - let rhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 1.0, 0.0, 0.0, // vector 1 - ], - )?; - - // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - #[test] - fn vector_constant_query() -> VortexResult<()> { - let data = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // vector 0 - 0.0, 1.0, 0.0, // vector 1 - 0.0, 0.0, 1.0, // vector 2 - 1.0, 0.0, 0.0, // vector 3 - ], - )?; - let query = Vector::constant_array(&[1.0, 0.0, 0.0], 4)?; - - assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 2 rows of dim-2 vectors. Row 1 of rhs is masked as null. - let lhs = tensor_array(&[2], &[3.0, 4.0, 1.0, 0.0])?; - let rhs = tensor_array(&[2], &[3.0, 4.0, 0.0, 1.0])?; - let rhs = MaskedArray::try_new(rhs, Validity::from_iter([true, false]))?.into_array(); - - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: self-similarity = 1.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[1.0]); - Ok(()) - } - - #[test] - fn both_normalized_self_similarity() -> VortexResult<()> { - // [3.0, 4.0] has norm 5.0, normalized [0.6, 0.8]. - // [1.0, 0.0] has norm 1.0, normalized [1.0, 0.0]. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - // Self-similarity should always be 1.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 1.0]); - Ok(()) - } - - #[test] - fn both_normalized_orthogonal() -> VortexResult<()> { - // [3.0, 0.0] normalized [1.0, 0.0], norm 3.0. - // [0.0, 4.0] normalized [0.0, 1.0], norm 4.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[1.0, 0.0], &[3.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.0, 1.0], &[4.0], &mut ctx)?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); - Ok(()) - } - - #[test] - fn both_normalized_zero_norm() -> VortexResult<()> { - // Zero-norm row: normalized is [0.0, 0.0], norm is 0.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - // Row 0: dot([0.6, 0.8], [0.6, 0.8]) = 1.0, row 1: dot([0,0], [1,0]) = 0.0. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lhs() -> VortexResult<()> { - // LHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // RHS is plain [3.0, 4.0]. - // cosine_similarity([3.0, 4.0], [3.0, 4.0]) = 1.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = tensor_array(&[2], &[3.0, 4.0])?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_rhs() -> VortexResult<()> { - // LHS is plain [1.0, 0.0], RHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // cosine_similarity([1.0, 0.0], [3.0, 4.0]) = 3.0 / (1.0 * 5.0) = 0.6. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = tensor_array(&[2], &[1.0, 0.0])?; - let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.6]); - Ok(()) - } - - #[test] - fn both_normalized_null_norms() -> VortexResult<()> { - // Row 0: valid, row 1: null (via nullable norms on rhs). - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - let normalized_r = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; - let norms_r = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); - let rhs = Normalized::try_new(normalized_r, norms_r, &mut ctx)?.into_array(); - - let scalar_fn = CosineSimilarity::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[1.0]); - Ok(()) - } - - #[test] - fn both_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { - // Mimics a lossy encoding where the stored norm is authoritative but - // the decoded normalized child is physically nonzero. With a stored norm of `0.0`, cosine - // similarity for that row must be `0.0` even though the dot product of the normalized - // children is nonzero. - let normalized_l = tensor_array(&[2], &[0.6, 0.8])?; - let norms_l = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Intentionally violates the unit-norm invariant by pairing a nonzero normalized row - // with a stored norm of `0.0`, mimicking lossy storage. - // SAFETY: The children are structurally valid. - let lhs = unsafe { Normalized::new_unchecked(normalized_l, norms_l) }.into_array(); - - let normalized_r = tensor_array(&[2], &[0.6, 0.8])?; - let norms_r = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Same as above for the rhs operand. - // SAFETY: The children are structurally valid. - let rhs = unsafe { Normalized::new_unchecked(normalized_r, norms_r) }.into_array(); - - // `dot(normalized_l, normalized_r) = 1.0`, but the authoritative stored norms are both - // `0.0`, so cosine similarity must be `0.0`. - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { - // Mimics a lossy encoding where the stored norm is authoritative but - // the decoded normalized child is physically nonzero. The plain side is a normal nonzero - // tensor with positive norm. cosine similarity must still be `0.0` because the - // authoritative stored norm on the normalized_array side is `0.0`. - let normalized = tensor_array(&[2], &[0.6, 0.8])?; - let norms = PrimitiveArray::from_iter([0.0f64]).into_array(); - // Intentionally pairs a nonzero normalized row with a stored norm of `0.0`, mimicking - // lossy storage where the stored norm is authoritative. - // SAFETY: The children are structurally valid. - let normalized_array = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); - - let plain = tensor_array(&[2], &[1.0, 0.0])?; - - // Normalized encoding on the lhs: `One { normalized_array: lhs, plain: rhs }`. - assert_close( - &eval_cosine_similarity(normalized_array.clone(), plain.clone())?, - &[0.0], - ); - - // Normalized encoding on the rhs: `One { normalized_array: rhs, plain: lhs }`. The same - // zero-norm guard must fire regardless of operand order. - assert_close(&eval_cosine_similarity(plain, normalized_array)?, &[0.0]); - Ok(()) - } - - #[test] - fn constant_lhs_matches_plain_tensor() -> VortexResult<()> { - // The constant query `[1, 2, 2]` has norm 3, so its normalized form is `[1/3, 2/3, 2/3]`. - // Expected cosine similarity against each row is `dot([1, 2, 2], row) / (3 * ||row||)`. - let lhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // dot=1, ||rhs||=1, expected=1/3 - 1.0, 2.0, 2.0, // dot=9, ||rhs||=3, expected=1 - 0.0, 0.0, 1.0, // dot=2, ||rhs||=1, expected=2/3 - 2.0, 1.0, 2.0, // dot=8, ||rhs||=3, expected=8/9 - ], - )?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[test] - fn constant_rhs_matches_plain_tensor() -> VortexResult<()> { - // Mirror of `constant_lhs_matches_plain_tensor` with the constant on the right. - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // - 1.0, 2.0, 2.0, // - 0.0, 0.0, 1.0, // - 2.0, 1.0, 2.0, // - ], - )?; - let rhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[test] - fn both_constant_tensors() -> VortexResult<()> { - // `[1, 0, 0]` vs `[1, 1, 0]`. dot=1, ||lhs||=1, ||rhs||=sqrt(2), expected=1/sqrt(2). - let lhs = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 3)?; - let rhs = constant_tensor_array(&[3], &[1.0, 1.0, 0.0], 3)?; - let expected = 1.0 / 2.0_f64.sqrt(); - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[expected, expected, expected], - ); - Ok(()) - } - - #[test] - fn constant_zero_norm_query() -> VortexResult<()> { - // A zero-norm constant query must produce `0.0` for every row via the zero-norm guard in - // `execute_one_normalized` and `execute_both_normalized`. - let lhs = constant_tensor_array(&[3], &[0.0, 0.0, 0.0], 3)?; - let rhs = tensor_array( - &[3], - &[ - 1.0, 2.0, 3.0, // - 4.0, 5.0, 6.0, // - 7.0, 8.0, 9.0, // - ], - )?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0, 0.0]); - Ok(()) - } - - #[test] - fn constant_self_similarity_nonunit() -> VortexResult<()> { - // A non-unit constant query compared to itself must produce `1.0`. This exercises the - // helper's division: after normalization, both sides must be exactly unit so the - // Normalized fast path's inner product yields 1. - let lhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; - let rhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; - assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0; 5]); - Ok(()) - } - - #[test] - fn vector_constant_matches_plain() -> VortexResult<()> { - // Exercise the `Vector` extension variant through the new pre-pass. - let lhs = Vector::constant_array(&[1.0, 2.0, 2.0], 4)?; - let rhs = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // - 1.0, 2.0, 2.0, // - 0.0, 0.0, 1.0, // - 2.0, 1.0, 2.0, // - ], - )?; - assert_close( - &eval_cosine_similarity(lhs, rhs)?, - &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], - ); - Ok(()) - } - - #[rstest] - #[case::vector(cosine_vector_lhs(), cosine_vector_rhs())] - #[case::fixed_shape_tensor(cosine_tensor_lhs(), cosine_tensor_rhs())] - fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { - let original = CosineSimilarity::try_new_array(lhs.clone(), rhs.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(CosineSimilarity); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("CosineSimilarity serialize must produce metadata"); - - let children = vec![lhs, rhs]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) +/// Computes the cosine similarity of one row, taking any hoisted norm from `norms` and computing +/// the rest exactly as [`cosine_similarity_row`] does. +/// +/// Each arm accumulates the same values in the same order as [`cosine_similarity_row`], and the +/// denominator keeps its lhs-times-rhs order, so the result is bit-identical whether a norm was +/// hoisted or not. The match costs one predictable branch per row: the arm is the same for the +/// whole batch. +fn cosine_similarity_row_prepared( + norms: &ConstNorms, + a: &[T], + b: &[T], +) -> T { + match (norms.lhs, norms.rhs) { + (None, None) => cosine_similarity_row(a, b), + (Some(norm_a), None) => { + let mut dot = T::zero(); + let mut norm_sq_b = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_b = norm_sq_b + y * y; + } + cosine_from_parts(dot, norm_a * norm_sq_b.sqrt()) + } + (None, Some(norm_b)) => { + let mut dot = T::zero(); + let mut norm_sq_a = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_a = norm_sq_a + x * x; + } + cosine_from_parts(dot, norm_sq_a.sqrt() * norm_b) + } + (Some(norm_a), Some(norm_b)) => { + let mut dot = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + } + cosine_from_parts(dot, norm_a * norm_b) + } } +} - fn cosine_vector_lhs() -> ArrayRef { - vector_array(3, &[1.0, 0.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") - } +/// Computes the cosine similarity of two equal-length float slices. +/// +/// Returns `dot(a, b) / (||a|| * ||b||)`, or `0.0` when either norm is zero. +fn cosine_similarity_row(a: &[T], b: &[T]) -> T { + let mut dot = T::zero(); + let mut norm_sq_a = T::zero(); + let mut norm_sq_b = T::zero(); + for (&x, &y) in a.iter().zip(b.iter()) { + dot = dot + x * y; + norm_sq_a = norm_sq_a + x * x; + norm_sq_b = norm_sq_b + y * y; + } + + cosine_from_parts(dot, norm_sq_a.sqrt() * norm_sq_b.sqrt()) +} - fn cosine_vector_rhs() -> ArrayRef { - vector_array(3, &[0.0, 1.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +/// The shared tail of every cosine arm: `dot / denom`, guarded to `0.0` when the denominator is +/// zero. +fn cosine_from_parts(dot: T, denom: T) -> T { + if denom == T::zero() { + T::zero() + } else { + dot / denom } +} - fn cosine_tensor_lhs() -> ArrayRef { - tensor_array(&[2], &[1.0, 0.0, 3.0, 4.0]).expect("valid tensor array") - } +/// Both sides are [`Normalized`]-encoded: the normalized children are authoritative, so their dot +/// product is the cosine similarity, except that a row with a zero *stored* norm is a zero vector. +/// +/// Unlike [`InnerProduct::reduce_encoded`], which composes lazy `Mul` arrays over the norm columns, +/// this executes and materializes. The zero-norm guard is a conditional per row rather than an +/// arithmetic factor, so there is no lazy array that expresses it; the norm columns are one value +/// per row rather than one per coordinate, so materializing them is cheap next to the decode this +/// avoids. +/// +/// [`InnerProduct::reduce_encoded`]: InnerProduct::reduce_encoded +/// [`Normalized`]: crate::encodings::normalized::Normalized +fn cosine_both_normalized( + lhs: &ArrayRef, + rhs: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let len = lhs.len(); + let (normalized_l, norms_l) = extract_normalized_children(lhs); + let (normalized_r, norms_r) = extract_normalized_children(rhs); + + let dot: PrimitiveArray = InnerProduct + .try_new_array(len, EmptyOptions, [normalized_l, normalized_r])? + .execute(ctx)?; + let norms_l: PrimitiveArray = norms_l.execute(ctx)?; + let norms_r: PrimitiveArray = norms_r.execute(ctx)?; + + match_each_float_ptype!(dot.ptype(), |T| { + let dots = dot.as_slice::(); + let norms_l = norms_l.as_slice::(); + let norms_r = norms_r.as_slice::(); + // Zipped rather than indexed by `0..len`: one bounds check per iterator instead of three + // per row. A length disagreement between the children shortens the result, which the + // lifting reports against the batch row count rather than panicking mid-loop. + let buffer: Buffer = dots + .iter() + .zip(norms_l) + .zip(norms_r) + .map(|((&dot, &norm_l), &norm_r)| { + if norm_l.is_zero() || norm_r.is_zero() { + T::zero() + } else { + dot + } + }) + .collect(); + + Ok(PrimitiveArray::new(buffer, Validity::NonNullable).into_array()) + }) +} - fn cosine_tensor_rhs() -> ArrayRef { - tensor_array(&[2], &[0.0, 1.0, 3.0, 4.0]).expect("valid tensor array") - } +/// One side is [`Normalized`]-encoded: `cos = dot(normalized, plain) / ||plain||`, forced to `0.0` +/// on rows where the stored norm or the plain norm is `0.0`. +/// +/// [`Normalized`]: crate::encodings::normalized::Normalized +fn cosine_one_normalized( + normalized_array: &ArrayRef, + plain: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let len = normalized_array.len(); + let (normalized, normalized_norms) = extract_normalized_children(normalized_array); + + let dot: PrimitiveArray = InnerProduct + .try_new_array(len, EmptyOptions, [normalized, plain.clone()])? + .execute(ctx)?; + let normalized_norms: PrimitiveArray = normalized_norms.execute(ctx)?; + let plain_norm: PrimitiveArray = L2Norm + .try_new_array(len, EmptyOptions, [plain.clone()])? + .execute(ctx)?; + + match_each_float_ptype!(dot.ptype(), |T| { + let dots = dot.as_slice::(); + let normalized_norms = normalized_norms.as_slice::(); + let plain_norms = plain_norm.as_slice::(); + // Zipped for the same reason as [`cosine_both_normalized`]. + let buffer: Buffer = dots + .iter() + .zip(normalized_norms) + .zip(plain_norms) + .map(|((&dot, &stored_norm), &plain_norm)| { + if stored_norm.is_zero() || plain_norm.is_zero() { + T::zero() + } else { + dot / plain_norm + } + }) + .collect(); + + Ok(PrimitiveArray::new(buffer, Validity::NonNullable).into_array()) + }) } diff --git a/vortex-tensor/src/scalar_fns/inner_product.rs b/vortex-tensor/src/scalar_fns/inner_product.rs index 53ae82eb4a2..3d8255f3599 100644 --- a/vortex-tensor/src/scalar_fns/inner_product.rs +++ b/vortex-tensor/src/scalar_fns/inner_product.rs @@ -6,40 +6,30 @@ use num_traits::Float; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; +use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; -use vortex_array::dtype::Nullability; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; use vortex_array::match_each_float_ptype; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::scalar_fn::fns::operators::Operator; use vortex_array::serde::ArrayChildren; -use vortex_buffer::Buffer; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::NormalizedOrientation; -use crate::matcher::AnyTensor; +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::BinaryTensorOpMetadata; -use crate::utils::extract_flat_elements; use crate::utils::extract_normalized_children; -use crate::utils::validate_binary_tensor_float_inputs; /// Inner product (dot product) between two columns. /// @@ -52,131 +42,82 @@ use crate::utils::validate_binary_tensor_float_inputs; /// /// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct InnerProduct; -impl InnerProduct { - /// Creates a new [`TypedScalarFnInstance`] wrapping the inner product operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(InnerProduct, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the inner product between `lhs` and - /// `rhs`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(InnerProduct::new().erased(), vec![lhs, rhs]) - } -} - -impl ScalarFnVTable for InnerProduct { +impl RowFn for InnerProduct { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["lhs", "rhs"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.inner_product"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(2) + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("lhs"), - 1 => ChildName::from("rhs"), - _ => unreachable!("InnerProduct must have exactly two children"), - } + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let lhs = &arg_dtypes[0]; - let rhs = &arg_dtypes[1]; - - // TODO(connor): relax the float-only gate once integer tensors are supported. - let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; - let ptype = tensor_match.element_ptype(); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow, TensorRow), ElementSink, _, _>( + |_| (), + |&(), (lhs, rhs), output| *output = inner_product_row(lhs, rhs), + ) + }) } - fn execute( + /// [`Normalized`]-encoded operands factor through their stored norms: with `D(x, s)` denoting + /// `x * s` rowwise, `dot(D(x, s), D(y, t)) = s * t * dot(x, y)` and + /// `dot(D(x, s), y) = s * dot(x, y)`. The rewrite is expressed with lazy [`Operator::Mul`] + /// arrays over the (much smaller) norm columns, so no denormalized coordinates are decoded. + /// + /// [`Normalized`]: crate::encodings::normalized::Normalized + fn reduce_encoded( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let lhs_ref = args.get(0)?; - let rhs_ref = args.get(1)?; - let len = args.row_count(); + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let len = args[0].len(); - // Take any Normalized read-through fast path that applies. - match NormalizedOrientation::classify(&lhs_ref, &rhs_ref) { + Ok(match NormalizedOrientation::classify(&args[0], &args[1]) { NormalizedOrientation::Both { lhs, rhs } => { - return self.execute_both_normalized(lhs, rhs, len, ctx); + let (normalized_l, norms_l) = extract_normalized_children(lhs); + let (normalized_r, norms_r) = extract_normalized_children(rhs); + let dot = + InnerProduct.try_new_array(len, EmptyOptions, [normalized_l, normalized_r])?; + Some( + dot.binary(norms_l, Operator::Mul)? + .binary(norms_r, Operator::Mul)?, + ) } NormalizedOrientation::One { normalized_array, plain, } => { - return self.execute_one_normalized(normalized_array, plain, len, ctx); + let (normalized, norms) = extract_normalized_children(normalized_array); + let dot = + InnerProduct.try_new_array(len, EmptyOptions, [normalized, plain.clone()])?; + Some(dot.binary(norms, Operator::Mul)?) } - NormalizedOrientation::Neither => {} - } - - // Compute combined validity. - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - // Canonicalize so we can perform the math directly. - let lhs: ExtensionArray = lhs_ref.execute(ctx)?; - let rhs: ExtensionArray = rhs_ref.execute(ctx)?; - - // We validated that both inputs have the same type. - let ext = lhs.dtype().as_extension(); - let tensor_match = ext - .metadata_opt::() - .vortex_expect("we already validated this in `return_dtype`"); - let dimensions = tensor_match.list_size() as usize; - - // Extract the storage array from each extension input. We pass the storage (FSL) rather - // than the extension array to avoid canonicalizing the extension wrapper. - let lhs_storage = lhs.storage_array(); - let rhs_storage = rhs.storage_array(); - - let lhs_flat = extract_flat_elements(lhs_storage, dimensions, ctx)?; - let rhs_flat = extract_flat_elements(rhs_storage, dimensions, ctx)?; - - match_each_float_ptype!(lhs_flat.ptype(), |T| { - let buffer: Buffer = (0..len) - .map(|i| inner_product_row(lhs_flat.row::(i), rhs_flat.row::(i))) - .collect(); - - // SAFETY: The buffer length equals `row_count`, which matches the source validity - // length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) + NormalizedOrientation::Neither => None, }) } - - fn validity( - &self, - _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if either input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false - } } impl ScalarFnArrayVTable for InnerProduct { @@ -205,72 +146,6 @@ impl ScalarFnArrayVTable for InnerProduct { } } -impl InnerProduct { - /// Both sides are [`Normalized`]-encoded: `inner_product = s_l * s_r * dot(n_l, n_r)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - fn execute_both_normalized( - &self, - lhs_ref: &ArrayRef, - rhs_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - - let (normalized_l, norms_l) = extract_normalized_children(lhs_ref); - let (normalized_r, norms_r) = extract_normalized_children(rhs_ref); - - let norms_l: PrimitiveArray = norms_l.execute(ctx)?; - let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r)? - .into_array() - .execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let nl = norms_l.as_slice::(); - let nr = norms_r.as_slice::(); - let buffer: Buffer = (0..len).map(|i| nl[i] * nr[i] * dots[i]).collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } - - /// One side is [`Normalized`]-encoded: `inner_product = s * dot(n, other)`. - /// - /// [`Normalized`]: crate::encodings::normalized::Normalized - /// - /// The caller must pass the [`Normalized`] array as `normalized_ref` and the plain array as `plain_ref`. - fn execute_one_normalized( - &self, - normalized_ref: &ArrayRef, - plain_ref: &ArrayRef, - len: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let validity = normalized_ref.validity()?.and(plain_ref.validity()?)?; - - let (normalized, norms) = extract_normalized_children(normalized_ref); - let normalized_norms: PrimitiveArray = norms.execute(ctx)?; - - let dot: PrimitiveArray = InnerProduct::try_new_array(normalized, plain_ref.clone())? - .into_array() - .execute(ctx)?; - - match_each_float_ptype!(dot.ptype(), |T| { - let dots = dot.as_slice::(); - let ns = normalized_norms.as_slice::(); - let buffer: Buffer = (0..len).map(|i| ns[i] * dots[i]).collect(); - - // SAFETY: The buffer length equals `len`, which matches the source validity length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) - }) - } -} - /// Computes the inner product (dot product) of two equal-length float slices. /// /// Returns `sum(a_i * b_i)`. @@ -280,254 +155,3 @@ fn inner_product_row(a: &[T], b: &[T]) -> T { .map(|(&x, &y)| x * y) .fold(T::zero(), |acc, v| acc + v) } - -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::encodings::normalized::Normalized; - use crate::scalar_fns::inner_product::InnerProduct; - use crate::tests::SESSION; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::normalized_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates inner product between two tensor arrays and returns the result as `Vec`. - fn eval_inner_product(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - /// Single-row inner product for various vector pairs. - #[rstest] - // Orthogonal: [1, 0] . [0, 1] = 0. - #[case::orthogonal(&[2], &[1.0, 0.0], &[0.0, 1.0], &[0.0])] - // Parallel: [3, 4] . [3, 4] = 9 + 16 = 25. - #[case::parallel(&[2], &[3.0, 4.0], &[3.0, 4.0], &[25.0])] - // Antiparallel: [1, 2] . [-1, -2] = -1 + -4 = -5. - #[case::antiparallel(&[2], &[1.0, 2.0], &[-1.0, -2.0], &[-5.0])] - // Scaled: [2, 0] . [3, 0] = 6. - #[case::scaled(&[2], &[2.0, 0.0], &[3.0, 0.0], &[6.0])] - fn single_row( - #[case] shape: &[usize], - #[case] lhs_elems: &[f64], - #[case] rhs_elems: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let lhs = tensor_array(shape, lhs_elems)?; - let rhs = tensor_array(shape, rhs_elems)?; - assert_close(&eval_inner_product(lhs, rhs)?, expected); - Ok(()) - } - - #[test] - fn multiple_rows() -> VortexResult<()> { - let lhs = tensor_array( - &[3], - &[ - 1.0, 0.0, 0.0, // tensor 0 - 3.0, 4.0, 0.0, // tensor 1 - 1.0, 1.0, 1.0, // tensor 2 - ], - )?; - let rhs = tensor_array( - &[3], - &[ - 0.0, 1.0, 0.0, // tensor 0: dot = 0 - 3.0, 4.0, 0.0, // tensor 1: dot = 25 - 2.0, 2.0, 2.0, // tensor 2: dot = 6 - ], - )?; - assert_close(&eval_inner_product(lhs, rhs)?, &[0.0, 25.0, 6.0]); - Ok(()) - } - - #[test] - fn vector_inner_product() -> VortexResult<()> { - let lhs = vector_array( - 2, - &[ - 3.0, 4.0, // vector 0 - 1.0, 0.0, // vector 1 - ], - )?; - let rhs = vector_array( - 2, - &[ - 3.0, 4.0, // vector 0: dot = 25 - 0.0, 1.0, // vector 1: dot = 0 - ], - )?; - assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 3 rows of dim-2 vectors. Row 1 of lhs is masked as null. - let lhs = tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; - let rhs = tensor_array(&[2], &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0])?; - let lhs = MaskedArray::try_new(lhs, Validity::from_iter([true, false, true]))?.into_array(); - - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: 1*7 + 2*8 = 23, row 1: null, row 2: 5*11 + 6*12 = 127. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert!(prim.is_valid(2, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[23.0]); - assert_close(&[prim.as_slice::()[2]], &[127.0]); - Ok(()) - } - - #[test] - fn rejects_non_extension_dtype() { - let lhs = PrimitiveArray::from_iter([1.0_f64, 2.0]).into_array(); - let rhs = PrimitiveArray::from_iter([3.0_f64, 4.0]).into_array(); - let result = InnerProduct::try_new_array(lhs, rhs); - assert!(result.is_err()); - } - - #[test] - fn rejects_mismatched_dtypes() -> VortexResult<()> { - let lhs = tensor_array(&[2], &[1.0_f64, 2.0])?; - let rhs = vector_array(2, &[3.0_f64, 4.0])?; - let result = InnerProduct::try_new_array(lhs, rhs); - assert!(result.is_err()); - Ok(()) - } - - #[test] - fn both_normalized() -> VortexResult<()> { - // LHS: [3.0, 4.0] = Normalized([0.6, 0.8], 5.0). - // RHS: [1.0, 0.0] = Normalized([1.0, 0.0], 1.0). - // dot([3.0, 4.0], [1.0, 0.0]) = 3.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[1.0, 0.0], &[1.0], &mut ctx)?; - - // Expected: 5.0 * 1.0 * dot([0.6, 0.8], [1.0, 0.0]) = 5.0 * 0.6 = 3.0. - assert_close(&eval_inner_product(lhs, rhs)?, &[3.0]); - Ok(()) - } - - #[test] - fn both_normalized_multiple_rows() -> VortexResult<()> { - // Row 0: [3.0, 4.0] dot [3.0, 4.0] = 25.0. - // Row 1: [1.0, 0.0] dot [0.0, 1.0] = 0.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - let rhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0], &mut ctx)?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_lhs() -> VortexResult<()> { - // LHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // RHS: plain [1.0, 2.0]. - // dot([3.0, 4.0], [1.0, 2.0]) = 3.0 + 8.0 = 11.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - let rhs = tensor_array(&[2], &[1.0, 2.0])?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); - Ok(()) - } - - #[test] - fn one_side_normalized_rhs() -> VortexResult<()> { - // LHS: plain [1.0, 2.0]. - // RHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. - // dot([1.0, 2.0], [3.0, 4.0]) = 3.0 + 8.0 = 11.0. - let mut ctx = SESSION.create_execution_ctx(); - let lhs = tensor_array(&[2], &[1.0, 2.0])?; - let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; - - assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); - Ok(()) - } - - #[test] - fn both_normalized_null_norms() -> VortexResult<()> { - // Row 0: valid, row 1: null (via nullable norms on lhs). - let normalized_l = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; - let norms_l = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); - let mut ctx = SESSION.create_execution_ctx(); - - let lhs = Normalized::try_new(normalized_l, norms_l, &mut ctx)?.into_array(); - let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - - let scalar_fn = InnerProduct::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: 5.0 * 5.0 * dot([0.6, 0.8], [0.6, 0.8]) = 25.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[25.0]); - Ok(()) - } - - #[rstest] - #[case::vector(inner_product_vector_lhs(), inner_product_vector_rhs())] - #[case::fixed_shape_tensor(inner_product_tensor_lhs(), inner_product_tensor_rhs())] - fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { - let original = InnerProduct::try_new_array(lhs.clone(), rhs.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(InnerProduct); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("InnerProduct serialize must produce metadata"); - - let children = vec![lhs, rhs]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) - } - - fn inner_product_vector_lhs() -> ArrayRef { - vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") - } - - fn inner_product_vector_rhs() -> ArrayRef { - vector_array(3, &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0]).expect("valid vector array") - } - - fn inner_product_tensor_lhs() -> ArrayRef { - tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0]).expect("valid tensor array") - } - - fn inner_product_tensor_rhs() -> ArrayRef { - tensor_array(&[2], &[5.0, 6.0, 7.0, 8.0]).expect("valid tensor array") - } -} diff --git a/vortex-tensor/src/scalar_fns/l2_norm.rs b/vortex-tensor/src/scalar_fns/l2_norm.rs index b7e9060ed3f..433a6527636 100644 --- a/vortex-tensor/src/scalar_fns/l2_norm.rs +++ b/vortex-tensor/src/scalar_fns/l2_norm.rs @@ -3,40 +3,23 @@ //! L2 norm expression for tensor-like types. -use num_traits::Float; use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::Constant; -use vortex_array::arrays::ConstantArray; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFn as ScalarFnArrayEncoding; -use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayView; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; -use vortex_array::dtype::NativePType; -use vortex_array::dtype::Nullability; use vortex_array::dtype::proto::dtype as pb; -use vortex_array::expr::Expression; -use vortex_array::expr::union_child_validities; use vortex_array::match_each_float_ptype; -use vortex_array::scalar::Scalar; -use vortex_array::scalar_fn::Arity; -use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ElementSink; use vortex_array::scalar_fn::EmptyOptions; -use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; use vortex_array::scalar_fn::ScalarFnId; -use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::scalar_fn::TypedScalarFnInstance; use vortex_array::serde::ArrayChildren; -use vortex_buffer::Buffer; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure_eq; use vortex_error::vortex_err; @@ -44,9 +27,10 @@ use vortex_session::VortexSession; use vortex_session::registry::CachedId; use crate::encodings::normalized::Normalized; -use crate::matcher::AnyTensor; -use crate::utils::extract_flat_elements; +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; use crate::utils::extract_normalized_children; +use crate::utils::l2_norm_row; use crate::utils::validate_tensor_float_input; /// L2 norm (Euclidean norm) of a tensor or vector column. @@ -62,139 +46,62 @@ use crate::utils::validate_tensor_float_input; /// of the storage contract, not a separate lossy-compute mode. /// /// [`Normalized`]: crate::encodings::normalized::Normalized -#[derive(Clone)] +#[derive(Clone, Debug, Default)] pub struct L2Norm; -impl L2Norm { - /// Creates a new [`TypedScalarFnInstance`] wrapping the L2 norm operation. - pub fn new() -> TypedScalarFnInstance { - TypedScalarFnInstance::new(L2Norm, EmptyOptions) - } - - /// Constructs a [`ScalarFnArray`] that lazily computes the L2 norm over `child`. - /// - /// # Errors - /// - /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype - /// mismatches). - pub fn try_new_array(child: ArrayRef) -> VortexResult { - ScalarFnArray::try_new(L2Norm::new().erased(), vec![child]) - } -} - -impl ScalarFnVTable for L2Norm { +impl RowFn for L2Norm { type Options = EmptyOptions; + const ARG_NAMES: &'static [&'static str] = &["input"]; + fn id(&self) -> ScalarFnId { static ID: CachedId = CachedId::new("vortex.tensor.l2_norm"); *ID } - fn arity(&self, _options: &Self::Options) -> Arity { - Arity::Exact(1) - } - - fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { - match child_idx { - 0 => ChildName::from("input"), - _ => unreachable!("L2Norm must have exactly one child"), - } + fn serialize(&self, _options: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) } - fn return_dtype(&self, _options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { - let input_dtype = &arg_dtypes[0]; - let tensor_match = validate_tensor_float_input(input_dtype)?; - let ptype = tensor_match.element_ptype(); - - let nullability = Nullability::from(input_dtype.is_nullable()); - Ok(DType::Primitive(ptype, nullability)) + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) } - fn execute( + fn dispatch( &self, _options: &Self::Options, - args: &dyn ExecutionArgs, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - let input_ref = args.get(0)?; - let row_count = args.row_count(); - - let ext = input_ref.dtype().as_extension(); - let tensor_match = ext - .metadata_opt::() - .vortex_expect("we already validated this in `return_dtype`"); - let tensor_flat_size = tensor_match.list_size() as usize; - let element_ptype = tensor_match.element_ptype(); - - let norm_dtype = DType::Primitive(element_ptype, ext.nullability()); - - // L2Norm over a `Normalized`-encoded column is defined to read back the authoritative stored - // norms. Callers of lossy encodings opt into that storage semantics instead of forcing a - // decode-and-recompute path here. - if input_ref.is::() { - let (_, norms) = extract_normalized_children(&input_ref); - vortex_ensure_eq!(norms.dtype(), &norm_dtype); - return Ok(norms); - } - - // Optimize for the constant array case. - if let Some(array) = input_ref.as_opt::() { - let scalar = array.scalar().as_extension().to_storage_scalar(); - - let Some(elements) = scalar.as_list().elements() else { - return Ok(ConstantArray::new(Scalar::null(norm_dtype), row_count).into_array()); - }; - - let norm_scalar = match_each_float_ptype!(element_ptype, |T| { - let values: Vec = elements - .iter() - .map(|s| { - s.as_primitive() - .as_::() - .vortex_expect("element was somehow not the correct float") - }) - .collect(); - let norm = l2_norm_row::(&values); - - Scalar::try_new(norm_dtype, Some(norm.into())) - })?; - - let norms = ConstantArray::new(norm_scalar, row_count).into_array(); - return Ok(norms); - } - - let input: ExtensionArray = input_ref.execute(ctx)?; - let validity = input.as_ref().validity()?; - - let storage = input.storage_array(); - let flat = extract_flat_elements(storage, tensor_flat_size, ctx)?; - - match_each_float_ptype!(flat.ptype(), |T| { - let buffer: Buffer = (0..row_count) - .map(|i| l2_norm_row(flat.row::(i))) - .collect(); - - // SAFETY: The buffer length equals `row_count`, which matches the source validity - // length. - Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow,), ElementSink, _, _>( + |_| (), + |&(), (row,), output| *output = l2_norm_row(row), + ) }) } - fn validity( + /// `L2Norm` over a [`Normalized`]-encoded column is defined to read back the authoritative + /// stored norms. Callers of lossy encodings opt into that storage semantics instead of forcing + /// a decode-and-recompute path here. + fn reduce_encoded( &self, _options: &Self::Options, - expression: &Expression, - ) -> VortexResult> { - // The result is null if the input tensor is null. - union_child_validities(expression) - } - - fn is_strict(&self, _options: &Self::Options) -> bool { - true - } - - fn is_fallible(&self, _options: &Self::Options) -> bool { - false + args: &[ArrayRef], + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let input = &args[0]; + if !input.is::() { + return Ok(None); + } + let element_ptype = validate_tensor_float_input(input.dtype())?.element_ptype(); + let (_, norms) = extract_normalized_children(input); + vortex_ensure_eq!(norms.dtype().as_ptype(), element_ptype); + Ok(Some(norms)) } } @@ -240,206 +147,3 @@ impl ScalarFnArrayVTable for L2Norm { }) } } - -/// Computes the L2 norm (Euclidean norm) of a float slice. -/// -/// Returns `sqrt(sum(v_i^2))`. A zero-length or all-zero input produces `0.0`. -fn l2_norm_row(v: &[T]) -> T { - let mut sum_sq = T::zero(); - for &x in v { - sum_sq = sum_sq + x * x; - } - sum_sq.sqrt() -} - -#[cfg(test)] -mod tests { - - use rstest::rstest; - use vortex_array::ArrayPlugin; - use vortex_array::ArrayRef; - use vortex_array::EmptyMetadata; - use vortex_array::IntoArray; - use vortex_array::VortexSessionExecute; - use vortex_array::arrays::Constant; - use vortex_array::arrays::ConstantArray; - use vortex_array::arrays::MaskedArray; - use vortex_array::arrays::PrimitiveArray; - use vortex_array::arrays::ScalarFnArray; - use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; - use vortex_array::dtype::DType; - use vortex_array::dtype::Nullability; - use vortex_array::dtype::PType; - use vortex_array::dtype::extension::ExtDType; - use vortex_array::scalar::Scalar; - use vortex_array::validity::Validity; - use vortex_error::VortexResult; - - use crate::scalar_fns::l2_norm::L2Norm; - use crate::tests::SESSION; - use crate::types::vector::Vector; - use crate::utils::test_helpers::assert_close; - use crate::utils::test_helpers::literal_vector_array; - use crate::utils::test_helpers::tensor_array; - use crate::utils::test_helpers::vector_array; - - /// Evaluates L2 norm on a tensor/vector array and returns the result as `Vec`. - fn eval_l2_norm(input: ArrayRef) -> VortexResult> { - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - Ok(prim.as_slice::().to_vec()) - } - - #[rstest] - #[case::three_four_five(&[2], &[3.0, 4.0], &[5.0])] - #[case::zero_vector(&[3], &[0.0, 0.0, 0.0], &[0.0])] - #[case::single_element(&[1], &[7.0], &[7.0])] - #[case::negative_elements(&[2], &[-3.0, -4.0], &[5.0])] - fn known_norms( - #[case] shape: &[usize], - #[case] elements: &[f64], - #[case] expected: &[f64], - ) -> VortexResult<()> { - let arr = tensor_array(shape, elements)?; - assert_close(&eval_l2_norm(arr)?, expected); - Ok(()) - } - - #[test] - fn multiple_rows() -> VortexResult<()> { - let arr = tensor_array( - &[3], - &[ - 3.0, 4.0, 0.0, // norm = 5.0 - 0.0, 0.0, 0.0, // norm = 0.0 - 1.0, 1.0, 1.0, // norm = sqrt(3) - ], - )?; - assert_close(&eval_l2_norm(arr)?, &[5.0, 0.0, 3.0_f64.sqrt()]); - Ok(()) - } - - #[test] - fn vector_multiple_rows() -> VortexResult<()> { - let arr = vector_array( - 3, - &[ - 1.0, 0.0, 0.0, // norm = 1.0 - 3.0, 4.0, 0.0, // norm = 5.0 - ], - )?; - assert_close(&eval_l2_norm(arr)?, &[1.0, 5.0]); - Ok(()) - } - - #[test] - fn null_input_row() -> VortexResult<()> { - // 2 rows of dim-2 vectors. Row 1 is masked as null. - let arr = tensor_array(&[2], &[3.0, 4.0, 0.0, 0.0])?; - let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![arr])?; - let mut ctx = SESSION.create_execution_ctx(); - let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - - // Row 0: norm = 5.0, row 1: null. - assert!(prim.is_valid(0, &mut ctx)?); - assert!(!prim.is_valid(1, &mut ctx)?); - assert_close(&[prim.as_slice::()[0]], &[5.0]); - Ok(()) - } - - /// A constant input whose scalar is a non-null tensor should short-circuit to a - /// [`ConstantArray`] output whose scalar is the precomputed norm. Uses [`execute_until`] so - /// execution stops at the [`Constant`] encoding instead of canonicalizing into a - /// [`PrimitiveArray`]. - #[test] - fn constant_non_null_input_yields_constant_output() -> VortexResult<()> { - let input = literal_vector_array(&[3.0f64, 4.0], 4); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); - let mut ctx = SESSION.create_execution_ctx(); - let output = result.execute_until::(&mut ctx)?; - - let constant = output - .as_opt::() - .expect("L2Norm over a constant input must produce a constant output"); - assert_eq!(constant.len(), 4); - let norm = constant - .scalar() - .as_primitive() - .as_::() - .expect("norm scalar must be a non-null primitive"); - assert_close(&[norm], &[5.0]); - Ok(()) - } - - /// A constant input whose scalar is null should short-circuit to a null [`ConstantArray`] of - /// the correct primitive dtype and length. - #[test] - fn constant_null_input_yields_null_constant_output() -> VortexResult<()> { - let storage_dtype = DType::FixedSizeList( - DType::Primitive(PType::F64, Nullability::NonNullable).into(), - 2, - Nullability::Nullable, - ); - let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); - let null_scalar = Scalar::null(DType::Extension(ext_dtype)); - let input = ConstantArray::new(null_scalar, 3).into_array(); - - let scalar_fn = L2Norm::new().erased(); - let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); - let mut ctx = SESSION.create_execution_ctx(); - let output = result.execute_until::(&mut ctx)?; - - let constant = output - .as_opt::() - .expect("null constant input must produce a constant output"); - assert_eq!(constant.len(), 3); - assert!(constant.scalar().is_null()); - assert_eq!( - constant.dtype(), - &DType::Primitive(PType::F64, Nullability::Nullable) - ); - Ok(()) - } - - #[rstest] - #[case::fixed_shape_tensor(l2_norm_tensor_child())] - #[case::vector(l2_norm_vector_child())] - fn serde_round_trip(#[case] child: ArrayRef) -> VortexResult<()> { - let original = L2Norm::try_new_array(child.clone())?.into_array(); - - let plugin = ScalarFnArrayPlugin::new(L2Norm); - let metadata = plugin - .serialize(&original, &SESSION)? - .expect("L2Norm serialize must produce metadata"); - - let children = vec![child]; - let recovered = plugin.deserialize( - original.dtype(), - original.len(), - &metadata, - &[], - &children, - &SESSION, - )?; - - assert_eq!(recovered.dtype(), original.dtype()); - assert_eq!(recovered.len(), original.len()); - assert_eq!(recovered.encoding_id(), original.encoding_id()); - Ok(()) - } - - fn l2_norm_tensor_child() -> ArrayRef { - tensor_array(&[3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid tensor array") - } - - fn l2_norm_vector_child() -> ArrayRef { - vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") - } -} diff --git a/vortex-tensor/src/scalar_fns/mod.rs b/vortex-tensor/src/scalar_fns/mod.rs index 68f10ca6b01..706392d3b25 100644 --- a/vortex-tensor/src/scalar_fns/mod.rs +++ b/vortex-tensor/src/scalar_fns/mod.rs @@ -6,3 +6,7 @@ pub mod cosine_similarity; pub mod inner_product; pub mod l2_norm; +pub mod row; + +#[cfg(test)] +mod tests; diff --git a/vortex-tensor/src/scalar_fns/row.rs b/vortex-tensor/src/scalar_fns/row.rs new file mode 100644 index 00000000000..3c02a1d2615 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/row.rs @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! What the tensor scalar functions add to the row-function machinery: an element type that reads a +//! tensor row and the width rule they share. + +use std::marker::PhantomData; + +use num_traits::Float; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::PType; +use vortex_array::scalar_fn::InputElement; +use vortex_buffer::Buffer; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; + +use crate::utils::extract_flat_elements; +use crate::utils::validate_tensor_float_input; +use crate::utils::validate_tensor_float_inputs; + +/// The width rule the tensor scalar functions share: every argument is the same float tensor dtype, +/// and the width is its element ptype. +pub(crate) fn tensor_element_ptype(args: &[DType]) -> VortexResult { + Ok(validate_tensor_float_inputs(args)?.element_ptype()) +} + +/// Marker for tensor-valued input elements: accepts any tensor-like extension column whose +/// elements are `T`, and presents each row as its flat elements, `&[T]`. +pub struct TensorRow(PhantomData); + +/// The decoded form of a [`TensorRow`] column: one flat typed buffer plus the stride to read it at. +/// +/// Typed at decode time rather than per row. `FlatElements::row` re-derives its typed slice on every +/// call, which costs a ptype check and a buffer downcast per row; a row loop reads every row, so it +/// pays that once here instead. +pub struct TensorRows { + /// Every row's elements, back to back. + elements: Buffer, + + /// Number of logical tensor rows, stored so zero-width tensors retain their length. + rows: usize, + + /// Elements per row, the length of each row slice. + list_size: usize, + + /// `list_size` for a full column and `0` for constant-backed storage, so `index * stride` pins a + /// constant to its single materialized row without a branch in the loop. + stride: usize, +} + +impl InputElement for TensorRow { + type Column = TensorRows; + type Varying<'a> = &'a TensorRows; + type Elem<'a> = &'a [T]; + + // Tensor storage is a fully materialized non-nullable primitive buffer, so the elements behind + // a null row are arbitrary values rather than an unresolvable reference. + const DENSE_SAFE: bool = true; + // Tensor storage is a primitive buffer; reading it cannot fail on account of its values. + const DECODE_FALLIBLE: bool = false; + + fn validate(dtype: &DType) -> VortexResult<()> { + let tensor_match = validate_tensor_float_input(dtype)?; + let expected = T::PTYPE; + vortex_ensure_eq!( + tensor_match.element_ptype(), + expected, + "expected a tensor of {expected} elements, got {dtype}", + ); + Ok(()) + } + + fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let rows = array.len(); + let list_size = validate_tensor_float_input(array.dtype())?.list_size() as usize; + let ext: ExtensionArray = array.execute(ctx)?; + let flat = extract_flat_elements(ext.storage_array(), list_size, ctx)?; + + Ok(TensorRows { + rows, + list_size: flat.list_size(), + stride: flat.row_stride(), + elements: flat.into_buffer::(), + }) + } + + fn get(column: &Self::Column, index: usize) -> &[T] { + let start = index * column.stride; + &column.elements.as_slice()[start..start + column.list_size] + } + + fn varying(column: &Self::Column) -> Self::Varying<'_> { + column + } + + fn varying_len(column: &Self::Varying<'_>) -> usize { + column.rows + } + + fn get_varying<'a>(column: &Self::Varying<'a>, index: usize) -> &'a [T] + where + Self: 'a, + { + Self::get(column, index) + } +} + +/// Test-only probe recording which operands the last `prepare` step saw as batch-constant, so a +/// test can assert its inputs took the stride-0 decode path rather than merely producing the right +/// values through the varying path. +#[cfg(test)] +pub(crate) mod probe { + use std::cell::Cell; + + thread_local! { + /// Bitmask of the constant operands the last `prepare` saw (bit 0 for the lhs, bit 1 for + /// the rhs). Thread-local rather than a process global so concurrent tests in one process + /// (plain `cargo test`) cannot race it; execution runs on the calling thread. + pub(crate) static SEEN_CONSTANTS: Cell = const { Cell::new(u8::MAX) }; + } + + /// Record which operands `prepare` saw as constant. + pub(crate) fn record(lhs_constant: bool, rhs_constant: bool) { + SEEN_CONSTANTS.set(u8::from(lhs_constant) | (u8::from(rhs_constant) << 1)); + } +} diff --git a/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs b/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs new file mode 100644 index 00000000000..60e75792109 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/cosine_similarity.rs @@ -0,0 +1,586 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::assert_arrays_eq; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::cosine_similarity::CosineSimilarity; +use crate::scalar_fns::row::probe; +use crate::tests::SESSION; +use crate::types::vector::Vector; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::constant_tensor_array; +use crate::utils::test_helpers::literal_vector_array; +use crate::utils::test_helpers::normalized_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates cosine similarity between two tensor arrays and returns the result as `Vec`. +fn eval_cosine_similarity(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +/// Like [`eval_cosine_similarity`], but returns the executed array for exact array comparisons. +fn eval_cosine_similarity_array( + lhs: ArrayRef, + rhs: ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + Ok(result + .into_array() + .execute::(ctx)? + .into_array()) +} + +#[test] +fn unit_vectors_1d() -> VortexResult<()> { + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // Tensor 1 + 0.0, 1.0, 0.0, // Tensor 2 + ], + )?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // Tensor 1 + 1.0, 0.0, 0.0, // Tensor 2 + ], + )?; + + // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +/// Single-row cosine similarity for various vector pairs. +#[rstest] +// Antiparallel -> -1.0. +#[case::opposite(&[3], &[1.0, 0.0, 0.0], &[-1.0, 0.0, 0.0], &[-1.0])] +// dot=24, both magnitudes=5 -> 24/25 = 0.96. +#[case::non_unit(&[2], &[3.0, 4.0], &[4.0, 3.0], &[0.96])] +// Zero vector -> guarded to 0.0. +#[case::zero_norm(&[2], &[0.0, 0.0], &[1.0, 0.0], &[0.0])] +fn single_row( + #[case] shape: &[usize], + #[case] lhs_elems: &[f64], + #[case] rhs_elems: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let lhs = tensor_array(shape, lhs_elems)?; + let rhs = tensor_array(shape, rhs_elems)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, expected); + Ok(()) +} + +/// Self-similarity across various tensor shapes should always produce 1.0. +#[rstest] +// 2x3 matrix, flattened to 6 elements. +#[case::matrix_2d( + &[2, 3], + &[ + 1.0, 0.0, 0.0, // row 0 + 0.0, 0.0, 0.0, // row 1 + ], +)] +// 2x2x2 tensor, 8 elements. +#[case::tensor_3d(&[2, 2, 2], &[1.0; 8])] +fn self_similarity(#[case] shape: &[usize], #[case] elements: &[f64]) -> VortexResult<()> { + let lhs = tensor_array(shape, elements)?; + let rhs = tensor_array(shape, elements)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); + Ok(()) +} + +#[test] +fn scalar_0d() -> VortexResult<()> { + // 0-dimensional tensor: each "tensor" is a single scalar value. + let lhs = tensor_array(&[], &[5.0, 3.0])?; + let rhs = tensor_array(&[], &[5.0, -3.0])?; + + // Same sign -> 1.0, opposite sign -> -1.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, -1.0]); + Ok(()) +} + +#[test] +fn many_rows() -> VortexResult<()> { + // 5 tensors of shape [4] compared against themselves -> all 1.0. + let lhs = tensor_array( + &[4], + &[ + 1.0, 2.0, 3.0, 4.0, // tensor 0 + 0.0, 1.0, 0.0, 0.0, // tensor 1 + 5.0, 0.0, 5.0, 0.0, // tensor 2 + 1.0, 1.0, 1.0, 1.0, // tensor 3 + 0.0, 0.0, 0.0, 7.0, // tensor 4 + ], + )?; + let rhs = lhs.clone(); + + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0, 1.0, 1.0, 1.0, 1.0], + ); + Ok(()) +} + +#[test] +fn constant_query_tensor() -> VortexResult<()> { + // Compare 4 tensors of shape [3] against a single constant query tensor [1,0,0]. + let data = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // tensor 0 + 0.0, 1.0, 0.0, // tensor 1 + 0.0, 0.0, 1.0, // tensor 2 + 1.0, 0.0, 0.0, // tensor 3 + ], + )?; + let query = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 4)?; + + assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); + Ok(()) +} + +#[test] +fn vector_unit_vectors() -> VortexResult<()> { + let lhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 0.0, 1.0, 0.0, // vector 1 + ], + )?; + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 1.0, 0.0, 0.0, // vector 1 + ], + )?; + + // Row 0: identical -> 1.0, row 1: orthogonal -> 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +#[test] +fn vector_constant_query() -> VortexResult<()> { + let data = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // vector 0 + 0.0, 1.0, 0.0, // vector 1 + 0.0, 0.0, 1.0, // vector 2 + 1.0, 0.0, 0.0, // vector 3 + ], + )?; + let query = Vector::constant_array(&[1.0, 0.0, 0.0], 4)?; + + assert_close(&eval_cosine_similarity(data, query)?, &[1.0, 0.0, 0.0, 1.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 2 rows of dim-2 vectors. Row 1 of rhs is masked as null. + let lhs = tensor_array(&[2], &[3.0, 4.0, 1.0, 0.0])?; + let rhs = tensor_array(&[2], &[3.0, 4.0, 0.0, 1.0])?; + let rhs = MaskedArray::try_new(rhs, Validity::from_iter([true, false]))?.into_array(); + + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: self-similarity = 1.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[1.0]); + Ok(()) +} + +#[test] +fn both_normalized_self_similarity() -> VortexResult<()> { + // [3.0, 4.0] has norm 5.0, normalized [0.6, 0.8]. + // [1.0, 0.0] has norm 1.0, normalized [1.0, 0.0]. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + // Self-similarity should always be 1.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 1.0]); + Ok(()) +} + +#[test] +fn both_normalized_orthogonal() -> VortexResult<()> { + // [3.0, 0.0] normalized [1.0, 0.0], norm 3.0. + // [0.0, 4.0] normalized [0.0, 1.0], norm 4.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[1.0, 0.0], &[3.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.0, 1.0], &[4.0], &mut ctx)?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); + Ok(()) +} + +#[test] +fn both_normalized_zero_norm() -> VortexResult<()> { + // Zero-norm row: normalized is [0.0, 0.0], norm is 0.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + // Row 0: dot([0.6, 0.8], [0.6, 0.8]) = 1.0, row 1: dot([0,0], [1,0]) = 0.0. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0, 0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lhs() -> VortexResult<()> { + // LHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // RHS is plain [3.0, 4.0]. + // cosine_similarity([3.0, 4.0], [3.0, 4.0]) = 1.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = tensor_array(&[2], &[3.0, 4.0])?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_rhs() -> VortexResult<()> { + // LHS is plain [1.0, 0.0], RHS is Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // cosine_similarity([1.0, 0.0], [3.0, 4.0]) = 3.0 / (1.0 * 5.0) = 0.6. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = tensor_array(&[2], &[1.0, 0.0])?; + let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.6]); + Ok(()) +} + +#[test] +fn both_normalized_null_norms() -> VortexResult<()> { + // Row 0: valid, row 1: null (via nullable norms on rhs). + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + let normalized_r = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms_r = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let rhs = Normalized::try_new(normalized_r, norms_r, &mut ctx)?.into_array(); + + let scalar_fn = CosineSimilarity.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[1.0]); + Ok(()) +} + +#[test] +fn both_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. With a stored norm of `0.0`, cosine + // similarity for that row must be `0.0` even though the dot product of the normalized + // children is nonzero. + let normalized_l = tensor_array(&[2], &[0.6, 0.8])?; + let norms_l = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally violates the unit-norm invariant by + // pairing a nonzero normalized row with a stored norm of `0.0`, mimicking lossy storage. + let lhs = unsafe { Normalized::new_unchecked(normalized_l, norms_l) }.into_array(); + + let normalized_r = tensor_array(&[2], &[0.6, 0.8])?; + let norms_r = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: Same as above for the rhs operand. + let rhs = unsafe { Normalized::new_unchecked(normalized_r, norms_r) }.into_array(); + + // `dot(normalized_l, normalized_r) = 1.0`, but the authoritative stored norms are both + // `0.0`, so cosine similarity must be `0.0`. + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. The plain side is a normal nonzero + // tensor with positive norm. cosine similarity must still be `0.0` because the + // authoritative stored norm on the denorm side is `0.0`. + let normalized = tensor_array(&[2], &[0.6, 0.8])?; + let norms = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally pairs a nonzero normalized row with a + // stored norm of `0.0`, mimicking lossy storage where the stored norm is authoritative. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + let plain = tensor_array(&[2], &[1.0, 0.0])?; + + // Denorm on the lhs: `One { denorm: lhs, plain: rhs }`. + assert_close( + &eval_cosine_similarity(denorm.clone(), plain.clone())?, + &[0.0], + ); + + // Denorm on the rhs: `One { denorm: rhs, plain: lhs }`. The same zero-norm guard must + // fire regardless of operand order. + assert_close(&eval_cosine_similarity(plain, denorm)?, &[0.0]); + Ok(()) +} + +#[test] +fn constant_lhs_matches_plain_tensor() -> VortexResult<()> { + // The constant query `[1, 2, 2]` has norm 3, so its normalized form is `[1/3, 2/3, 2/3]`. + // Expected cosine similarity against each row is `dot([1, 2, 2], row) / (3 * ||row||)`. + let lhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // dot=1, ||rhs||=1, expected=1/3 + 1.0, 2.0, 2.0, // dot=9, ||rhs||=3, expected=1 + 0.0, 0.0, 1.0, // dot=2, ||rhs||=1, expected=2/3 + 2.0, 1.0, 2.0, // dot=8, ||rhs||=3, expected=8/9 + ], + )?; + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) +} + +#[test] +fn constant_rhs_matches_plain_tensor() -> VortexResult<()> { + // Mirror of `constant_lhs_matches_plain_tensor` with the constant on the right. + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let rhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) +} + +#[test] +fn both_constant_tensors() -> VortexResult<()> { + // `[1, 0, 0]` vs `[1, 1, 0]`. dot=1, ||lhs||=1, ||rhs||=sqrt(2), expected=1/sqrt(2). + let lhs = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 3)?; + let rhs = constant_tensor_array(&[3], &[1.0, 1.0, 0.0], 3)?; + let expected = 1.0 / 2.0_f64.sqrt(); + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[expected, expected, expected], + ); + Ok(()) +} + +#[test] +fn constant_zero_norm_query() -> VortexResult<()> { + // A zero-norm constant query must produce `0.0` for every row via the zero-norm guard in + // `cosine_one_normalized` and `execute_both_normalized`. + let lhs = constant_tensor_array(&[3], &[0.0, 0.0, 0.0], 3)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 2.0, 3.0, // + 4.0, 5.0, 6.0, // + 7.0, 8.0, 9.0, // + ], + )?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0, 0.0]); + Ok(()) +} + +#[test] +fn constant_self_similarity_nonunit() -> VortexResult<()> { + // A non-unit constant query compared to itself must produce `1.0`. This exercises the + // helper's division: after normalization, both sides must be exactly unit so the + // Normalized fast path's inner product yields 1. + let lhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + let rhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[1.0; 5]); + Ok(()) +} + +/// An extension array over constant storage (what [`Vector::constant_array`] builds) is a batch +/// constant like any other: the row layer sees through the wrapper, so `prepare` hoists its norm +/// exactly as it does for the literal shape. This used to be intercepted by a hand-written +/// `reduce_encoded` rewrite into `Normalized`, deleted in favor of the framework path. +#[test] +fn vector_constant_matches_plain() -> VortexResult<()> { + let lhs = Vector::constant_array(&[1.0, 2.0, 2.0], 4)?; + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + + assert_close( + &eval_cosine_similarity(lhs, rhs)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + assert_eq!( + probe::SEEN_CONSTANTS.get(), + 0b01, + "the extension-over-constant lhs must reach prepare as a batch constant", + ); + Ok(()) +} + +/// The literal-constant shape (a [`ConstantArray`] over a [`Vector`] extension scalar, what a +/// `lit(query)` expression produces) reaches the row loop, unlike an extension-wrapped constant, +/// which `reduce_encoded` rewrites into `Normalized`. There the prepared kernel hoists the query's +/// norm once per batch, and the result must be exactly the result of expanding the same query +/// into a full column, which hoists nothing. +/// +/// [`ConstantArray`]: vortex_array::arrays::ConstantArray +#[test] +fn literal_constant_rhs_matches_expanded_column() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); + let lhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let query = [1.0, 2.0, 2.0]; + + let from_constant = + eval_cosine_similarity_array(lhs.clone(), literal_vector_array(&query, 4), &mut ctx)?; + let from_expanded = + eval_cosine_similarity_array(lhs, vector_array(3, &query.repeat(4))?, &mut ctx)?; + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// The mirror of [`literal_constant_rhs_matches_expanded_column`], exercising the hoisted-lhs arm +/// of the prepared kernel. +#[test] +fn literal_constant_lhs_matches_expanded_column() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let query = [1.0, 2.0, 2.0]; + + let from_constant = + eval_cosine_similarity_array(literal_vector_array(&query, 4), rhs.clone(), &mut ctx)?; + let from_expanded = + eval_cosine_similarity_array(vector_array(3, &query.repeat(4))?, rhs, &mut ctx)?; + + assert_arrays_eq!(from_constant, from_expanded, &mut ctx); + Ok(()) +} + +/// A zero-norm literal constant query must be guarded to `0.0` on every row by the prepared row +/// kernel, exactly as the unprepared kernel guards it. +#[test] +fn literal_constant_zero_norm_query_yields_zero() -> VortexResult<()> { + let lhs = vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; + let rhs = literal_vector_array(&[0.0f64, 0.0, 0.0], 2); + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[0.0, 0.0]); + Ok(()) +} + +/// Two literal constants are folded to a single-row execution by the row lifting, and that row +/// still runs the prepared kernel with both norms hoisted. +#[test] +fn both_literal_constants() -> VortexResult<()> { + let lhs = literal_vector_array(&[1.0f64, 0.0, 0.0], 3); + let rhs = literal_vector_array(&[1.0f64, 1.0, 0.0], 3); + let expected = 1.0 / 2.0_f64.sqrt(); + assert_close(&eval_cosine_similarity(lhs, rhs)?, &[expected; 3]); + Ok(()) +} + +#[rstest] +#[case::vector(cosine_vector_lhs(), cosine_vector_rhs())] +#[case::fixed_shape_tensor(cosine_tensor_lhs(), cosine_tensor_rhs())] +fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { + let original = + CosineSimilarity.try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(CosineSimilarity); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("CosineSimilarity serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn cosine_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 0.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +} + +fn cosine_vector_rhs() -> ArrayRef { + vector_array(3, &[0.0, 1.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") +} + +fn cosine_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 0.0, 3.0, 4.0]).expect("valid tensor array") +} + +fn cosine_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[0.0, 1.0, 3.0, 4.0]).expect("valid tensor array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/inner_product.rs b/vortex-tensor/src/scalar_fns/tests/inner_product.rs new file mode 100644 index 00000000000..af7fbb7bc1a --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/inner_product.rs @@ -0,0 +1,253 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::inner_product::InnerProduct; +use crate::tests::SESSION; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::normalized_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates inner product between two tensor arrays and returns the result as `Vec`. +fn eval_inner_product(lhs: ArrayRef, rhs: ArrayRef) -> VortexResult> { + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +/// Single-row inner product for various vector pairs. +#[rstest] +// Orthogonal: [1, 0] . [0, 1] = 0. +#[case::orthogonal(&[2], &[1.0, 0.0], &[0.0, 1.0], &[0.0])] +// Parallel: [3, 4] . [3, 4] = 9 + 16 = 25. +#[case::parallel(&[2], &[3.0, 4.0], &[3.0, 4.0], &[25.0])] +// Antiparallel: [1, 2] . [-1, -2] = -1 + -4 = -5. +#[case::antiparallel(&[2], &[1.0, 2.0], &[-1.0, -2.0], &[-5.0])] +// Scaled: [2, 0] . [3, 0] = 6. +#[case::scaled(&[2], &[2.0, 0.0], &[3.0, 0.0], &[6.0])] +fn single_row( + #[case] shape: &[usize], + #[case] lhs_elems: &[f64], + #[case] rhs_elems: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let lhs = tensor_array(shape, lhs_elems)?; + let rhs = tensor_array(shape, rhs_elems)?; + assert_close(&eval_inner_product(lhs, rhs)?, expected); + Ok(()) +} + +#[test] +fn multiple_rows() -> VortexResult<()> { + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // tensor 0 + 3.0, 4.0, 0.0, // tensor 1 + 1.0, 1.0, 1.0, // tensor 2 + ], + )?; + let rhs = tensor_array( + &[3], + &[ + 0.0, 1.0, 0.0, // tensor 0: dot = 0 + 3.0, 4.0, 0.0, // tensor 1: dot = 25 + 2.0, 2.0, 2.0, // tensor 2: dot = 6 + ], + )?; + assert_close(&eval_inner_product(lhs, rhs)?, &[0.0, 25.0, 6.0]); + Ok(()) +} + +#[test] +fn vector_inner_product() -> VortexResult<()> { + let lhs = vector_array( + 2, + &[ + 3.0, 4.0, // vector 0 + 1.0, 0.0, // vector 1 + ], + )?; + let rhs = vector_array( + 2, + &[ + 3.0, 4.0, // vector 0: dot = 25 + 0.0, 1.0, // vector 1: dot = 0 + ], + )?; + assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 3 rows of dim-2 vectors. Row 1 of lhs is masked as null. + let lhs = tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?; + let rhs = tensor_array(&[2], &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0])?; + let lhs = MaskedArray::try_new(lhs, Validity::from_iter([true, false, true]))?.into_array(); + + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: 1*7 + 2*8 = 23, row 1: null, row 2: 5*11 + 6*12 = 127. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert!(prim.is_valid(2, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[23.0]); + assert_close(&[prim.as_slice::()[2]], &[127.0]); + Ok(()) +} + +#[test] +fn rejects_non_extension_dtype() { + let lhs = PrimitiveArray::from_iter([1.0_f64, 2.0]).into_array(); + let rhs = PrimitiveArray::from_iter([3.0_f64, 4.0]).into_array(); + let result = InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs, rhs]); + assert!(result.is_err()); +} + +#[test] +fn rejects_mismatched_dtypes() -> VortexResult<()> { + let lhs = tensor_array(&[2], &[1.0_f64, 2.0])?; + let rhs = vector_array(2, &[3.0_f64, 4.0])?; + let result = InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs, rhs]); + assert!(result.is_err()); + Ok(()) +} + +#[test] +fn both_normalized() -> VortexResult<()> { + // LHS: [3.0, 4.0] = Normalized([0.6, 0.8], 5.0). + // RHS: [1.0, 0.0] = Normalized([1.0, 0.0], 1.0). + // dot([3.0, 4.0], [1.0, 0.0]) = 3.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[1.0, 0.0], &[1.0], &mut ctx)?; + + // Expected: 5.0 * 1.0 * dot([0.6, 0.8], [1.0, 0.0]) = 5.0 * 0.6 = 3.0. + assert_close(&eval_inner_product(lhs, rhs)?, &[3.0]); + Ok(()) +} + +#[test] +fn both_normalized_multiple_rows() -> VortexResult<()> { + // Row 0: [3.0, 4.0] dot [3.0, 4.0] = 25.0. + // Row 1: [1.0, 0.0] dot [0.0, 1.0] = 0.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = normalized_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0], &mut ctx)?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[25.0, 0.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_lhs() -> VortexResult<()> { + // LHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // RHS: plain [1.0, 2.0]. + // dot([3.0, 4.0], [1.0, 2.0]) = 3.0 + 8.0 = 11.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = tensor_array(&[2], &[1.0, 2.0])?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); + Ok(()) +} + +#[test] +fn one_side_normalized_rhs() -> VortexResult<()> { + // LHS: plain [1.0, 2.0]. + // RHS: Normalized([0.6, 0.8], 5.0) representing [3.0, 4.0]. + // dot([1.0, 2.0], [3.0, 4.0]) = 3.0 + 8.0 = 11.0. + let mut ctx = SESSION.create_execution_ctx(); + let lhs = tensor_array(&[2], &[1.0, 2.0])?; + let rhs = normalized_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + + assert_close(&eval_inner_product(lhs, rhs)?, &[11.0]); + Ok(()) +} + +#[test] +fn both_normalized_null_norms() -> VortexResult<()> { + // Row 0: valid, row 1: null (via nullable norms on lhs). + let normalized_l = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms_l = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + + let lhs = Normalized::try_new(normalized_l, norms_l, &mut ctx)?.into_array(); + let rhs = normalized_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + + let scalar_fn = InnerProduct.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: 5.0 * 5.0 * dot([0.6, 0.8], [0.6, 0.8]) = 25.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[25.0]); + Ok(()) +} + +#[rstest] +#[case::vector(inner_product_vector_lhs(), inner_product_vector_rhs())] +#[case::fixed_shape_tensor(inner_product_tensor_lhs(), inner_product_tensor_rhs())] +fn serde_round_trip(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) -> VortexResult<()> { + let original = + InnerProduct.try_new_array(lhs.len(), EmptyOptions, [lhs.clone(), rhs.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(InnerProduct); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("InnerProduct serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn inner_product_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") +} + +fn inner_product_vector_rhs() -> ArrayRef { + vector_array(3, &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0]).expect("valid vector array") +} + +fn inner_product_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0]).expect("valid tensor array") +} + +fn inner_product_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[5.0, 6.0, 7.0, 8.0]).expect("valid tensor array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/l2_norm.rs b/vortex-tensor/src/scalar_fns/tests/l2_norm.rs new file mode 100644 index 00000000000..a9fda0326d8 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/l2_norm.rs @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::EmptyMetadata; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::Constant; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ScalarFnVTableExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use crate::encodings::normalized::Normalized; +use crate::scalar_fns::l2_norm::L2Norm; +use crate::tests::SESSION; +use crate::types::vector::Vector; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::literal_vector_array; +use crate::utils::test_helpers::tensor_array; +use crate::utils::test_helpers::vector_array; + +/// Evaluates L2 norm on a tensor/vector array and returns the result as `Vec`. +fn eval_l2_norm(input: ArrayRef) -> VortexResult> { + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) +} + +#[rstest] +#[case::three_four_five(&[2], &[3.0, 4.0], &[5.0])] +#[case::zero_vector(&[3], &[0.0, 0.0, 0.0], &[0.0])] +#[case::single_element(&[1], &[7.0], &[7.0])] +#[case::negative_elements(&[2], &[-3.0, -4.0], &[5.0])] +fn known_norms( + #[case] shape: &[usize], + #[case] elements: &[f64], + #[case] expected: &[f64], +) -> VortexResult<()> { + let arr = tensor_array(shape, elements)?; + assert_close(&eval_l2_norm(arr)?, expected); + Ok(()) +} + +#[test] +fn multiple_rows() -> VortexResult<()> { + let arr = tensor_array( + &[3], + &[ + 3.0, 4.0, 0.0, // norm = 5.0 + 0.0, 0.0, 0.0, // norm = 0.0 + 1.0, 1.0, 1.0, // norm = sqrt(3) + ], + )?; + assert_close(&eval_l2_norm(arr)?, &[5.0, 0.0, 3.0_f64.sqrt()]); + Ok(()) +} + +#[test] +fn vector_multiple_rows() -> VortexResult<()> { + let arr = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // norm = 1.0 + 3.0, 4.0, 0.0, // norm = 5.0 + ], + )?; + assert_close(&eval_l2_norm(arr)?, &[1.0, 5.0]); + Ok(()) +} + +#[test] +fn null_input_row() -> VortexResult<()> { + // 2 rows of dim-2 vectors. Row 1 is masked as null. + let arr = tensor_array(&[2], &[3.0, 4.0, 0.0, 0.0])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![arr])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + // Row 0: norm = 5.0, row 1: null. + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +/// A constant input whose scalar is a non-null tensor should short-circuit to a +/// [`ConstantArray`] output whose scalar is the precomputed norm. Uses [`execute_until`] so +/// execution stops at the [`Constant`] encoding instead of canonicalizing into a +/// [`PrimitiveArray`]. +#[test] +fn constant_non_null_input_yields_constant_output() -> VortexResult<()> { + let input = literal_vector_array(&[3.0f64, 4.0], 4); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("L2Norm over a constant input must produce a constant output"); + assert_eq!(constant.len(), 4); + let norm = constant + .scalar() + .as_primitive() + .as_::() + .expect("norm scalar must be a non-null primitive"); + assert_close(&[norm], &[5.0]); + Ok(()) +} + +/// An extension array over constant storage is folded just like a top-level constant instead of +/// recomputing the same norm once per row. +#[test] +fn extension_backed_constant_yields_constant_output() -> VortexResult<()> { + let input = Vector::constant_array(&[3.0f64, 4.0], 4)?; + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("L2Norm over constant-backed extension storage must produce a constant output"); + assert_eq!(constant.len(), 4); + let norm = constant + .scalar() + .as_primitive() + .as_::() + .expect("norm scalar must be a non-null primitive"); + assert_close(&[norm], &[5.0]); + Ok(()) +} + +/// A constant input whose scalar is null should short-circuit to a null [`ConstantArray`] of +/// the correct primitive dtype and length. +#[test] +fn constant_null_input_yields_null_constant_output() -> VortexResult<()> { + let storage_dtype = DType::FixedSizeList( + DType::Primitive(PType::F64, Nullability::NonNullable).into(), + 2, + Nullability::Nullable, + ); + let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); + let null_scalar = Scalar::null(DType::Extension(ext_dtype)); + let input = ConstantArray::new(null_scalar, 3).into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![input])?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("null constant input must produce a constant output"); + assert_eq!(constant.len(), 3); + assert!(constant.scalar().is_null()); + assert_eq!( + constant.dtype(), + &DType::Primitive(PType::F64, Nullability::Nullable) + ); + Ok(()) +} + +/// An `f32` column must dispatch at `f32` and produce an `f32` result, which is the property that +/// makes width polymorphism load-bearing rather than decorative. +#[rstest] +#[case::f32(&[3.0f32, 4.0], PType::F32)] +#[case::f64(&[3.0f64, 4.0], PType::F64)] +fn dispatches_at_input_width( + #[case] elements: &[T], + #[case] expected: PType, +) -> VortexResult<()> { + let arr = tensor_array(&[2], elements)?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = L2Norm + .try_new_array(arr.len(), EmptyOptions, [arr])? + .execute(&mut ctx)?; + assert_eq!(prim.ptype(), expected); + Ok(()) +} + +/// `L2Norm(Normalized(normalized, norms))` reads back the authoritative stored norms rather than +/// recomputing over decoded coordinates. The normalized child here is deliberately *not* +/// unit-norm, mimicking lossy storage, so readthrough and recompute disagree: row 0 decodes to +/// `[6, 8]` (norm `10`) and row 1 to `[6, 0]` (norm `6`), while the stored norms are `5` and `2`. +#[test] +fn normalized_readthrough_returns_stored_norms() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[1.2, 1.6, 3.0, 0.0])?; + let norms = PrimitiveArray::from_iter([5.0f64, 2.0]).into_array(); + // SAFETY: A focused test of the lossy storage contract: the stored norms are authoritative + // even though this normalized child violates the unit-norm invariant. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + assert_close(&eval_l2_norm(denorm)?, &[5.0, 2.0]); + Ok(()) +} + +/// The readthrough must survive a partially-null column. +/// +/// This pins the dense policy the row contract derives. Filtering could hand `reduce_encoded` a +/// filtered input, which is no longer an `ExactScalarFn`, silently falling back to +/// decode-and-recompute. For a lossy child that changes the answer: row 0 below would come back as +/// `10` (recomputed from `[6, 8]`) instead of the authoritative stored `5`. +#[test] +fn normalized_readthrough_survives_null_rows() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[1.2, 1.6, 3.0, 0.0])?; + let norms = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + // SAFETY: Intentionally lossy, as in `normalized_readthrough_returns_stored_norms`, so that + // a recompute fallback is observable. + let denorm = unsafe { Normalized::new_unchecked(normalized, norms) }.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![denorm])?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +/// The readthrough must still propagate nulls carried by the `norms` child. +#[test] +fn normalized_readthrough_propagates_null_norms() -> VortexResult<()> { + let normalized = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; + let norms = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let denorm = Normalized::try_new(normalized, norms, &mut ctx)?.into_array(); + + let scalar_fn = L2Norm.bind(EmptyOptions); + let result = ScalarFnArray::try_new(scalar_fn, vec![denorm])?; + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[5.0]); + Ok(()) +} + +#[rstest] +#[case::fixed_shape_tensor(l2_norm_tensor_child())] +#[case::vector(l2_norm_vector_child())] +fn serde_round_trip(#[case] child: ArrayRef) -> VortexResult<()> { + let original = L2Norm.try_new_array(child.len(), EmptyOptions, [child.clone()])?; + + let plugin = ScalarFnArrayPlugin::new(L2Norm); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("L2Norm serialize must produce metadata"); + + let children = vec![child]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} + +fn l2_norm_tensor_child() -> ArrayRef { + tensor_array(&[3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid tensor array") +} + +fn l2_norm_vector_child() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") +} diff --git a/vortex-tensor/src/scalar_fns/tests/mod.rs b/vortex-tensor/src/scalar_fns/tests/mod.rs new file mode 100644 index 00000000000..bb3726e9329 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/mod.rs @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for the tensor scalar functions. + +mod cosine_similarity; +mod inner_product; +mod l2_norm; +mod row; diff --git a/vortex-tensor/src/scalar_fns/tests/row.rs b/vortex-tensor/src/scalar_fns/tests/row.rs new file mode 100644 index 00000000000..f08f614cfef --- /dev/null +++ b/vortex-tensor/src/scalar_fns/tests/row.rs @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use num_traits::Float; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::MaskedArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::match_each_float_ptype; +use vortex_array::scalar_fn::ElementSink; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::RowFn; +use vortex_array::scalar_fn::RowVisitor; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::assert_element_conforms; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; +use vortex_session::registry::CachedId; + +use crate::scalar_fns::row::TensorRow; +use crate::scalar_fns::row::tensor_element_ptype; +use crate::utils::test_helpers::assert_close; +use crate::utils::test_helpers::tensor_array; + +/// The marginal cost of a new tensor scalar function is this entire definition. Everything else +/// (null propagation, constants, validity, f16/f32/f64 dispatch, dtype checks, and constructors) is +/// derived. +#[derive(Clone, Debug, Default)] +struct L1Norm; + +impl RowFn for L1Norm { + type Options = EmptyOptions; + + const ARG_NAMES: &'static [&'static str] = &["input"]; + + fn id(&self) -> ScalarFnId { + static ID: CachedId = CachedId::new("vortex.test.l1_norm"); + *ID + } + + fn dispatch( + &self, + _options: &Self::Options, + args: &[DType], + visitor: V, + ) -> VortexResult { + match_each_float_ptype!(tensor_element_ptype(args)?, |T| { + visitor.visit_prepared_into::<(TensorRow,), ElementSink, _, _>( + |_| (), + |&(), (row,), output| *output = l1_norm_row(row), + ) + }) + } +} + +fn l1_norm_row(row: &[T]) -> T { + row.iter().fold(T::zero(), |acc, &x| acc + x.abs()) +} + +#[test] +fn derived_fn_executes_with_nulls() -> VortexResult<()> { + let arr = tensor_array(&[2], &[3.0, -4.0, 1.0, 1.0])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + let prim: PrimitiveArray = L1Norm + .try_new_array(arr.len(), EmptyOptions, [arr])? + .execute(&mut ctx)?; + + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert_close(&[prim.as_slice::()[0]], &[7.0]); + Ok(()) +} + +/// A kernel written once serves every float width. +#[test] +fn derived_fn_dispatches_at_input_width() -> VortexResult<()> { + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + + let f32_result: PrimitiveArray = L1Norm + .try_new_array(1, EmptyOptions, [tensor_array(&[2], &[3.0f32, -4.0])?])? + .execute(&mut ctx)?; + assert_eq!(f32_result.ptype(), PType::F32); + assert_eq!(f32_result.as_slice::(), &[7.0f32]); + + let f64_result: PrimitiveArray = L1Norm + .try_new_array(1, EmptyOptions, [tensor_array(&[2], &[3.0f64, -4.0])?])? + .execute(&mut ctx)?; + assert_eq!(f64_result.ptype(), PType::F64); + Ok(()) +} + +/// Runs the out-of-crate [`TensorRow`] element through `vortex-array`'s shared element conformance +/// check, with `NaN` and infinities sitting behind the null row so a wrong `DENSE_SAFE` would be +/// read rather than skipped. +#[test] +fn tensor_row_element_conforms() -> VortexResult<()> { + let mut ctx = crate::tests::SESSION.create_execution_ctx(); + let arr = tensor_array(&[2], &[3.0, -4.0, f64::NAN, f64::INFINITY])?; + let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); + + assert_element_conforms::>( + arr, + &DType::Primitive(PType::F64, Nullability::NonNullable), + &mut ctx, + ) +} diff --git a/vortex-tensor/src/utils.rs b/vortex-tensor/src/utils.rs index 488694bd47f..460dde82ea7 100644 --- a/vortex-tensor/src/utils.rs +++ b/vortex-tensor/src/utils.rs @@ -1,13 +1,17 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +//! Shared helpers for the tensor scalar functions. + use half::f16; +use num_traits::Float; use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::Constant; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFn; @@ -20,6 +24,8 @@ use vortex_array::dtype::NativePType; use vortex_array::dtype::PType; use vortex_array::dtype::proto::dtype as pb; use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure; @@ -58,6 +64,20 @@ pub fn unit_norm_tolerance(element_ptype: PType, dimensions: usize) -> f64 { SAFETY_FACTOR as f64 * machine_epsilon * dimensions_root } +/// The L2 norm of one row: `sqrt(sum(v_i^2))`. A zero-length or all-zero row gives `0.0`. +/// +/// Shared by `l2_norm` and by cosine similarity's hoisted constant norm. The accumulation order is +/// part of the contract rather than an implementation detail: cosine's prepared and per-row arms +/// must agree bit for bit, which only holds while both sum in this order. Keeping one copy is what +/// stops the two drifting apart. +pub(crate) fn l2_norm_row(v: &[T]) -> T { + let mut sum_sq = T::zero(); + for &x in v { + sum_sq = sum_sq + x * x; + } + sum_sq.sqrt() +} + /// Extracts the `(normalized, norms)` children of a [`Normalized`]-encoded array. /// /// # Panics @@ -97,17 +117,78 @@ pub fn validate_tensor_float_input(input_dtype: &DType) -> VortexResult( - lhs: &'a DType, - rhs: &DType, -) -> VortexResult> { - vortex_ensure!( - lhs.eq_ignore_nullability(rhs), - "binary tensor expression expects inputs to have the same dtype, got {lhs} and {rhs}" - ); - validate_tensor_float_input(lhs) +pub fn validate_tensor_float_inputs(args: &[DType]) -> VortexResult> { + let (first, rest) = args + .split_first() + .ok_or_else(|| vortex_err!("tensor expression expects at least one input"))?; + for arg in rest { + vortex_ensure!( + first.eq_ignore_nullability(arg), + "tensor expression expects inputs to have the same dtype, got {first} and {arg}" + ); + } + validate_tensor_float_input(first) +} + +/// Metadata for a serialized binary tensor-op array (shared by [`InnerProduct`] and +/// [`CosineSimilarity`]). Both operands share the same extension dtype up to nullability +/// (enforced by their `return_dtype` checks), but their individual nullabilities are lost in the +/// parent's unioned output, so both are persisted. +/// +/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity +/// [`InnerProduct`]: crate::scalar_fns::inner_product::InnerProduct +#[derive(Clone, prost::Message)] +pub(crate) struct BinaryTensorOpMetadata { + #[prost(message, optional, tag = "1")] + pub(crate) lhs_dtype: Option, + #[prost(message, optional, tag = "2")] + pub(crate) rhs_dtype: Option, +} + +impl BinaryTensorOpMetadata { + /// Encodes the two children of `view` into a [`BinaryTensorOpMetadata`] byte blob. + pub(crate) fn encode_from_view( + view: &ScalarFnArrayView, + ) -> VortexResult> { + let scalar_fn_array = view.as_::(); + let lhs_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + let rhs_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); + Ok(Self { + lhs_dtype, + rhs_dtype, + } + .encode_to_vec()) + } + + /// Decodes `metadata` and fetches both children from `children` using the decoded dtypes, + /// validating that `lhs` and `rhs` are compatible tensor operands. + pub(crate) fn decode_children( + metadata: &[u8], + len: usize, + children: &dyn vortex_array::serde::ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = Self::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode BinaryTensorOpMetadata: {e}"))?; + let lhs_pb = metadata + .lhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing lhs_dtype"))?; + let rhs_pb = metadata + .rhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing rhs_dtype"))?; + + let lhs_dtype = DType::from_proto(lhs_pb, session)?; + let rhs_dtype = DType::from_proto(rhs_pb, session)?; + validate_tensor_float_inputs(&[lhs_dtype.clone(), rhs_dtype.clone()])?; + + let lhs = children.get(0, &lhs_dtype, len)?; + let rhs = children.get(1, &rhs_dtype, len)?; + Ok(vec![lhs, rhs]) + } } /// The flat primitive elements of a tensor storage array, with typed row access. @@ -132,12 +213,58 @@ impl FlatElements { /// /// When the source was a constant-backed storage, all indices resolve to the single stored /// row. + /// + /// This re-derives the typed slice on every call, which costs a ptype check and a buffer + /// downcast per row. A caller reading every row in a loop should take [`into_buffer`](Self::into_buffer) + /// instead and pay that once. #[must_use] pub fn row(&self, i: usize) -> &[T] { let row_idx = if self.is_constant { 0 } else { i }; let slice = self.elems.as_slice::(); &slice[row_idx * self.list_size..][..self.list_size] } + + /// Elements per row. + #[must_use] + pub fn list_size(&self) -> usize { + self.list_size + } + + /// The row stride: `list_size` for a full column, and `0` for constant-backed storage, whose + /// single materialized row every index reads. + #[must_use] + pub fn row_stride(&self) -> usize { + if self.is_constant { 0 } else { self.list_size } + } + + /// The elements as a typed buffer, checking the ptype once instead of once per row. + pub fn into_buffer(self) -> Buffer { + self.elems.into_buffer::() + } +} + +/// Rebuilds a tensor-like extension array from flat primitive elements. +/// +/// # Errors +/// +/// Returns an error if `elements` does not hold exactly `tensor_flat_size * row_count` values. +pub(crate) fn build_tensor_array( + dtype: DType, + tensor_flat_size: usize, + row_count: usize, + validity: Validity, + elements: Buffer, +) -> VortexResult { + let list_size = + u32::try_from(tensor_flat_size).vortex_expect("tensor flat size must fit into `u32`"); + + // SAFETY: Tensor elements are always non-nullable, so the validity carries no length. + let elements = unsafe { PrimitiveArray::new_unchecked(elements, Validity::NonNullable) }; + + let storage = + FixedSizeListArray::try_new(elements.into_array(), list_size, validity, row_count)?; + + Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) } /// Extracts the flat primitive elements from a tensor storage array (FixedSizeList). @@ -161,10 +288,10 @@ pub fn extract_flat_elements( let fsl: FixedSizeListArray = source.execute(ctx)?; let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + let dtype = elems.dtype(); vortex_ensure!( !elems.nullability().is_nullable(), - "tensor storage elements must be non-nullable, got {}", - elems.dtype(), + "tensor storage elements must be non-nullable, got {dtype}", ); Ok(FlatElements { elems, @@ -216,73 +343,14 @@ pub fn extract_constant_flat_row( let single = ConstantArray::new(constant.scalar().clone(), 1).into_array(); let fsl: FixedSizeListArray = single.execute(ctx)?; let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + let dtype = elems.dtype(); vortex_ensure!( !elems.nullability().is_nullable(), - "tensor storage elements must be non-nullable, got {}", - elems.dtype(), + "tensor storage elements must be non-nullable, got {dtype}", ); Ok(FlatRow { elems }) } -/// Metadata for a serialized binary tensor-op array (shared by [`InnerProduct`] and -/// [`CosineSimilarity`]). Both operands share the same extension dtype up to nullability -/// (enforced by their `return_dtype` checks), but their individual nullabilities are lost in the -/// parent's unioned output, so both are persisted. -/// -/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity -/// [`InnerProduct`]: crate::scalar_fns::inner_product::InnerProduct -#[derive(Clone, prost::Message)] -pub(crate) struct BinaryTensorOpMetadata { - #[prost(message, optional, tag = "1")] - pub(crate) lhs_dtype: Option, - #[prost(message, optional, tag = "2")] - pub(crate) rhs_dtype: Option, -} - -impl BinaryTensorOpMetadata { - /// Encodes the two children of `view` into a [`BinaryTensorOpMetadata`] byte blob. - pub(crate) fn encode_from_view( - view: &ScalarFnArrayView, - ) -> VortexResult> { - let scalar_fn_array = view.as_::(); - let lhs_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); - let rhs_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); - Ok(Self { - lhs_dtype, - rhs_dtype, - } - .encode_to_vec()) - } - - /// Decodes `metadata` and fetches both children from `children` using the decoded dtypes, - /// validating that `lhs` and `rhs` are compatible tensor operands. - pub(crate) fn decode_children( - metadata: &[u8], - len: usize, - children: &dyn vortex_array::serde::ArrayChildren, - session: &VortexSession, - ) -> VortexResult> { - let metadata = Self::decode(metadata) - .map_err(|e| vortex_err!("Failed to decode BinaryTensorOpMetadata: {e}"))?; - let lhs_pb = metadata - .lhs_dtype - .as_ref() - .ok_or_else(|| vortex_err!("metadata missing lhs_dtype"))?; - let rhs_pb = metadata - .rhs_dtype - .as_ref() - .ok_or_else(|| vortex_err!("metadata missing rhs_dtype"))?; - - let lhs_dtype = DType::from_proto(lhs_pb, session)?; - let rhs_dtype = DType::from_proto(rhs_pb, session)?; - validate_binary_tensor_float_inputs(&lhs_dtype, &rhs_dtype)?; - - let lhs = children.get(0, &lhs_dtype, len)?; - let rhs = children.get(1, &rhs_dtype, len)?; - Ok(vec![lhs, rhs]) - } -} - #[cfg(test)] pub mod test_helpers { use vortex_array::ArrayRef; @@ -358,9 +426,9 @@ pub mod test_helpers { } /// Builds a [`ConstantArray`] whose scalar is itself a [`Vector`] extension scalar, broadcast - /// to `len` rows. This is the shape produced by an `lit(vector_scalar)` literal expression — - /// the constant lives at the extension level rather than inside the FSL storage, in contrast - /// to [`Vector::constant_array`]. + /// to `len` rows. This is the shape produced by an `lit(vector_scalar)` literal expression, where + /// the constant lives at the extension level rather than inside the FSL storage, in contrast to + /// [`Vector::constant_array`]. pub fn literal_vector_array>( elements: &[T], len: usize, @@ -401,10 +469,10 @@ pub mod test_helpers { if a.is_nan() && e.is_nan() { continue; } + let diff = (a - e).abs(); assert!( (a - e).abs() < 1e-10, - "element {i}: got {a}, expected {e} (diff = {})", - (a - e).abs() + "element {i}: got {a}, expected {e} (diff = {diff})" ); } } diff --git a/vortex-tensor/src/vector_search.rs b/vortex-tensor/src/vector_search.rs index ad3b96d1bff..492bc837b89 100644 --- a/vortex-tensor/src/vector_search.rs +++ b/vortex-tensor/src/vector_search.rs @@ -35,11 +35,13 @@ use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; use vortex_array::scalar::PValue; use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexResult; @@ -79,7 +81,7 @@ pub fn build_similarity_search_tree>( let num_rows = data.len(); let query_vec = Vector::constant_array(query, num_rows)?; - let cosine = CosineSimilarity::try_new_array(data, query_vec)?.into_array(); + let cosine = CosineSimilarity.try_new_array(num_rows, EmptyOptions, [data, query_vec])?; let threshold_scalar = Scalar::primitive(threshold, Nullability::NonNullable); let threshold_array = ConstantArray::new(threshold_scalar, num_rows).into_array();