[zhouyc-ali]

Purpose

Linked issue: #319.

Push supported predicates, projections and bitmap selections into native Vortex scans while preserving physical file row IDs and Paimon's residual-filter semantics.

• Support comparisons, NULL checks, IN and NOT IN with a configurable list-size limit. Normalize decimal literals only when the value is unchanged; fall back for lossy timestamps, floating-point comparisons and legal temporal constants outside the native range. Type mismatches and genuine construction errors are reported.
• Reduce selective-read overhead by reusing default callback-reader sessions and lazily allocating dictionary filter caches. Add independent I/O coalescing byte limits while retaining the object-storage defaults and Vortex's default cache sharding.
• Bound view-to-standard Arrow materialization by splitting native batches first and trimming nested list children to referenced ranges. Validate pinned native patch states and extend the benchmark runner to support Vortex. Normalize new LTZ file metadata to UTC without changing epoch values.

Related Java NaN fix: apache/paimon#10985.

Tests

Added and updated tests covering:

• Predicate conversion and native scan results for comparisons, NULL checks,
  IN and NOT IN, including duplicates, integer boundaries, configurable list limits,
  and combined AND/OR predicates.
• Safe fallback for unsupported conditions, unbound columns, lossy timestamps,
  floating-point semantics, and membership size limits. Type mismatches and
  malformed/native construction errors are checked separately; legal temporal
  constants outside the native range fall back. Fixed-seed differential tests
  check native results before SDK residual filtering.
• Lossless decimal normalization to the file's precision/scale for comparisons
  and IN/NOT IN, with fallback for rounding or overflow.
• Reordered, nested and empty projections, schema metadata preservation,
  duplicate/missing-field and incompatible-type rejection, and SDK NULL filling
  for evolved fields. Empty-projection tests inspect native schemas and
  callback reads to verify omitted payload columns are skipped.
• Bounded export of encoded strings/binary and nested List/ListView/Struct arrays,
  including empty/NULL nested children and shared child arrays; unreferenced
  elements must not be materialized.
• Sparse, dense, full, and empty bitmap selections combined with predicates
  and projection, preserving physical row IDs across batches. The single-deletion
  regression verifies one exclusion index instead of expanding all surviving rows.
  Large full selections and single deletions cover compressed bounds checks,
  physical row mapping and signed 32-bit row-ID boundaries without modifying
  the input bitmap.
• Reader reset and invalid reconfiguration, ensuring failed resets preserve the
  active scan and returned batches remain valid after reset or reader destruction.
  Closing a retained reader releases selection storage and hidden row-index
  arrays while separately retained output batches remain valid.
• Arrow stream schema failures preserve native error details, prefer callback
  errors and handle null native messages before releasing the stream.
• Independent I/O coalescing limits preserving values and row IDs, injected
  read/write failures, waiting for in-flight callbacks on abort, and deferred
  stream closure.
• C++ SDK reads with both output APIs, residual filtering and late materialization
  enabled/disabled. Complete append probe/payload passes and enabled residual
  filtering require exact rows; other candidate paths must retain all matches.
  Primary-key updates/deletes check merged-row integrity before value filtering,
  and schema evolution covers missing columns and a user column named _row_idx.

Local validation passed 130 Vortex C++ tests and 33 SDK/ORC/Parquet regressions.
The subsequent declaration-style fixes passed the affected regression and
clang-tidy checks. The unchanged native library previously passed 106 Rust
tests; TSAN was not rerun locally for this revision.

API and Format

No changes to public C++ API signatures or the table protocol. The stored format remains Vortex; new LTZ files use UTC metadata without changing epoch values. Add the runtime reader options:

• vortex.read.in-pushdown.max-values: default 32; 0 disables IN/NOT IN pushdown. The original list length, including duplicates, is counted; oversized lists fall back without truncation.
• vortex.read.io-coalesce.max-gap: default 1048576 bytes (1 MiB).
• vortex.read.io-coalesce.max-size: default 16777216 bytes (16 MiB).

The private Vortex FFI is extended; its patched header and native library must be rebuilt together. Readers continue to require SDK residual filtering (EnablePredicateFilter(true)) or equivalent filtering by the caller for exact results. The shared read path and the meanings of its filtering flags are unchanged.

Documentation

Document the numeric Vortex reader options, defaults, concrete fallback rules, residual-filter requirement and LTZ metadata compatibility in docs/source/api/file_format.rst. Remove the format-specific additions from docs/source/user_guide/read.rst.

Generative AI tooling

Generated-by: OpenAI Codex 0.163.0

[SteNicholas]

LGTM.