Skip to content

[vortex] Fix incorrect floating-point predicate pushdown - #10985

Open
zhouyc-ali wants to merge 2 commits into
apache:masterfrom
zhouyc-ali:fix/vortex-floating-point-pushdown
Open

zhouyc-ali wants to merge 2 commits into
apache:masterfrom
zhouyc-ali:fix/vortex-floating-point-pushdown

Conversation

@zhouyc-ali

@zhouyc-ali zhouyc-ali commented Nov 4, 2026 •

Copy link
Copy Markdown

Purpose

Paimon and Vortex use different NaN comparison semantics. For FLOAT/DOUBLE columns containing NaNs, native filtering can drop matching rows even for finite literals such as value > 0. Residual filtering cannot recover those rows.

Decline floating-point value predicates in the Vortex converter so the caller can evaluate them with Paimon's residual filter. Keep IS NULL / IS NOT NULL and supported AND conjuncts pushed down. Floating-point filters may read more rows as a result.

Tests

  • Add FLOAT/DOUBLE regressions for all six comparisons, IN/NOT IN lists of 1/20/21 values, NULL checks, and mixed AND/OR predicates.
  • Write/read finite values, signed zeros, infinities, NaNs with different signs/payloads, and NULLs. Verify candidate row IDs retain every expected match and residual-filtered IDs match Paimon filtering of an unfiltered read.
  • The four new parameterized test invocations fail without the fix and pass with it.
  • Module verify: 73 tests passed; 4 optional Java/Python E2E tests skipped. Checkstyle, Spotless, Maven Enforcer and Apache RAT passed.
  • Fix the existing CSV dynamic-option test to trigger validation at scan creation. All 32 CSV/runtime-validation tests pass on JDK 8 and 11.

Decline FLOAT/DOUBLE value predicate pushdown because Vortex and Paimon
use different NaN comparison semantics. Retain null checks and safe
conjunct pushdown, with regressions for candidate preservation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant