Vortex’s existing total ordering distinguishes -0.0 from +0.0 and compares NaNs by their bit patterns. Spark treats signed zeros as equal and all NaNs as equal. Spark currently pushes float and double predicates using Vortex’s existing comparison functions, so a pushed filter can return different rows than Spark evaluation.
Add an explicit comparison mode or separate bound scalar functions for Spark-compatible floating point equality and ordering. Preserve Vortex’s current behavior as the default. Until Spark-compatible operations exist, the Spark connector should decline affected pushdown.
Regression cases should cover =, !=, ranges, and IN with signed zeros and NaNs, including distinct NaN payloads.
Vortex’s existing total ordering distinguishes -0.0 from +0.0 and compares NaNs by their bit patterns. Spark treats signed zeros as equal and all NaNs as equal. Spark currently pushes float and double predicates using Vortex’s existing comparison functions, so a pushed filter can return different rows than Spark evaluation.
Add an explicit comparison mode or separate bound scalar functions for Spark-compatible floating point equality and ordering. Preserve Vortex’s current behavior as the default. Until Spark-compatible operations exist, the Spark connector should decline affected pushdown.
Regression cases should cover =, !=, ranges, and IN with signed zeros and NaNs, including distinct NaN payloads.