Spark 4 string comparisons and substring predicates can use a column’s collation. Vortex’s current UTF-8 comparisons and LIKE evaluation use binary, case-sensitive semantics. The Spark connector accepts StringType predicates without checking collation, so pushing a predicate on a case-insensitive column can discard matching rows.
Push string predicates only when Vortex’s operation matches the resolved Spark collation. Add explicit collation-aware bound operations if broader pushdown is desired; otherwise leave those predicates with Spark. Cover equality, ordering, IN, STARTS_WITH, ENDS_WITH, and CONTAINS in regression tests.
Spark 4 string comparisons and substring predicates can use a column’s collation. Vortex’s current UTF-8 comparisons and LIKE evaluation use binary, case-sensitive semantics. The Spark connector accepts StringType predicates without checking collation, so pushing a predicate on a case-insensitive column can discard matching rows.
Push string predicates only when Vortex’s operation matches the resolved Spark collation. Add explicit collation-aware bound operations if broader pushdown is desired; otherwise leave those predicates with Spark. Cover equality, ordering, IN, STARTS_WITH, ENDS_WITH, and CONTAINS in regression tests.