fix(spec): match Java whitespace for CHAR/VARCHAR partition values - #985
jackylee-ch wants to merge 2 commits into
Conversation
The CHAR/VARCHAR arm of the partition-value computer folded a value to the default partition name when `str::trim` left it empty, but Rust's whitespace set differs from Java's `Character.isWhitespace`: `str::trim` strips a non-breaking space (U+00A0) and U+2007 / U+202F, which Java keeps, and keeps U+001C-U+001F, which Java strips. Because the partition value is recomputed on both the write and read paths, such a value produced a different partition directory in Rust than in Java, so cross-engine readers miss the data. Reuse `is_java_whitespace_only` — already used by the Binary arm — so the string arm folds exactly what Java folds.
`format_partition_value` folded a CHAR/VARCHAR value to the default partition with `str::trim().is_empty()`, whose whitespace set differs from Java's `Character.isWhitespace`: U+00A0 (which Java keeps) folded in Rust, and U+001C-U+001F (which Java folds) did not, so a Format Table partition directory diverged from a Java-written one on both the write and read paths. The `name_prefix_pattern` pushdown skip used the same `str::trim` check. Reuse `is_java_whitespace_only` (the predicate the spec-side partition computer already uses) in both, matching Java `InternalRowPartitionComputer` (value folding) and `PartitionPathUtils.buildPartitionNamePrefixPattern` (the pushdown skip). Add a partition-value test covering both divergent directions.
c69d471 to
85b3f91
Compare
|
Requirement fit: SUPPORTED — partition-path interoperability and correct pruning have end-to-end value. Implementation: FINDINGS at [P2] Preserve U+180E partitions written by supported modern JVMs ( I reproduced this with a real Parquet format table and SQL: an unfiltered scan of a Verification: 200/200 partition-filtered core tests passed. The existing DataFusion partition-filter integration test passed; the added U+180E regression failed on the PR and both tests passed on the baseline. Diff check and current-main merge-tree are clean; all 14 head CI checks are green. Temporary edits were restored. |
The CHAR/VARCHAR arm of the partition-value computer folds a value to the default partition name when
str::trimleaves it empty, but Rust's whitespace set differs from Java'sCharacter.isWhitespace:str::trimstrips a non-breaking space (U+00A0) and U+2007 / U+202F, which Java keeps, and keeps U+001C-U+001F, which Java strips.The partition value is recomputed on both the write and read paths (BinaryRow -> path), so such a value produces a different partition directory in Rust than in Java, and a cross-engine reader misses the data. The Binary arm already handled this correctly via
is_java_whitespace_only(added in #958); the string arm was left onstr::trim.The same
str::trimfold also appeared in the Format Table partition paths —format_partition_value(the CHAR/VARCHAR directory value) and thename_prefix_patternpushdown skip — so those diverged the same way. They are fixed too, matching JavaInternalRowPartitionComputer(value folding) andPartitionPathUtils.buildPartitionNamePrefixPattern(the pushdown skip).Reuse
is_java_whitespace_onlyin every arm so the fold matches exactly what Java folds. Tests cover both divergent directions (U+001C folds, U+00A0 does not) for the spec-side computer and the Format Table value.