Skip to content

feat(datafusion): support bool and numeric schema evolution to Utf8 - #9263

Draft
haohuaijin wants to merge 5 commits into
vortex-data:developfrom
haohuaijin:codex/datafusion-bool-numeric-to-utf8
Draft

feat(datafusion): support bool and numeric schema evolution to Utf8#9263
haohuaijin wants to merge 5 commits into
vortex-data:developfrom
haohuaijin:codex/datafusion-bool-numeric-to-utf8

Conversation

@haohuaijin

Copy link
Copy Markdown
Contributor

Depends on #9261 and should remain in draft until that PR is merged.

Summary

  • Accept Bool and numeric physical columns under a unified Utf8 or LargeUtf8 schema.
  • Preserve each file physical type so the expression adapter can insert the cast to the logical string type.
  • Keep numeric and Bool columns incompatible with Binary and LargeBinary schemas.

E2E coverage

The schema-evolution test covers the complete progression:

  • Int64 schema reads Bool and Int64 files.
  • Float64 schema reads Bool, Int64, and Float64 files.
  • Utf8 schema reads Bool, Int64, Float64, and Utf8 files.

Each stage verifies full scans and filtered queries with projection pushdown both enabled and disabled.

Checks

  • cargo nextest run -p vortex-datafusion test_schema_evolution_type_widening_chain
  • cargo nextest run -p vortex-datafusion
  • cargo clippy -p vortex-datafusion --tests --all-features
  • cargo +nightly fmt --all
  • git diff --check

Signed-off-by: Huaijin <haohuaijin@gmail.com>
…r path

Signed-off-by: Huaijin <haohuaijin@gmail.com>
Signed-off-by: Huaijin <haohuaijin@gmail.com>
Drop the TestResult alias and inline crate::Canonical qualifications in the
primitive cast tests, move to_bit_buffer out of the all-null bool cast path,
and cover f16 NaN/inf formatting in the array-path utf8 cast.

Signed-off-by: Huaijin <haohuaijin@gmail.com>
Signed-off-by: Huaijin <haohuaijin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant