You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Vortex scans currently execute through the Vortex Spark reader. Native scan support would allow Comet to read Vortex files directly and feed the resulting Arrow batches into downstream Comet operators.
Spark and the Vortex Spark connector would remain responsible for scan planning, including file discovery, projection, and filter pushdown. A Vortex contrib module would consume the resulting scan information through the existing Comet contrib interfaces and execute supported scans using vortex-datafusion.
Build-time and runtime support would be opt-in. Scans outside the supported scope would retain the Vortex Spark reader.
The companion API change, vortex-data/vortex#9956, exposes the full table schema and accepted predicates. This lets the native reader preserve filtering semantics when columns used only by predicates have been pruned from the scan output.
Add the native reader and Spark-side integration, together with supported-scan checks, fallback behavior, and end-to-end tests.
Extend scan capabilities and add performance optimizations in follow-up PRs.
Additional context
We have a working prototype with local and distributed correctness checks, including full output comparisons against Spark, and plan to contribute it in stages.
The packaged prototype also encountered an Arrow C Data class conflict similar to the one reported during the Lance integration. We would like to coordinate the packaging approach with that effort.
Does this integration approach and proposed contribution split fit the direction of Comet’s contrib integrations?
What is the problem the feature request solves?
Vortex scans currently execute through the Vortex Spark reader. Native scan support would allow Comet to read Vortex files directly and feed the resulting Arrow batches into downstream Comet operators.
Describe the potential solution
Follow the contrib architecture proposed for Lance in #4632 and #4633, including the maintainer recommendation to keep format-specific integration in contrib.
Spark and the Vortex Spark connector would remain responsible for scan planning, including file discovery, projection, and filter pushdown. A Vortex contrib module would consume the resulting scan information through the existing Comet contrib interfaces and execute supported scans using
vortex-datafusion.Build-time and runtime support would be opt-in. Scans outside the supported scope would retain the Vortex Spark reader.
The companion API change, vortex-data/vortex#9956, exposes the full table schema and accepted predicates. This lets the native reader preserve filtering semantics when columns used only by predicates have been pruned from the scan output.
Proposed contribution stages:
Additional context
We have a working prototype with local and distributed correctness checks, including full output comparisons against Spark, and plan to contribute it in stages.
The packaged prototype also encountered an Arrow C Data class conflict similar to the one reported during the Lance integration. We would like to coordinate the packaging approach with that effort.
Does this integration approach and proposed contribution split fit the direction of Comet’s contrib integrations?