feat: feature-gate datafusion-substrait behind optional 'substrait' feature#21268
feat: feature-gate datafusion-substrait behind optional 'substrait' feature#21268zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
Conversation
…eature Makes datafusion-substrait an optional dependency in datafusion-sqllogictest, gated behind a new 'substrait' feature flag. This saves compile time when substrait is not needed (the common case for most developers). Changes: - datafusion-substrait is now optional in sqllogictest/Cargo.toml - Added 'substrait' feature flag - Gated substrait imports, module, and round-trip function behind cfg - Added no-op fallback when feature is disabled - Updated CI to pass --features substrait for round-trip test
There was a problem hiding this comment.
Pull request overview
This PR makes the datafusion-substrait integration in the datafusion-sqllogictest crate opt-in behind a new substrait feature flag to reduce compile time and dependency weight for the default build.
Changes:
- Make
datafusion-substraitan optional dependency and add asubstraitfeature indatafusion/sqllogictest/Cargo.toml. - Gate the Substrait round-trip engine module and public re-exports behind
#[cfg(feature = "substrait")]. - Add a non-
substraitfallback for the Substrait round-trip runner and update CI to enable the feature when running that mode.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| datafusion/sqllogictest/src/lib.rs | Gate the public re-export of DataFusionSubstraitRoundTrip behind the substrait feature. |
| datafusion/sqllogictest/src/engines/mod.rs | Gate the Substrait round-trip engine module and re-export behind the substrait feature. |
| datafusion/sqllogictest/Cargo.toml | Make datafusion-substrait optional and introduce a substrait feature enabling it. |
| datafusion/sqllogictest/bin/sqllogictests.rs | Conditionally import/use the Substrait engine; provide a runtime error fallback when feature is disabled. |
| .github/workflows/rust.yml | Update Substrait round-trip CI job to pass --features substrait. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # until most of the tickets in https://github.com/apache/datafusion/issues/16248 are addressed | ||
| # and this command can be run without filters. | ||
| run: cargo test --test sqllogictests -- --substrait-round-trip limit.slt | ||
| run: cargo test --test sqllogictests --features substrait -- --substrait-round-trip limit.slt |
There was a problem hiding this comment.
The cargo test --test sqllogictests --features substrait ... invocation is run from the workspace root in a virtual workspace. Adding --features substrait here is likely to fail because features must be applied to a specific package (e.g. --package datafusion-sqllogictest) or by running the command from datafusion/sqllogictest/. Consider updating this step to either cd datafusion/sqllogictest before running, or add --package datafusion-sqllogictest (and optionally --profile ci to match other CI invocations).
| run: cargo test --test sqllogictests --features substrait -- --substrait-round-trip limit.slt | |
| run: cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip limit.slt |
|
CI failure in |
Which issue does this PR close?
Closes #21269
Rationale for this change
datafusion-substraitis a heavyweight dependency that most developers and users don't need. Making it optional saves compile time for the common case.What changes are included in this PR?
datafusion-substraitis now an optional dependency indatafusion-sqllogictest/Cargo.tomlsubstraitfeature flag gates the dependency#[cfg(feature = "substrait")]gates the module, imports, andrun_test_file_substrait_round_tripfunction--features substraitfor the round-trip testAre these changes tested?
substraitfeature (default)--features substraitAre there any user-facing changes?
No — this only affects the sqllogictest binary, not the datafusion library itself. The substrait round-trip test requires
--features substraitto run.