Skip to content

feat: feature-gate datafusion-substrait behind optional 'substrait' feature#21268

Open
zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
zhuqi-lucas:feat/substrait-feature-gate
Open

feat: feature-gate datafusion-substrait behind optional 'substrait' feature#21268
zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
zhuqi-lucas:feat/substrait-feature-gate

Conversation

@zhuqi-lucas
Copy link
Copy Markdown
Contributor

@zhuqi-lucas zhuqi-lucas commented Mar 31, 2026

Which issue does this PR close?

Closes #21269

Rationale for this change

datafusion-substrait is a heavyweight dependency that most developers and users don't need. Making it optional saves compile time for the common case.

What changes are included in this PR?

  • datafusion-substrait is now an optional dependency in datafusion-sqllogictest/Cargo.toml
  • New substrait feature flag gates the dependency
  • #[cfg(feature = "substrait")] gates the module, imports, and run_test_file_substrait_round_trip function
  • No-op fallback function returns an error when the feature is disabled
  • CI workflow updated to pass --features substrait for the round-trip test

Are these changes tested?

  • Compiles without substrait feature (default)
  • Compiles with --features substrait
  • CI substrait round-trip test updated to enable the feature

Are there any user-facing changes?

No — this only affects the sqllogictest binary, not the datafusion library itself. The substrait round-trip test requires --features substrait to run.

…eature

Makes datafusion-substrait an optional dependency in datafusion-sqllogictest,
gated behind a new 'substrait' feature flag. This saves compile time
when substrait is not needed (the common case for most developers).

Changes:
- datafusion-substrait is now optional in sqllogictest/Cargo.toml
- Added 'substrait' feature flag
- Gated substrait imports, module, and round-trip function behind cfg
- Added no-op fallback when feature is disabled
- Updated CI to pass --features substrait for round-trip test
Copilot AI review requested due to automatic review settings March 31, 2026 07:02
@github-actions github-actions bot added development-process Related to development process of DataFusion sqllogictest SQL Logic Tests (.slt) labels Mar 31, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the datafusion-substrait integration in the datafusion-sqllogictest crate opt-in behind a new substrait feature flag to reduce compile time and dependency weight for the default build.

Changes:

  • Make datafusion-substrait an optional dependency and add a substrait feature in datafusion/sqllogictest/Cargo.toml.
  • Gate the Substrait round-trip engine module and public re-exports behind #[cfg(feature = "substrait")].
  • Add a non-substrait fallback for the Substrait round-trip runner and update CI to enable the feature when running that mode.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
datafusion/sqllogictest/src/lib.rs Gate the public re-export of DataFusionSubstraitRoundTrip behind the substrait feature.
datafusion/sqllogictest/src/engines/mod.rs Gate the Substrait round-trip engine module and re-export behind the substrait feature.
datafusion/sqllogictest/Cargo.toml Make datafusion-substrait optional and introduce a substrait feature enabling it.
datafusion/sqllogictest/bin/sqllogictests.rs Conditionally import/use the Substrait engine; provide a runtime error fallback when feature is disabled.
.github/workflows/rust.yml Update Substrait round-trip CI job to pass --features substrait.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# until most of the tickets in https://github.com/apache/datafusion/issues/16248 are addressed
# and this command can be run without filters.
run: cargo test --test sqllogictests -- --substrait-round-trip limit.slt
run: cargo test --test sqllogictests --features substrait -- --substrait-round-trip limit.slt
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cargo test --test sqllogictests --features substrait ... invocation is run from the workspace root in a virtual workspace. Adding --features substrait here is likely to fail because features must be applied to a specific package (e.g. --package datafusion-sqllogictest) or by running the command from datafusion/sqllogictest/. Consider updating this step to either cd datafusion/sqllogictest before running, or add --package datafusion-sqllogictest (and optionally --profile ci to match other CI invocations).

Suggested change
run: cargo test --test sqllogictests --features substrait -- --substrait-round-trip limit.slt
run: cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip limit.slt

Copilot uses AI. Check for mistakes.
@zhuqi-lucas
Copy link
Copy Markdown
Contributor Author

CI failure in verify benchmark results is a pre-existing issue on main — explain.slt JSON format mismatch, unrelated to this PR (we only changed substrait feature gating).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature-gate datafusion-substrait behind optional feature to reduce compile time

2 participants