feat: Add selectivity-tracking wrapper for dynamic filters#20160
Open
adriangb wants to merge 14 commits intoapache:mainfrom
Open
feat: Add selectivity-tracking wrapper for dynamic filters#20160adriangb wants to merge 14 commits intoapache:mainfrom
adriangb wants to merge 14 commits intoapache:mainfrom
Conversation
github-actions
bot
added
physical-expr
adriangb
force-pushed
the
dynamic-dynamic-filters
branch
from
8ccef6c to
9a1ebd9
Compare
adriangb
commented
Feb 5, 2026
datafusion/common/src/config.rs
Outdated
Comment on lines
1118
to
1144
| /// Enable selectivity-based disabling of dynamic filters from joins. | ||
| /// | ||
| /// When enabled, join dynamic filters that pass most rows (above the threshold) | ||
| /// will be automatically disabled to avoid evaluation overhead. This is useful | ||
| /// when the build side of a join covers most of the probe side values, making | ||
| /// the filter expensive to evaluate for little benefit. | ||
| /// | ||
| /// The selectivity tracking resets when the dynamic filter is updated (e.g., when | ||
| /// the hash table is built), allowing the filter to be re-evaluated with new data. | ||
| pub enable_dynamic_filter_selectivity_tracking: bool, default = false | ||
|
|
||
| /// Selectivity threshold for disabling join dynamic filters. | ||
| /// | ||
| /// If the filter passes this fraction or more of rows, it will be disabled. | ||
| /// Value should be between 0.0 and 1.0. | ||
| /// | ||
| /// For example, 0.95 means if 95% or more of rows pass the filter, it will be disabled. | ||
| /// Only used when `enable_dynamic_filter_selectivity_tracking` is true. | ||
| pub dynamic_filter_selectivity_threshold: f64, default = 0.95 | ||
|
|
||
| /// Minimum number of rows to process before making a selectivity decision | ||
| /// for join dynamic filters. | ||
| /// | ||
| /// The filter will remain in a tracking state until this many rows have been | ||
| /// processed. This ensures statistical stability before making the disable decision. | ||
| /// Only used when `enable_dynamic_filter_selectivity_tracking` is true. | ||
| pub dynamic_filter_min_rows_for_selectivity: usize, default = 10_000 |
Contributor
Author
There was a problem hiding this comment.
Not sure we need all of these, or at least not sure they should be prefixed with dynamic_filter
| pub struct SelectivityConfig { | ||
| /// Threshold above which the filter is disabled (e.g., 0.95 = 95% selectivity). | ||
| /// If the filter passes this fraction or more of rows, it will be disabled. | ||
| pub threshold: f64, |
Contributor
Author
There was a problem hiding this comment.
Could be in GB/s? Rows/s?
| /// If the filter passes this fraction or more of rows, it will be disabled. | ||
| pub threshold: f64, | ||
| /// Minimum rows to process before making a selectivity decision. | ||
| pub min_rows: usize, |
Contributor
Author
There was a problem hiding this comment.
Could be in GB? Number of batches? Time? (or all of the above?)
Contributor
Author
|
run benchmark tpchds tpch |
Contributor
Author
|
run benchmark tpchds tpch tpch10 |
Contributor
Author
|
run benchmarks |
github-actions
bot
added
the
documentation
Contributor
Author
|
show benchmark queue |
Contributor
|
run benchmark tpcds |
Contributor
|
run benchmark tpch |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
Author
|
I expect benchmarks to look bad - there's overhead this wrapper introduces right now even in no-op mode. I'm going to push a fix and run benches again. |
Contributor
Author
|
run benchmark tpch |
|
🤖 |
Contributor
Author
|
run benchmark tpcds |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
Author
|
run benchmark tpcds |
|
🤖 |
|
🤖: Benchmark completed Details
|
The snapshot() method was returning None, causing the wrapper to be preserved during snapshotting. Since PruningPredicate doesn't recognize the wrapper type, it fell back to lit(true) which disabled all row group and file pruning. Changed snapshot() to return the inner expression directly, stripping the wrapper during snapshotting so pruning predicates work correctly. Also set enable_adaptive_filter_selectivity_tracking default to false since this is an experimental feature. Benchmarks show no slowdowns after this fix. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Simplifies the implementation by removing generation-aware reset logic: - No longer tracks inner filter's generation - snapshot_generation() returns inner's generation, or 0 when disabled - Faster evaluate() path without generation checks The generation tracking was unnecessary for hash join filters and added overhead to the hot path. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
adriangb
force-pushed
the
dynamic-dynamic-filters
branch
from
22439b2 to
4124aae
Compare
… tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
|
run benchmark tpcds |
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
Author
|
run benchmark clickbench_partitioned |
Contributor
Author
|
@Dandandan numbers looking pretty good here! |
|
🤖 |
|
🤖: Benchmark completed Details
|
2 tasks
Contributor
|
run benchmarks |
Dandandan
reviewed
Feb 14, 2026
Contributor
|
run benchmarks |
Contributor
|
run benchmark tpch |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
So an improvement for 2 queries, but still most queries are slower than without pushdown. |
Contributor
Author
|
This is the main improvement. |
Contributor
Ok - yes I see some improvements here and there but it is still largely regressing main with ~30s (TPCDS runs in ~50s without and ~80s with filter pushdown). See e.g. this run #20318 (comment) against main without dynamic filter pushdown. This ~26x regression (and many others) is still unchanged in this PR: As we're running with I think I now have an understanding why the current approaches adaptiveness isn't helping that much yet. I think for it to work effectively, it needs to integrate more with the parquet reader to remove or add a filter based on the adaptiveness during the scan. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
SelectivityAwareFilterExpr, a wrapperPhysicalExprthat tracks filter selectivity at runtime and automatically disables filters that aren't pruning enough rows. This addresses the issue where dynamic filters fromHashJoinExeccan be expensive to evaluate for little benefit when the build side covers most of the probe side values.Key Features
rows_passed / rows_total >= thresholdmin_rowsbefore making a decisionNew Configuration Options
Added to
OptimizerOptions:enable_dynamic_filter_selectivity_tracking(default:false)dynamic_filter_selectivity_threshold(default:0.95)dynamic_filter_min_rows_for_selectivity(default:10000)Files Changed
datafusion/physical-expr/src/expressions/selectivity_aware_filter.rsdatafusion/physical-expr/src/expressions/mod.rsdatafusion/common/src/config.rsOptimizerOptionsdatafusion/physical-plan/src/joins/hash_join/exec.rsdatafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.sltTest plan
SelectivityAwareFilterExpr(6 tests)🤖 Generated with Claude Code