Is your feature request related to a problem or challenge?
DataFusion has statistics estimation inside individual operators and separate implementations in the bundled StatisticsProviders. Enabling those providers can replace an operator's estimate with a different calculation for the same operation.
This surfaced in #25570: for TPC-H SF1 Q14, the bundled providers produced a join estimate of 14.73 billion rows instead of 73,650, against 75,983 actual rows. Calling the optional set “built-in providers” also makes it hard to tell which implementation is used by default.
Describe the solution you'd like
Could we consolidate built-in statistics estimation so each operator has one authoritative implementation? The registry makes sense as an extension point for custom estimators, but DataFusion's own providers should share the operator's estimation logic rather than maintain a separate algorithm.
This would make it clearer where to fix estimation bugs and avoid improvements reaching only one implementation.
Describe alternatives you've considered
Renaming or documenting the optional providers would clarify their behavior, but would leave the duplicated estimation logic in place.
Additional context
#25570 aligns the CLI with the session's configuration. This issue is about consolidating the underlying estimators.
Is your feature request related to a problem or challenge?
DataFusion has statistics estimation inside individual operators and separate implementations in the bundled
StatisticsProviders. Enabling those providers can replace an operator's estimate with a different calculation for the same operation.This surfaced in #25570: for TPC-H SF1 Q14, the bundled providers produced a join estimate of 14.73 billion rows instead of 73,650, against 75,983 actual rows. Calling the optional set “built-in providers” also makes it hard to tell which implementation is used by default.
Describe the solution you'd like
Could we consolidate built-in statistics estimation so each operator has one authoritative implementation? The registry makes sense as an extension point for custom estimators, but DataFusion's own providers should share the operator's estimation logic rather than maintain a separate algorithm.
This would make it clearer where to fix estimation bugs and avoid improvements reaching only one implementation.
Describe alternatives you've considered
Renaming or documenting the optional providers would clarify their behavior, but would leave the duplicated estimation logic in place.
Additional context
#25570 aligns the CLI with the session's configuration. This issue is about consolidating the underlying estimators.