branch-4.2: [improvement](hive) Push partition filters to HMS #67725 - #68363
Open
zhaorongsheng wants to merge 1 commit into
Open
zhaorongsheng wants to merge 1 commit into
zhaorongsheng wants to merge 1 commit into
Conversation
### What problem does this PR solve? Issue Number: apache#67724 Related PR: apache#67725 Problem Summary: Hive partition pruning previously enumerated every HMS partition before applying a selective predicate. The branch-4.2 architecture predates the master connector modules, so this change ports the direct HMS filter path to the legacy HMS table and scan pipeline. Plain Hive scans defer partition materialization, convert safe equality and IN predicates to the HMS filter grammar, and fetch only matching partitions through get_partitions_by_filter. Unsupported predicates, unsafe identifiers, saturated responses, and RPC failures retain the full-list local-pruning fallback; no-filter scans are materialized before translation. ### Release note Improve planning latency for selective Hive partition queries when HMS supports get_partitions_by_filter. ### Check List (For Author) - Test: Unit Test - HivePartitionFilterBuilderTest (3 tests), LogicalFileScanTest (10 tests), and PruneFileScanPartitionTest (1 test) - Maven offline FE test-compile for fe-core and dependent modules - Maven offline FE checkstyle validation - git diff --check - Behavior changed: Yes (selective plain-Hive scans can materialize only the HMS-filtered partition view) - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #67724
Related PR: #67725
Problem Summary: Cherry-pick #67725 to branch-4.2. The branch uses the legacy HMS table and scan architecture rather than the master connector modules, so the port keeps the same user-visible behavior while adapting the implementation: plain Hive scans defer partition materialization, convert safe equality and IN predicates to the HMS filter grammar, and fetch only matching partitions through get_partitions_by_filter. Unsupported predicates, unsafe identifiers, saturated responses, and RPC failures retain full-list local pruning; no-filter scans are materialized before translation.
Release note
Improve planning latency for selective Hive partition queries when HMS supports get_partitions_by_filter.
Check List (For Author)