Move partial-upsert resolveComparisonTies method from base class to implementations#18640
Open
deepthi912 wants to merge 2 commits into
Open
Move partial-upsert resolveComparisonTies method from base class to implementations#18640deepthi912 wants to merge 2 commits into
deepthi912 wants to merge 2 commits into
Conversation
…lasses The base addOrReplaceSegment wrapper in BasePartitionUpsertMetadataManager existed only to call resolveComparisonTies for partial-upsert before delegating to the subclass-specific doAddOrReplaceSegment. This made the abstract doAddOrReplaceSegment signature redundant with its only caller. Inline the wrapper: callers now invoke doAddOrReplaceSegment directly, and the resolveComparisonTies call moves into ConcurrentMapPartitionUpsertMetadataManager and ConcurrentMapPartitionUpsertMetadataManagerForConsistentDeletes (the only two non-abstract subclasses). Removes one virtual call per segment add/replace and eliminates a layer of indirection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nager Line 661 was 121 chars; wrap argument list so each line stays ≤120. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18640 +/- ##
=========================================
Coverage 64.38% 64.39%
Complexity 1282 1282
=========================================
Files 3359 3359
Lines 207825 207824 -1
Branches 32447 32447
=========================================
+ Hits 133818 133825 +7
+ Misses 63244 63237 -7
+ Partials 10763 10762 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
J-HowHuang
reviewed
Jun 1, 2026
Comment on lines
+74
to
+76
| if (_partialUpsertHandler != null) { | ||
| recordInfoIterator = resolveComparisonTies(recordInfoIterator, _hashFunction); | ||
| } |
Collaborator
There was a problem hiding this comment.
For this method resolveComparisonTies can we try to not construct the entire Map in-memory here just to get the iterator?
We have something similar in BasePartitionDedupMetadataManager#addOrReplaceSegment, and it's creating the iterator properly via some readers without allocating the memory all at once.
Collaborator
There was a problem hiding this comment.
Okay it seems we need to do a deduplication here on all records from recordInfoIterator, so a huge on-heap allocation is inevitable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove the logic of comparison ties to implementations and remove unnecessary methods from the base class.