Skip to content

Optimizes SpillingGrouper for high cardinality dimension(s) group by with large memory footprint aggregators#19357

Draft
maytasm wants to merge 6 commits intoapache:masterfrom
maytasm:spill_file_improvement
Draft

Optimizes SpillingGrouper for high cardinality dimension(s) group by with large memory footprint aggregators#19357
maytasm wants to merge 6 commits intoapache:masterfrom
maytasm:spill_file_improvement

Conversation

@maytasm
Copy link
Copy Markdown
Contributor

@maytasm maytasm commented Apr 21, 2026

Fixes #XXXX.

Description

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

Release note

Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

github-advanced-security[bot]
github-advanced-security AI found potential problems Apr 21, 2026
View reviewed changes
Comment thread processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/SpillingGrouper.java
try {
for (final byte[] runBytes : pendingSpillRuns) {
readers.add(spillMapper.readValues(
spillMapper.getFactory().createParser(new LZ4BlockInputStream(new ByteArrayInputStream(runBytes))),
@maytasm maytasm changed the title Optimizes SpillingGrouper for high cardinality group by dimensions Optimizes SpillingGrouper for high cardinality group by dimensions with large memory footprint aggregators Apr 21, 2026
@maytasm maytasm changed the title Optimizes SpillingGrouper for high cardinality group by dimensions with large memory footprint aggregators Optimizes SpillingGrouper for high cardinality dimension(s) group by with large memory footprint aggregators Apr 22, 2026
FrankChen021
FrankChen021 approved these changes Apr 25, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes LGTM, no correctness issues found.

@maytasm maytasm marked this pull request as draft April 25, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FrankChen021 FrankChen021 FrankChen021 approved these changes

+1 more reviewer

@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@maytasm @FrankChen021 @github-advanced-security