Prune segments from column metadata instead of materializing them

[xiangfu0]

What

ColumnValueSegmentPruner asks each segment for a column's DataSource and then reads nothing from it but the DataSourceMetadata — data type, min/max, partition function. Reaching that metadata through getDataSource forces a segment that builds its columns lazily to construct the whole ColumnIndexContainer — every index reader for the column — for a segment it is about to discard.

This adds IndexSegment#getDataSourceMetadata(String, Schema), which names what the caller actually wants and defaults to getDataSource(column, schema).getDataSourceMetadata(), so no existing implementation has to change. ImmutableSegmentImpl overrides it to answer from column metadata: ImmutableDataSourceMetadata already delegates to ColumnMetadata and holds no readers, so nothing needs building. A column the segment does not have still falls through to the data source, which is where the schema-driven default and virtual columns are created.

The pruner keeps its per-segment data-source cache for mutable segments, whose metadata is not derivable without the data source.

Why

ValueBasedSegmentPruner#prune is a serial loop over every segment the server holds, on the query thread — it implements only the two-arg prune, so SegmentPruner's three-arg default silently discards the ExecutorService (BloomFilterSegmentPruner, by contrast, does override it and prunes in parallel).

Combined with lazy column materialization (#19477) that means the pruner materializes a column per held segment, and on a tiered-storage external table each materialization parses a Parquet footer. Measured on a production server holding 44,780 segments, this dominated a query that timed out at 300 s for a filter matching 1,839 segments — the pruner was doing tens of thousands of footer parses, in series, for segments it then discarded.

The win is not limited to lazy mode: building an index container to read min/max is wasted work in any configuration.

Tests

ImmutableSegmentImplTest#testDataSourceMetadataDoesNotMaterializeTheColumn builds a lazy segment with a mocked ColumnMaterializer, reads the metadata, and asserts verifyNoInteractions(materializer) — then shows that asking for the data source does materialize, and that the two agree. It fails when the override is removed.

51 pruner tests in pinot-core and 35 segment tests in pinot-segment-local pass; spotless, checkstyle and license clean.

Stack

Part 11, based on #19486. Review only this part's own commit; the earlier parts account for the rest of the diff.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.82%. Comparing base (1bad6fd) to head (9a69689).

Files with missing lines	Patch %	Lines
...l/indexsegment/immutable/ImmutableSegmentImpl.java	33.33%	1 Missing and 1 partial ⚠️
...ot/core/query/pruner/ColumnValueSegmentPruner.java	88.88%	0 Missing and 1 partial ⚠️
...ava/org/apache/pinot/segment/spi/IndexSegment.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                                Coverage Diff                                @@
##             xiangfu0/data-3221-10-physical-column-names   #19511      +/-   ##
=================================================================================
+ Coverage                                          57.80%   57.82%   +0.01%     
  Complexity                                             7        7              
=================================================================================
  Files                                               2692     2692              
  Lines                                             164721   164723       +2     
  Branches                                           26757    26753       -4     
=================================================================================
+ Hits                                               95212    95244      +32     
+ Misses                                             61466    61435      -31     
- Partials                                            8043     8044       +1     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (?)
java-25	57.82% <71.42%> (+0.01%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	57.82% <71.42%> (+0.01%)	⬆️
unittests	57.81% <71.42%> (+0.01%)	⬆️
unittests1	57.81% <71.42%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.