Build METADATA push metadata from the local segment instead of re-downloading the staged tar

[KKcorps]

Stacked on #19488 (the first commit here is that PR; review the second commit).

TL;DR

Every minion METADATA push uploads the segment tar to the output filesystem and then downloads the
same tar back to extract metadata.properties and creation.meta. This PR builds that two-file
metadata tar from the local converted segment before the segment directory is deleted, and adds
SegmentPushUtils overloads that push caller-supplied metadata tars. Both minion base executors
use them. Wire behavior is unchanged: same headers, same URIs, same controller handling. Minion S3
traffic per pushed segment halves and the full-segment copy in java.io.tmpdir goes away.

The problem

SegmentPushUtils.sendSegmentUriAndMetadata(spec, fileSystem, uriToTarPath, ...) and its batch
twin take a map of staged tar paths and call generateSegmentMetadataFile(fileSystem, tarUri),
which copies the whole tar into java.io.tmpdir, scans the gzip stream twice with untarOneFile,
tars the two entries and deletes the copy. The preferMetadataTarGz shortcut only helps when a
<segment>.metadata.tar.gz sidecar already exists next to the tar, and the minion executors never
write one. So for a task that pushes N segments of size S in METADATA mode, the minion moves
2·N·S bytes: N·S up, N·S back down. For hundreds-of-MB segments that is the dominant cost of the
push step, and the temporary full copy lands on the minion's java.io.tmpdir.

The two files are already on local disk at that point. The executor tars the converted segment
directory and only then deletes it.

The approach

1. SegmentPushUtils.generateSegmentMetadataFile(File segmentDir, File outputDir, String segmentName)
   copies metadata.properties and creation.meta out of a local segment directory (v1/v3 layouts
   via SegmentDirectoryPaths) and tars them as <segmentName>.metadata.tar.gz.
2. sendSegmentUriAndMetadata(spec, Map<String, File> segmentUriToMetadataFile, headers, params)
   and the batch sendSegmentsUriAndMetadata(...) twin push caller-supplied metadata tars. The
   segment name comes from the metadata file name, the caller keeps ownership of the files. The
   existing PinotFS overloads are unchanged in behavior and now share one private push loop with
   the new ones, so the retry, header and batch-tar logic exists once.
3. BaseSingleSegmentConversionExecutor builds the metadata tar right after the conversion for
   every METADATA push (the controller-copy path from Add an opt-in controller-copy METADATA push for same-name segment refreshes #19488 already did), and the default path
   pushes it with the new overload instead of the PinotFS one.
4. BaseMultipleSegmentsConversionExecutor builds one metadata tar per output segment in the tar
   loop, keeps them alongside the tars, pushes them with the new overloads in both the per-segment
   and the batch mode, and deletes them with the tars.

flowchart LR
  subgraph Before["Before"]
    A[tar converted segment] --> B[upload tar to output FS]
    B --> C[download tar back to java.io.tmpdir]
    C --> D[extract 2 files, tar them]
    D --> E[POST metadata]
  end
  subgraph After["After"]
    F[tar converted segment] --> G[copy 2 files from local dir, tar them]
    G --> H[upload segment tar to output FS]
    H --> I[POST metadata]
  end

Loading

Key components

Class / file	Role
SegmentPushUtils	generateSegmentMetadataFile(File, File, String), the two local-metadata push overloads, shared private pushSegmentMetadata / pushSegmentsMetadata loops, getSegmentMetadataFile (sidecar-or-download) used by both PinotFS paths.
BaseSingleSegmentConversionExecutor	Builds the metadata tar for every METADATA push and hands it to the default path.
BaseMultipleSegmentsConversionExecutor	Builds one metadata tar per output segment, pushes with the new overloads. updateSegmentUriToTarPathMap becomes getSegmentUris.
BaseTaskExecutor	createSegmentMetadataTarFile from #19488 moves into SegmentPushUtils.

Compatibility

• Wire behavior is identical. The controller receives the same metadata tar contents, the same
  DOWNLOAD_URI, UPLOAD_TYPE and copy flag, and the same batch tar layout.
• The existing PinotFS overloads keep their signatures and behavior for external callers (the
  ingestion job runners, StarTree's ingestion tasks).
• No new config keys. preferMetadataTarGz still applies to the PinotFS overloads.

Testing

• SegmentPushUtilsTest: local metadata tar contents from a v3 layout (data file excluded); single
  and batch local-metadata pushes against the TLS test server, checking UPLOAD_TYPE,
  DOWNLOAD_URI, the copy flag and that the caller's files survive; rejection of a metadata file
  not named <segment>.metadata.tar.gz.
• BaseSingleSegmentConversionExecutorTest: the default METADATA path pushes a locally built
  metadata tar describing the converted segment through the new overload and never calls the
  PinotFS overload.
• BaseMultipleSegmentsConversionExecutorTest: end-to-end executeTask on METADATA push, per-segment
  and batch, with two output segments: one metadata tar per segment named after it, containing
  exactly the two files and describing that segment; the PinotFS overloads are never called; the
  staged tars stay in the output dir as the download URLs.
• Integration: PurgeMinionClusterIntegrationTest (single executor, METADATA table),
  RealtimeToOfflineSegmentsMinionClusterIntegrationTest#testRealtimeToOfflineSegmentsMetadataPushTask
  and MergeRollupMinionClusterIntegrationTest#testOfflineTableSingleLevelConcatWithMetadataPush
  (multi executor, per-segment and batch) on a local-FS cluster.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y375AgHYsh1YqNSsfvF1a8

[codecov-commenter]

Codecov Report

❌ Patch coverage is 73.84615% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.79%. Comparing base (5e05b9c) to head (3f20c40).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/pinot/segment/local/utils/SegmentPushUtils.java	68.22%	28 Missing and 6 partials ⚠️
...ot/plugin/minion/tasks/SegmentConversionUtils.java	0.00%	7 Missing ⚠️
...ion/tasks/BaseSingleSegmentConversionExecutor.java	88.00%	2 Missing and 4 partials ⚠️
...he/pinot/plugin/minion/tasks/BaseTaskExecutor.java	70.00%	2 Missing and 1 partial ⚠️
.../tasks/BaseMultipleSegmentsConversionExecutor.java	95.23%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #19490       +/-   ##
=============================================
+ Coverage     39.44%   67.79%   +28.35%     
- Complexity     1429     1470       +41     
=============================================
  Files          3488     3488               
  Lines        224424   224507       +83     
  Branches      35429    35437        +8     
=============================================
+ Hits          88528   152211    +63683     
+ Misses       127912    60268    -67644     
- Partials       7984    12028     +4044     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.79% <73.84%> (+28.35%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.79% <73.84%> (+28.35%)	⬆️
unittests	67.79% <73.84%> (+28.35%)	⬆️
unittests1	57.75% <0.00%> (?)
unittests2	39.53% <73.84%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.