Record upsert snapshot outcomes and saved content fingerprints

[KKcorps]

PR flow

Records snapshot outcomes and segment fingerprint metadata during upsert snapshots.

flowchart TD
  N0["Start snapshot with context #40;F4#41;"]:::stAdded
  N1["Begin diagnostic capture #40;F8#41;"]:::stAdded
  N2["Process segments#58; track#44; update counters#44; persist snapshots #40;F4#44; F3#41;"]:::stModified
  N3["Finish capture#58; build attempt#44; collect content#44; update counters #40;F8#41;"]:::stAdded
  N4["Publish metadata#58; update latest and persist #40;F8#41;"]:::stAdded
  N5["Persist metadata to sidecar file #40;F11#41;"]:::stAdded
  N0 -->|"calls"| N1
  N1 -->|"uses capture"| N2
  N2 -->|"completes attempt"| N3
  N3 -->|"calls publish"| N4
  N4 -->|"calls persist"| N5
  classDef stAdded fill:#dafbe1,stroke:#1a7f37,color:#1f2328,stroke-width:2px
  classDef stModified fill:#fff8c5,stroke:#9a6700,color:#1f2328,stroke-width:2px
  classDef stRemoved fill:#ffebe9,stroke:#cf222e,color:#1f2328,stroke-width:2px
  classDef stUnchanged fill:#f6f8fa,stroke:#656d76,color:#1f2328,stroke-width:1px

Loading

AI-generated · Green: added · Yellow: modified · Red: removed · Gray: existing

Diff evidence

• F3: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java — before · after
• F4: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java — before · after
• F8: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/UpsertSnapshotDiagnostics.java — after
• F11: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/UpsertSnapshotMetadataStore.java — after

• Regenerate PR flow

Summary

An upsert snapshot can skip a segment lock, and a later successful attempt can hide that event. Saved bitmap counts can also match while their document memberships differ. This adds opt-in producer evidence for those two cases: snapshot outcomes and saved-content fingerprints.

The implementation follows four steps:

1. Count selected, written, directly lock-skipped, deferred, and failed segments in the existing snapshot loops.
2. Cache hashes of bytes already written or loaded, including retained unchanged snapshot files.
3. Aggregate those contributions and publish one compact partition report, both live and in a local sidecar.
4. Expose the report and optional raw-file hashes through the server APIs for a future checker.

Cumulative counters are scoped to one manager runtime so a poller can observe earlier failures. Cleanup is immutable before/after data, with current cleanup exposed separately; concurrent snapshots have one explicit flag. There is no provisional lifecycle/configuration comparison machinery.

Scope and cost

upsertConfig.metadataManagerConfigs.enableSnapshotMetadata defaults to false. No per-record hooks, key scans, extra bitmap serialization, blocking locks, or predecessor waits are added. Enabled captures pay for hashing, sorting/aggregation, two optional cached hashes per immutable segment, and synchronous compact sidecar publication.

The payload does not grow with the segment inventory. Missing hashes remain unknown. Raw Roaring bytes are not canonical membership, so a mismatch requires logical bitmap comparison after binding fetched files to the target report. boundaryStatus remains UNVERIFIED; shared source-boundary attestation and controller comparison/alerts are follow-ups. Version 3 rejects earlier draft sidecars. Existing no-argument snapshot overrides and bitmap response constructors are preserved.

Validation

• 79 tests passed across the snapshot manager, fingerprint, sidecar, immutable segment, realtime startup, and server API classes. This includes direct lock skips versus deferred work, retained files, stale/overlapping writes, nested snapshots, publication failure, and old/unknown format rejection.
• The companion PR passed seven native RocksDB cleanup/TTL regressions against this OSS sibling build.
• Spotless, Checkstyle, license format/check, normal reactor compilation, and git diff --check passed.
• The prescribed compiler warning/deprecation check failed, including a smaller diagnostic retry, at unchanged ZstandardDecompressor.java:51: org.jetbrains.annotations.NotNull is absent from the zstd compilation classpath. No warning matched added lines before failure; this check is not claimed as passed.

Release notes

Adds optional upsert snapshot outcome counters, saved-content fingerprints, and server APIs for diagnostics.

Related: #19499. Documentation: docs/upsert-snapshot-metadata.md. RocksDB cleanup companion: startreedata/startree-pinot#4290.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.64350% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.74%. Comparing base (c611609) to head (50ec0dc).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...cal/upsert/BasePartitionUpsertMetadataManager.java	55.55%	18 Missing and 14 partials ⚠️
...l/indexsegment/immutable/ImmutableSegmentImpl.java	76.19%	6 Missing and 4 partials ⚠️
...egment/local/upsert/UpsertSnapshotDiagnostics.java	85.07%	4 Missing and 6 partials ⚠️
...egment/local/upsert/UpsertSnapshotFingerprint.java	91.17%	4 Missing and 2 partials ⚠️
...che/pinot/server/api/resources/TablesResource.java	84.84%	1 Missing and 4 partials ⚠️
...t/local/upsert/PartitionUpsertMetadataManager.java	0.00%	3 Missing ⚠️
...n/restlet/resources/ValidDocIdsBitmapResponse.java	60.00%	2 Missing ⚠️
...t/segment/local/upsert/UpsertSnapshotMetadata.java	87.50%	1 Missing and 1 partial ⚠️
...a/manager/realtime/RealtimeSegmentDataManager.java	50.00%	1 Missing ⚠️
...psert/ConcurrentMapTableUpsertMetadataManager.java	0.00%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19515      +/-   ##
============================================
- Coverage     67.74%   67.74%   -0.01%     
- Complexity     1424     1450      +26     
============================================
  Files          3489     3495       +6     
  Lines        224672   225322     +650     
  Branches      35468    35603     +135     
============================================
+ Hits         152210   152636     +426     
- Misses        60445    60619     +174     
- Partials      12017    12067      +50     

Flag	Coverage Δ
integration	100.00% <ø> (+100.00%)	⬆️
integration1	100.00% <ø> (?)
integration2	0.00% <ø> (ø)
java-25	67.74% <77.64%> (-0.01%)	⬇️
lane-a	100.00% <ø> (+100.00%)	⬆️
lane-b	0.00% <ø> (ø)
temurin	67.74% <77.64%> (-0.01%)	⬇️
unittests	67.73% <77.64%> (-0.01%)	⬇️
unittests1	57.71% <1.67%> (-0.10%)	⬇️
unittests2	39.52% <77.64%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.