Add a running XOR key digest to upsert snapshot metadata

[KKcorps]

Stacked on #19515. Only the last commit is new. The diff view includes #19515 until that PR merges.

TL;DR

Two replicas of an upsert partition can hold the same segments, the same valid-doc counts, and still
disagree on which row is current or on its comparison value. This PR adds a running XOR digest of the
live primary-key map to the snapshot report from #19515. Two replicas at the same stream boundary hold
the same digest exactly when they hold the same set of (primary key, comparison value) entries. It is
opt-in behind the existing enableSnapshotMetadata flag and costs one hash and two XORs per map write.

The problem

The saved-bitmap fingerprints in #19515 compare bytes on disk. They need every segment to have a saved
file, they say nothing about comparison values, and a mismatch still needs a logical bitmap compare to
confirm. The count-based consensus checks miss the same-count-different-rows shape entirely.

flowchart LR
  subgraph Before["❌ Today"]
    A[replica A: key k at cv 120] --> C[equal counts, equal bitmaps]
    B[replica B: key k at cv 100] --> C
    C --> D[no signal]
  end
  subgraph After["✅ With this PR"]
    E[replica A: XOR includes h k,120] --> G[digests differ at the same boundary]
    F[replica B: XOR includes h k,100] --> G
    G --> H[bucket names 1/256th of the key space]
  end

Loading

The approach

1. Each partition keeps 256 long buckets. Bucket = top byte of hash64(storedKey).
2. Every live entry that is not a tombstone contributes hash64(storedKey, comparisonValue) to its bucket.
   An update XORs the old contribution out and the new one in. A removal XORs out. A delete record XORs
   the old contribution out and adds nothing.
3. Segment and docId are not hashed. A commit swap, reload or refresh that lands the same rows leaves the
   digest unchanged.
4. Segment-level operations on Helix threads (add, preload, replace, remove) run inside a seqlock window.
   The snapshot capture freezes the buckets at start and end and publishes them only if both reads were
   outside any window and nothing changed in between.
5. The report gains a keyDigest block: algorithm, total, base64 buckets, entry count, stable flag.

Key components

Class / file	Role
UpsertKeyDigest	256 XOR buckets, entry counter, seqlock, typed hashing of keys and comparison values, freeze
UpsertSnapshotMetadata.KeyDigest	Report block, built from the start and end marks of one capture
UpsertSnapshotDiagnostics	Freezes the digest at capture begin and finish
BasePartitionUpsertMetadataManager	Owns the digest, opens the seqlock window around segment-level operations
ConcurrentMapPartitionUpsertMetadataManager	Calls the digest at every map write and removal; RecordLocation carries the tombstone flag
docs/upsert-snapshot-metadata.md	New "Key digest" section

Flow

A consuming record

sequenceDiagram
  participant C as Consumer thread
  participant M as ConcurrentMapPartitionUpsertMetadataManager
  participant K as ConcurrentHashMap
  participant D as UpsertKeyDigest
  C->>M: addRecord(segment, recordInfo)
  M->>K: compute(storedKey)
  alt new key
    K-->>M: no entry
    M->>D: add(storedKey, cv, isDelete)
  else newer comparison value
    K-->>M: current location
    M->>D: update(storedKey, oldCv, oldIsDelete, cv, isDelete)
  else out of order
    K-->>M: current location
    Note over M,D: no map change, no digest change
  end

Loading

A snapshot capture

sequenceDiagram
  participant C as Consumer thread
  participant B as BasePartitionUpsertMetadataManager
  participant S as UpsertSnapshotDiagnostics
  participant D as UpsertKeyDigest
  participant H as Helix thread
  C->>B: takeSnapshot(consumingSegment, startOffset)
  B->>S: begin
  S->>D: freeze (epoch, depth, copy, depth, epoch)
  B->>B: write bitmap files
  opt segment add / replace / remove
    H->>D: beginUnstable
    H->>D: add / update / remove per key
    H->>D: endUnstable
  end
  B->>S: finish
  S->>D: freeze
  alt both marks stable and equal
    S-->>B: keyDigest with total and buckets
  else
    S-->>B: keyDigest with stable=false, total and buckets null
  end

Loading

What the digest ignores on purpose

• Tombstones. A key whose latest record is a delete contributes nothing. The deletedKeysTTL sweep only
  removes tombstones, so it never touches the digest. A swallowed delete still shows: one replica removed
  the old contribution, the other still holds it.
• Segment and docId. Replicas assign docIds by build order and segment ids by load order. Hashing
  them would flag every reload.
• The consuming segment's timing. The digest includes consuming-segment keys, so compare only reports
  with the same consumingSegmentName and startOffset, taken at the commit boundary.

Sweeps

The on-heap manager applies the metadataTTL sweep to the digest. Its sweep runs at the commit boundary
on every replica, and its map is rebuilt from segments on restart, so the digest must describe the live
map. One capture after a restart on a metadataTTL table can differ until the first sweep runs. A manager
whose sweep runs on a timer must keep the sweep off the digest and persist the buckets instead. That is
the RocksDB companion, not this PR.

Configuration

No new keys. The digest exists only when upsertConfig.metadataManagerConfigs.enableSnapshotMetadata
is true, same as the rest of the report. When it is off, every hook is one null check.

Performance considerations

• Per map write: one XXH64 over the stored key bytes, one over the comparison value, one SplitMix64
  finalizer, one atomic XOR on a long, one LongAdder bump. For hashFunction=NONE the key hash calls
  PrimaryKey.asBytes(), which allocates once per write. Hashed key functions use the stored 16 bytes.
• Per capture: two copies of 256 longs and one 2 KB base64 string in the report and sidecar.
• State: 2 KB of buckets per partition. Nothing per key.
• RecordLocation does not grow. The tombstone flag rides in the sign bit of the docId.

Compatibility notes

• UpsertSnapshotMetadata.FORMAT_VERSION moves from 3 to 4. Version 3 sidecars read as unavailable.
  Version 3 never shipped.
• RecordLocation gains a four-argument constructor and isDeleteRecord(). The three-argument
  constructor stays and means "not a delete".
• UpsertSnapshotDiagnostics takes the digest in its package-private constructor.
• No change to PartitionUpsertMetadataManager or TableUpsertMetadataManager.

Validation

• UpsertKeyDigestTest: order independence, add/remove/update algebra, tombstones contribute zero,
  distinct hashes across comparison-value types, seqlock stability, report gating.
• ConcurrentMapPartitionUpsertMetadataManagerTest: two replicas fed the same records agree; a missed
  update and a swallowed delete each break agreement until applied; an out-of-order record and a segment
  replace leave the digest unchanged; segment removal removes the owned key on both; a fresh manager fed
  only the surviving rows lands on the same digest; the metadataTTL sweep removes swept keys from the
  digest.
• BasePartitionUpsertMetadataManagerTest: the report carries a stable zero digest, and a segment
  operation in flight publishes stable=false with null total and buckets.
• UpsertSnapshotMetadataStoreTest: round trip, version 3 rejected.
• TablesResourceTest passes with the new report shape (19 tests).
• All 120 tests under pinot-segment-local upsert pass. Spotless and Checkstyle pass.

Release notes

Adds an opt-in per-partition XOR digest of upsert primary keys and comparison values to the snapshot
metadata report.

Related: #19515, #19499. Documentation: docs/upsert-snapshot-metadata.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_016sih7cgUT7mAFHtTYQq1mk

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.68442% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.78%. Comparing base (c611609) to head (4b55ac7).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...cal/upsert/BasePartitionUpsertMetadataManager.java	65.16%	17 Missing and 14 partials ⚠️
...t/ConcurrentMapPartitionUpsertMetadataManager.java	68.51%	13 Missing and 4 partials ⚠️
...egment/local/upsert/UpsertSnapshotDiagnostics.java	83.33%	4 Missing and 8 partials ⚠️
...l/indexsegment/immutable/ImmutableSegmentImpl.java	76.19%	6 Missing and 4 partials ⚠️
...he/pinot/segment/local/upsert/UpsertKeyDigest.java	89.58%	4 Missing and 6 partials ⚠️
...egment/local/upsert/UpsertSnapshotFingerprint.java	91.17%	4 Missing and 2 partials ⚠️
...che/pinot/server/api/resources/TablesResource.java	84.84%	1 Missing and 4 partials ⚠️
...t/local/upsert/PartitionUpsertMetadataManager.java	0.00%	3 Missing ⚠️
...t/segment/local/upsert/UpsertSnapshotMetadata.java	85.00%	1 Missing and 2 partials ⚠️
...n/restlet/resources/ValidDocIdsBitmapResponse.java	60.00%	2 Missing ⚠️
... and 4 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19525      +/-   ##
============================================
+ Coverage     67.74%   67.78%   +0.03%     
- Complexity     1424     1450      +26     
============================================
  Files          3489     3496       +7     
  Lines        224672   225482     +810     
  Branches      35468    35633     +165     
============================================
+ Hits         152210   152835     +625     
- Misses        60445    60547     +102     
- Partials      12017    12100      +83     

Flag	Coverage Δ
integration	100.00% <ø> (+100.00%)	⬆️
integration1	100.00% <ø> (?)
integration2	0.00% <ø> (ø)
java-25	67.78% <79.68%> (+0.03%)	⬆️
lane-a	100.00% <ø> (+100.00%)	⬆️
lane-b	0.00% <ø> (ø)
temurin	67.78% <79.68%> (+0.03%)	⬆️
unittests	67.77% <79.68%> (+0.03%)	⬆️
unittests1	57.65% <1.26%> (-0.16%)	⬇️
unittests2	39.59% <79.68%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[KKcorps]

Closing. The digest is being built in the StarTree RocksDB upsert manager instead, where the write sites, value format and persistence live. This branch stays as an on-heap reference.