[Improvement] Reuse primary-key sorted index after a source file is retired

[wangyong9999]

Paimon version

Reproduced on master at 052adc366f63adc613dce01e2f1e7fec82f3c449; the fix branch is rebased onto d904d000091339b4b030bb581a9e4ebd45785b45.

Compute Engine

Java core and Python scan planning. The issue is engine-independent and affects readers using the source-backed primary-key sorted index plan.

Minimal reproduce step

1. Build one primary-key sorted-index payload for two compacted source files at the same positive data level. The payload metadata keeps the ordered source group [A, B] and its row positions use that group-wide ordinal space.
2. Advance to a snapshot where A has been retired while B remains active at the same data level. Keep the payload active because it still covers B.
3. Plan an indexed predicate scan for B.

Java PkSortedBucketIndexState.fromActiveDataFiles and Python PkSortedBucketIndexState.from_active_data_files currently compare the current active level sources [B] with the immutable payload sources [A, B] using exact list equality. They reject the payload, so B falls back to a raw scan.

What doesn't meet your expectations?

Retiring one source file should not invalidate index coverage for the other active files in the same immutable payload group.

The reader should:

• retain the payload's complete ordered source list for group-ordinal localization;
• accept the payload when it has a non-empty intersection with eligible active files at the metadata-declared level and matching file row counts;
• mark only that active intersection as covered;
• leave newly active files not listed by the payload uncovered so they are scanned normally; and
• continue to reject malformed, ambiguous, wrong-level, or zero-active-source payloads.

This changes neither the persisted metadata format nor writer behavior. The aligned Java/Python fix is in apache/paimon#9460. Apache Paimon C++ has the same reader-side issue and its corresponding fix is under review in apache/paimon-cpp#253.

I searched existing issues and pull requests and found no report of this lifecycle case.

Are you willing to submit a PR?

Yes. apache/paimon#9460 contains the focused Java/Python fix with state-planning, row-offset localization, raw fallback, and cross-level binding tests.