Skip to content

feat(snapshot): add snapshot file listing API - #270

Open
lxy-9602 wants to merge 3 commits into
apache:mainfrom
lxy-9602:file-list
Open

feat(snapshot): add snapshot file listing API#270
lxy-9602 wants to merge 3 commits into
apache:mainfrom
lxy-9602:file-list

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Sep 1, 2026

Copy link
Copy Markdown
Member

Purpose

Linked issue: N/A

Add a public SnapshotFileScan::ListFiles API to list the physical files required to scan and read a selected table snapshot.

The API supports:

  • an optional snapshot ID, using the latest snapshot when omitted;
  • an explicit branch;
  • partition and bucket filters through ScanFilter;
  • custom table options, file system, executor, and memory pool;
  • deterministic and deduplicated results through std::set<std::string>.

The returned files include:

  • the selected snapshot file;
  • the latest schema and schemas referenced by the snapshot data files;
  • base, delta, and changelog manifest lists;
  • manifest files that cannot be pruned using manifest-level partition and bucket statistics;
  • live data and changelog files;
  • data-file external indexes and extra files;
  • index manifests and live deletion-vector/global-index files;
  • statistics and real-time offset files when present.

Mutable snapshot hints such as LATEST and EARLIEST are intentionally excluded.

Predicate filters are rejected because listing files with arbitrary data predicates could incorrectly omit files required by scan or read.

The implementation reuses the existing manifest-entry merge semantics from FileStoreScan so that deleted and superseded data files are not returned. Manifest pruning is aligned with the regular scan path: a manifest is included whenever scan needs to open it, even if entry-level filtering later removes all of its entries.

Tests

Added SnapshotFileScanTest coverage for:

  • latest and explicit snapshots;
  • exact physical file sets;
  • latest-schema inclusion for an older snapshot;
  • partition and bucket filtering;
  • manifest-level versus entry-level filtering;
  • filters with no matching partitions;
  • external data paths;
  • external data-file indexes;
  • deletion-vector indexes;
  • global indexes;
  • branch snapshots and branch schemas;
  • invalid snapshot IDs and unsupported predicate filters.

API and Format

This change introduces the public header:

include/paimon/snapshot/snapshot_file_scan.h

The API is exported through paimon/api.h.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

@lxy-9602

lxy-9602 commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@ljyss9 PTAL, thanks!

@ljyss9

ljyss9 commented Sep 2, 2026

Copy link
Copy Markdown

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants