Skip to content

refactor(realtime): move offset handling to Paimon readers - #269

Open
lxy-9602 wants to merge 1 commit into
apache:mainfrom
lxy-9602:refactor-rt
Open

refactor(realtime): move offset handling to Paimon readers#269
lxy-9602 wants to merge 1 commit into
apache:mainfrom
lxy-9602:refactor-rt

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158

This PR moves real-time offset handling from RealtimeStore plugins into the Paimon read and write layers.

The main changes are:

  • Remove offset_begin from RealtimeStore::CreateQueryReaders. Store plugins now return candidate rows without applying visibility offsets.
  • Remove enable_predicate_pushdown from RealtimeQueryContext. Callers decide whether a predicate should be passed to the plugin.
  • Add _REALTIME_OFFSET to append-mode transport batches before they are handed to the real-time store.
  • Add RealtimeOffsetBatchReader to filter rows by the visible half-open offset range and remove _REALTIME_OFFSET from query output.
  • Reuse RealtimeOffsetBatchReader and KeyValueDataFileRecordReader for primary-key real-time reads.
  • Generalize KeyValueDataFileRecordReader to accept a BatchReader. File row positions remain available when the wrapped reader is a FileBatchReader.
  • Resolve primary-key transport fields by name, making their field order insignificant and removing unnecessary upfront schema and offset coverage validation.
  • Simplify the built-in Arrow real-time store query reader to return candidate batches with an all-valid bitmap.

Tests

Added RealtimeOffsetBatchReaderTest.

API and Format

This PR changes the public real-time store plugin API:

  • RealtimeStore::CreateQueryReaders no longer accepts offset_begin.
  • RealtimeQueryContext no longer contains enable_predicate_pushdown.
  • Append-mode transport batches now contain a leading _REALTIME_OFFSET field.
  • Primary-key transport fields are resolved by name rather than a fixed order.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant