fix(snapshot/cdc): keyset pagination, cursor resume, per-pipeline offsets, rollback on cancel - #47
Merged
Merged
Conversation
…sets, rollback on cancel Snapshot: - Keyset pagination over a single-column PK (WHERE pk > cursor ORDER BY pk LIMIT size) instead of OFFSET — stable under concurrent writes, no O(n^2) or offset drift; falls back to OFFSET when no single-column PK - PK identifier sanitized before SQL interpolation (injection guard) - Cursor is carried through the read loop and checkpointed, so resume continues at the exact next row (no partial-batch offset drift) - estimateRows prefers the Postgres reltuples planner estimate over a full SELECT COUNT(*); falls back to COUNT on non-Postgres - Progress aggregates row/batch estimates across ALL mapped tables - writer.rollback() on cancel/failure so partial writes are not committed (a later resume would otherwise duplicate rows); commit only on success CDC: - Offset file keyed per-pipeline (host_db_pipeline) so multiple pipelines on the same database do not share/corrupt Debezium position; CaptureLifecycle passes pipelineId in the connector context - updateOffset merges instead of replaces, retaining per-table LSNs for multi-table captures - CdcIntegrationTest reset path matches the new keyed offset filename
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens snapshot reading and CDC change capture. Fixes pagination correctness under concurrent writes, resume accuracy, per-pipeline offset isolation, and commit/rollback semantics on cancel.
Snapshot
WHERE pk > cursor ORDER BY pk LIMIT size) replaces OFFSET — stable under concurrent writes, single-pass, no offset drift or row duplication. Falls back to OFFSET when no single-column PK exists.[A-Za-z_][A-Za-z0-9_]*before interpolation.reltuplesplanner estimate instead of a fullSELECT COUNT(*); falls back to COUNT on non-Postgres.writer.rollback()for cancelled/failed jobs so partial writes are not committed — a later resume would otherwise duplicate rows. Commit happens only on success.CDC
host_db_pipeline): multiple pipelines on the same database no longer share/corrupt the Debezium position.CaptureLifecyclepassespipelineIdin the connector context.updateOffsetmerges instead of replaces.Validation
./gradlew testgreen