Skip to content

fix(snapshot/cdc): keyset pagination, cursor resume, per-pipeline offsets, rollback on cancel - #47

Merged
lekhrocks merged 1 commit into
mainfrom
fix/snapshot-cdc
Aug 6, 2026
Merged

fix(snapshot/cdc): keyset pagination, cursor resume, per-pipeline offsets, rollback on cancel#47
lekhrocks merged 1 commit into
mainfrom
fix/snapshot-cdc

Conversation

@lekhrocks

Copy link
Copy Markdown
Owner

Summary

Hardens snapshot reading and CDC change capture. Fixes pagination correctness under concurrent writes, resume accuracy, per-pipeline offset isolation, and commit/rollback semantics on cancel.

Snapshot

  • Keyset pagination over a single-column PK (WHERE pk > cursor ORDER BY pk LIMIT size) replaces OFFSET — stable under concurrent writes, single-pass, no offset drift or row duplication. Falls back to OFFSET when no single-column PK exists.
  • SQL injection guard: the PK identifier is validated against [A-Za-z_][A-Za-z0-9_]* before interpolation.
  • Cursor-driven resume: the keyset cursor is threaded through the read loop and checkpointed, so a resume continues at the exact next row (was recomputing the offset from a batch counter, which drifted on partial batches).
  • Row estimation: uses the Postgres reltuples planner estimate instead of a full SELECT COUNT(*); falls back to COUNT on non-Postgres.
  • Progress: aggregates row/batch estimates across all mapped tables (was first-table-only).
  • Rollback on cancel/failure: writer.rollback() for cancelled/failed jobs so partial writes are not committed — a later resume would otherwise duplicate rows. Commit happens only on success.

CDC

  • Per-pipeline offset file (host_db_pipeline): multiple pipelines on the same database no longer share/corrupt the Debezium position. CaptureLifecycle passes pipelineId in the connector context.
  • Per-table LSN retention: updateOffset merges instead of replaces.
  • Test reset path aligned to the keyed offset filename.

Validation

  • All 11 CDC integration tests pass
  • Full ./gradlew test green

…sets, rollback on cancel

Snapshot:
- Keyset pagination over a single-column PK (WHERE pk > cursor ORDER BY pk
  LIMIT size) instead of OFFSET — stable under concurrent writes, no O(n^2) or
  offset drift; falls back to OFFSET when no single-column PK
- PK identifier sanitized before SQL interpolation (injection guard)
- Cursor is carried through the read loop and checkpointed, so resume continues
  at the exact next row (no partial-batch offset drift)
- estimateRows prefers the Postgres reltuples planner estimate over a full
  SELECT COUNT(*); falls back to COUNT on non-Postgres
- Progress aggregates row/batch estimates across ALL mapped tables
- writer.rollback() on cancel/failure so partial writes are not committed
  (a later resume would otherwise duplicate rows); commit only on success

CDC:
- Offset file keyed per-pipeline (host_db_pipeline) so multiple pipelines on
  the same database do not share/corrupt Debezium position; CaptureLifecycle
  passes pipelineId in the connector context
- updateOffset merges instead of replaces, retaining per-table LSNs for
  multi-table captures
- CdcIntegrationTest reset path matches the new keyed offset filename
@lekhrocks lekhrocks self-assigned this Aug 6, 2026
@lekhrocks
lekhrocks merged commit 47a3d8d into main Aug 6, 2026
18 checks passed
@lekhrocks
lekhrocks deleted the fix/snapshot-cdc branch August 6, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant