Skip to content

Add --defer-validate-fks to create FK constraints as NOT VALID#37

Merged
teknogeek0 merged 1 commit into
mainfrom
feature/defer-validate-fks
Jun 8, 2026
Merged

Add --defer-validate-fks to create FK constraints as NOT VALID#37
teknogeek0 merged 1 commit into
mainfrom
feature/defer-validate-fks

Conversation

@teknogeek0

Copy link
Copy Markdown
Collaborator

Problem

For large migrations, foreign-key validation can take hours or even days. Because FK creation happens before the transition into CDC, that validation scan blocks the migration from reaching streaming mode — and while it runs, the replication slot accumulates a large WAL backlog that single-threaded CDC replay must then chew through, delaying completion further.

Solution

Add --defer-validate-fks. When set, pgcopydb creates the target foreign keys as NOT VALID, skipping the one-time validation scan of pre-existing rows so the COPY → CDC handoff happens immediately. The operator validates the constraints later, on their own schedule, with ALTER TABLE ... VALIDATE CONSTRAINT.

This is safe for CDC: NOT VALID still enforces referential integrity on every new write — it only skips the historical-row scan. Any change replayed from the source already satisfied the source's constraint, so it satisfies the target's NOT VALID constraint too. FK creation still completes before CDC apply is enabled, so enforcement is in place from the first replayed change.

Hardening for the common online-migration flag combination

This was reviewed and tested specifically for clone --follow --defer-indexes --defer-validate-fks --split-tables-larger-than:

  • Resume consistency: the flag is persisted in the catalog setup table and a resume that changes it is rejected — preventing a resume from silently creating the remaining FKs VALID and triggering the scan the flag exists to avoid.
  • Missing referenced unique constraint (SQLSTATE 42830): e.g. a filtered-out unique index on the referenced table. The FK is skipped and captured in an end-of-run summary instead of aborting the migration.
  • Partitioned parents (SQLSTATE 42809): PostgreSQL forbids NOT VALID foreign keys declared on a partitioned table. Previously this would abort the whole migration. Now the parent-level constraint is skipped with a captured warning (per-partition FK rows are still created NOT VALID), so the partition data is still enforced and deferred without blocking.
  • Cutover reminder: the deferred-FK validation reminder is re-emitted at follow/cutover completion so it is not buried mid-log.

Tests

  • tests/defer-validate-fks — clone path: validated FKs land NOT VALID, source-NOT VALID FKs stay NOT VALID, data fully copied, violating writes still rejected, manual VALIDATE succeeds, and the partitioned-parent constraint is skipped without aborting.
  • tests/follow-defer-validate-fks — the full --follow --defer-indexes --defer-validate-fks --split-tables-larger-than combination with DML injected during CDC; asserts all FKs land NOT VALID, data stays consistent after replay, enforcement holds, and deferred validation completes.

Registered in tests/Makefile and the tiered CI workflow. Full local suite (PG16) passes.

Creates foreign keys on the target as NOT VALID so the validation scan
(which can take hours/days on large tables) does not block the transition
into CDC. The operator validates them later with ALTER TABLE ... VALIDATE
CONSTRAINT. NOT VALID still enforces referential integrity on new writes, so
CDC replay stays safe: every change replayed from the source already
satisfies the constraint there.

Hardening for the common online-migration flag combination
(clone --follow --defer-indexes --defer-validate-fks --split-tables-larger-than):

- Persist the flag in the catalog setup table and reject a resume that
  changes it, so a resume cannot silently create the remaining FKs VALID and
  trigger the scan the flag exists to avoid.
- Tolerate SQLSTATE 42830 (no matching unique/PK constraint on the referenced
  table, e.g. a filtered-out unique index): skip and capture rather than
  aborting the migration.
- Handle SQLSTATE 42809 on partitioned parents (PostgreSQL forbids NOT VALID
  FKs there): skip the parent-level constraint with a captured warning instead
  of aborting; per-partition FK rows are still created NOT VALID.
- Re-emit the deferred-FK validation reminder at follow/cutover completion.

Adds tests/defer-validate-fks (clone, incl. partitioned-parent skip) and
tests/follow-defer-validate-fks (the full --follow combination with CDC DML
injection); registered in tests/Makefile and CI.
@teknogeek0 teknogeek0 merged commit f227342 into main Jun 8, 2026
87 checks passed
@teknogeek0 teknogeek0 deleted the feature/defer-validate-fks branch June 8, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant