Add --defer-validate-fks to create FK constraints as NOT VALID#37
Merged
Conversation
Creates foreign keys on the target as NOT VALID so the validation scan (which can take hours/days on large tables) does not block the transition into CDC. The operator validates them later with ALTER TABLE ... VALIDATE CONSTRAINT. NOT VALID still enforces referential integrity on new writes, so CDC replay stays safe: every change replayed from the source already satisfies the constraint there. Hardening for the common online-migration flag combination (clone --follow --defer-indexes --defer-validate-fks --split-tables-larger-than): - Persist the flag in the catalog setup table and reject a resume that changes it, so a resume cannot silently create the remaining FKs VALID and trigger the scan the flag exists to avoid. - Tolerate SQLSTATE 42830 (no matching unique/PK constraint on the referenced table, e.g. a filtered-out unique index): skip and capture rather than aborting the migration. - Handle SQLSTATE 42809 on partitioned parents (PostgreSQL forbids NOT VALID FKs there): skip the parent-level constraint with a captured warning instead of aborting; per-partition FK rows are still created NOT VALID. - Re-emit the deferred-FK validation reminder at follow/cutover completion. Adds tests/defer-validate-fks (clone, incl. partitioned-parent skip) and tests/follow-defer-validate-fks (the full --follow combination with CDC DML injection); registered in tests/Makefile and CI.
This was referenced Jun 8, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
For large migrations, foreign-key validation can take hours or even days. Because FK creation happens before the transition into CDC, that validation scan blocks the migration from reaching streaming mode — and while it runs, the replication slot accumulates a large WAL backlog that single-threaded CDC replay must then chew through, delaying completion further.
Solution
Add
--defer-validate-fks. When set, pgcopydb creates the target foreign keys asNOT VALID, skipping the one-time validation scan of pre-existing rows so the COPY → CDC handoff happens immediately. The operator validates the constraints later, on their own schedule, withALTER TABLE ... VALIDATE CONSTRAINT.This is safe for CDC:
NOT VALIDstill enforces referential integrity on every new write — it only skips the historical-row scan. Any change replayed from the source already satisfied the source's constraint, so it satisfies the target'sNOT VALIDconstraint too. FK creation still completes before CDC apply is enabled, so enforcement is in place from the first replayed change.Hardening for the common online-migration flag combination
This was reviewed and tested specifically for
clone --follow --defer-indexes --defer-validate-fks --split-tables-larger-than:setuptable and a resume that changes it is rejected — preventing a resume from silently creating the remaining FKsVALIDand triggering the scan the flag exists to avoid.NOT VALIDforeign keys declared on a partitioned table. Previously this would abort the whole migration. Now the parent-level constraint is skipped with a captured warning (per-partition FK rows are still createdNOT VALID), so the partition data is still enforced and deferred without blocking.Tests
tests/defer-validate-fks— clone path: validated FKs landNOT VALID, source-NOT VALIDFKs stayNOT VALID, data fully copied, violating writes still rejected, manualVALIDATEsucceeds, and the partitioned-parent constraint is skipped without aborting.tests/follow-defer-validate-fks— the full--follow --defer-indexes --defer-validate-fks --split-tables-larger-thancombination with DML injected during CDC; asserts all FKs landNOT VALID, data stays consistent after replay, enforcement holds, and deferred validation completes.Registered in
tests/Makefileand the tiered CI workflow. Full local suite (PG16) passes.