Skip to content

feat: apso import data — copy table data from Supabase into the rebuilt DB (phase 2)#69

Open
cultron wants to merge 1 commit into
feat/import-supabase-schemafrom
feat/import-supabase-data
Open

feat: apso import data — copy table data from Supabase into the rebuilt DB (phase 2)#69
cultron wants to merge 1 commit into
feat/import-supabase-schemafrom
feat/import-supabase-data

Conversation

@cultron

@cultron cultron commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 2 of migrating off Supabase. Phase 1 (#68) generates the .apsorc and you rebuild the schema on Apso; this command then copies the row data from the source Postgres into the target database.

Stacked on #68 (feat/import-supabase-schema). Review/merge #68 first; this PR's diff is just the phase-2 additions.

apso import data -s "postgresql://…source…" -t "postgresql://…target…"
apso import data --dry-run            # plan + row counts, no writes
apso import data --tables users,orders

How it works

  1. Introspect both source and target with phase 1's PgIntrospector (reused as-is).
  2. Plan (planCopy, pure): topologically order tables by FK (parents first; self-refs ignored, cycles reported + best-effort), map each source column to its target column — exactly mirroring phase 1: target table = snakeCase(entity), single-column FK → camelCase <ref>Id (via the shared deriveToName/getRelationshipIdField), everything else identity — then verify each mapped column against the target introspection, dropping + reporting any mismatch.
  3. Execute (executeCopy): insert-only — pre-flight aborts before writing if any target table already has rows; batched inserts; serial sequences reset after load; whole copy in one transaction (rollback on error).

Decisions (confirmed)

  • Compute + verify against target for name mapping.
  • Insert-only; abort if a target table is non-empty (no destructive default).
  • Target connection supplied explicitly (-t / TARGET_DATABASE_URL / APSO_DATABASE_URL / prompt) — there's no platform API exposing it. Connection strings are never logged or persisted.

Verification

  • 16 new unit tests for the pure layer (table ordering incl. self-ref + cycle, FK column mapping, target-verification + skips, missing-target-table, serial-PK detection, value coercion, insert SQL) and the executor via fakes (batching, sequence reset, pre-flight abort, dry-run, rollback on failure).
  • End-to-end against two live PGlite databases: snake_case source → camelCase-FK target, JSONB + null values preserved, serial sequence reset (next id = 3, no collision), and FK integrity verified across a join.
  • Full suite: 264 tests, lint clean.

Notes / follow-ups

  • Paging uses OFFSET/LIMIT ordered by the PK — fine for a one-shot migration; a keyset cursor would be better for very large tables under concurrent writes.
  • The whole copy is one transaction (simple + safe); for very large datasets a per-table transaction mode could be a future flag.
  • Composite-FK columns copy as-is under their original names (they remain scalar columns in the generated schema), consistent with phase 1.

🤖 Generated with Claude Code

…lt DB

Phase 2 of migrating off Supabase: after 'apso import supabase' generates
the schema and it's deployed, this copies the rows from the source
Postgres into the target database.

- src/lib/import/copy-data.ts (pure, fully unit-tested): planCopy() builds
  a topologically-ordered, FK-aware plan. Mapping mirrors phase 1 exactly
  (target table = snakeCase(entity); single-col FK -> camelCase <ref>Id via
  the shared deriveToName/getRelationshipIdField; all other columns
  identity), then verifies every mapped column against an introspection of
  the target and drops/reports mismatches. executeCopy() is insert-only
  (pre-flight aborts if any target table is non-empty), batches inserts,
  resets serial sequences, and runs in one transaction (rollback on error).
- src/lib/import/pg-data.ts: PgSourceReader/PgTargetWriter (lazy pg, SSL on)
  implementing the injectable SourceReader/TargetWriter seams.
- src/commands/import/data.ts: 'apso import data' with source/target conn
  resolution (flag / SUPABASE_DB_URL|DATABASE_URL and TARGET_DATABASE_URL|
  APSO_DATABASE_URL / masked prompt), --schema/--tables/--batch-size/
  --dry-run/--yes; never logs or persists connection strings.

Reuses phase 1's PgIntrospector for both source and target schemas.

Verified end-to-end against two live PGlite databases: snake_case source
-> camelCase-FK target with JSONB/null preservation, serial sequence
reset (next id = 3, no collision), and FK integrity across a join.
Full suite 264 tests, lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant