From 8cfb9ca2af43015148d72a9ec8ffe7c6311193aa Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:49:04 +0200 Subject: [PATCH 01/11] docs: the chain guide New docs/the-chain.md: the two-workflow model (push vs chain), the three verbs (revision/migrate/stamp), the file format with a real generated example, migrate's gates, CONCURRENTLY replay semantics, the backfill workflow, the hand-authored 0000_extensions.sql pattern, the public.sqlpush_versions registry, and an honest-limitations section. The 0000_extensions pattern was verified live against a scratch DB on the dev server (localhost:5433, timescaledb-ha:pg17), never against sqlpush_test: - migrate on an empty scratch DB applied a hand-authored 0000_extensions.sql (header `-- sqlpush: revision=0000 risk=SAFE`, one labeled op, CREATE EXTENSION IF NOT EXISTS postgis + timescaledb): applied: 1, exit 0; both extensions present in pg_extension afterwards; registry row ('0000_extensions.sql', sha256) recorded. This also proves parse accepts the hand-authored header/op-label form. - Idempotency: with the registry row deleted, migrate re-EXECUTED the file cleanly (IF NOT EXISTS); with the row present, it skipped. - stamp on a second fresh scratch DB: applied: 0, skipped (registered): 1, registry row present, postgis NOT installed -> nothing was executed. (timescaledb appears on every new DB on this image via template1 preload; postgis does not, so it is the clean indicator.) - Numbering after 0000: next_revision_id on a dir containing only 0000 returns 0001; a real `sqlpush revision` run against the scratch DB (at chain head) generated 0001_create_users.sql with header revision=0001. - End to end: migrate applied 0001 (0000 skipped), then `sqlpush check` against the same DB exited 0. --- docs/the-chain.md | 198 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 docs/the-chain.md diff --git a/docs/the-chain.md b/docs/the-chain.md new file mode 100644 index 0000000..e992a7f --- /dev/null +++ b/docs/the-chain.md @@ -0,0 +1,198 @@ +# The chain: migration files for sqlpush + +sqlpush has two ways to apply a model change. The push workflow +(`diff` / `push` / `check`) computes a plan from your models against +the live database and applies it directly, no files involved. The +chain is the other workflow: the same diff engine and renderer, but +the plan is written to a numbered SQL file you can review, edit and +replay. Files add three things the push workflow does not have: +ordering, checksums and per-file gates. + +## The three verbs + +| verb | what it does | +| --- | --- | +| `revision` | write the next file, from models vs. a reference DB | +| `migrate` | replay pending files, with gates and bookkeeping | +| `stamp` | adopt an existing DB: register files without executing them | + +`revision` diffs your models against a reference database you supply +with `--ref-dsn`. The flag is required, and there is no `DATABASE_URL` +fallback on purpose: the reference DB sits at the chain head, which is +a different database from the push target, and conflating them would +generate the next file against the wrong baseline. Empty drift refuses +("nothing to revise"). An existing file is never overwritten. + +```console +$ sqlpush revision "myapp.models:metadata" \ + --ref-dsn postgresql://user:pass@host:5432/db \ + --message add_users +migrations/versions/0001_add_users.sql +``` + +`migrate` applies every file not yet recorded, in order: + +```console +$ sqlpush migrate +applied: 1, skipped: 0, blocked: 0, partial_failure: False +``` + +`stamp` registers every parseable file as applied without running a +single statement. Use it to adopt a database whose schema already +reflects the chain. Only the header must parse; the SQL bodies are +never executed, so they are not checked. + +## The file format + +A generated file looks like this: + +```sql +-- sqlpush: revision=0001 risk=SAFE +-- parent= +-- create_users + +-- op 1 [SAFE] add_table users +CREATE TABLE users ( + id SERIAL NOT NULL, + email VARCHAR(255) NOT NULL, + PRIMARY KEY (id) +); +``` + +The first line is the only structured requirement: +`-- sqlpush: revision=NNNN risk=SAFE|RISKY|DESTRUCTIVE`. The risk +value is the file's gate; a `DESTRUCTIVE` file needs +`--allow-destructive` on `migrate`. The other comment lines (`parent`, +the message) are informative. You can edit or delete them. Each op +starts with a `-- op N [label] description` line followed by its SQL. + +The structure rides on legal SQL comments, so a chain file runs +directly under `psql`. There is no sqlpush-specific syntax anywhere in +the body. + +The checksum is sha256 over the whole file, newline-normalized. Files +run in lexicographic filename order; the `NNNN_` prefix keeps that +order meaningful, and `revision` numbers the next file max+1. There is +no merge graph, just one linear chain. A missing header, an unknown +risk or a non-numeric revision is a hard parse error: the file is +refused, never guessed at. + +## Gates on migrate + +- The destructive gate reads the header. A file marked + `risk=DESTRUCTIVE` is blocked until you pass `--allow-destructive`. +- The checksum gate catches edits. A file that changed after it was + applied is refused, with an error naming the file. `stamp --force` + is the only override. +- Strict order. A blocked file stops the chain; nothing after it runs. +- One lock for everyone. `migrate` takes the same advisory lock as + `push`, keyed to the database, so chain workers and pushers take + turns instead of interleaving. The wait is bounded + (`--advisory-wait`, default 30s); a stuck holder raises a typed + error rather than hanging. +- Budgets per file. `--lock-timeout` (default 5s) bounds lock waits; + `--statement-timeout` bounds each statement's runtime if you set + one. + +## CONCURRENTLY inside files + +`revision` renders `CREATE INDEX CONCURRENTLY` for indexes on existing +tables by default, like `push` does (`--no-concurrently` opts out). +Replay handles those files specially: + +- A file whose text contains no `CONCURRENTLY` replays whole: the + entire file goes to the server as one call inside one transaction. + Nothing is tokenized, so dollar-quoted function bodies are safe. +- A file that contains `CONCURRENTLY` replays per-op on the op labels. + Plain ops run first in one transaction, then concurrent ops run one + statement at a time on a dedicated autocommit connection. The + registry row is written only after every op succeeds. + +If a concurrent op fails, the file is blocked: partial failure, no +registry row, and the chain stops. The plain segment that already +committed is reported in the notes, not hidden. + +## Backfilling data with a revision + +This is the workflow files exist for. Your table has rows, and the +model gains a column that needs data before it can be tightened. A +schema-only change cannot carry the data; with the chain, schema +change and data change ship as one reviewable unit. + +Generate the revision: + +```console +$ sqlpush revision "myapp.models:metadata" \ + --ref-dsn postgresql://user:pass@host:5432/db \ + --message add_slug +migrations/versions/0003_add_slug.sql +``` + +Then open the file and append a backfill op by hand, before the first +`migrate`: + +```sql +-- op 2 [raw_sql] backfill slugs +UPDATE my_table SET slug = 'u' || id WHERE slug IS NULL; +``` + +The file has never been applied, so its checksum is not recorded yet +and the edit is free. (Editing after apply is refused; see +limitations.) Run `migrate`: the `ALTER` and the `UPDATE` apply in one +transaction. Verify the data, tighten the column in a follow-up +revision if needed, and `check` confirms models and database agree. + +One rule when editing by hand: if the SQL you add is destructive, bump +the file's `risk=` header to match. The header is the gate. + +## Installing extensions first + +Extensions are cluster state, not model state, and never appear in a +plan. If your models use PostGIS `geometry` or TimescaleDB +hypertables, a push against a fresh database fails with +`type "geometry" does not exist` before any of your tables build. + +Hand-author the chain's first file before generating any revision. +The name `0000_extensions.sql` makes it sort first: + +```sql +-- sqlpush: revision=0000 risk=SAFE + +-- op 1 [raw_sql] install extensions the models depend on +CREATE EXTENSION IF NOT EXISTS postgis; +CREATE EXTENSION IF NOT EXISTS timescaledb; +``` + +Fresh environments run `sqlpush migrate` and get the extensions +installed before any generated file, because `0000` sorts before +everything. A database that already has the extensions is adopted +with `stamp`, which registers the file without executing it. Re-runs +are safe: `IF NOT EXISTS` keeps the statements idempotent even if the +registry row is missing. The next `revision` after a hand-authored +`0000` numbers itself `0001`. + +## The registry + +Bookkeeping lives in `public.sqlpush_versions` (`name`, `sha256`, +`applied_at`). The table always lives in `public`, whatever schemas +your models target: it is chain state, not schema state. It is pruned +from every diff, same as `alembic_version`, so a `check` after +`migrate` comes back clean. + +## Honest limitations + +- Editing a file after it was applied is refused (checksum gate). + Editing before the first apply is free; that is the backfill + workflow above. +- A body with no op labels replays as one unit. If such a body + contains `CONCURRENTLY`, it executes statement-by-statement on + autocommit, with no atomicity across statements. +- Per-op parsing treats lines starting with `--` as comments. Keep + them out of a labeled op's SQL unless they really are comments. +- A failed or timed-out `CREATE INDEX CONCURRENTLY` leaves an `INVALID` + index. Recover with `DROP INDEX CONCURRENTLY `, then re-run. +- Mixed files have a crash window: plain segment committed, concurrent + op applied, no registry row yet. A re-run fails loud on the existing + objects instead of silently re-applying. +- Generated concurrent index creates carry no `IF NOT EXISTS`. A re-run + against an existing index fails loud, by design. From 149449500dfa239bc466b41056292faa9d1e834a Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:50:25 +0200 Subject: [PATCH 02/11] docs: migrating from alembic The refugee recipe, honest about what does not exist: no automated conversion of alembic revisions, the old chain stays untouched (alembic_version is pruned from every diff forever). Four phases: observe parity first (check + --exclude for legacy objects), baseline the empty chain (0000_extensions link), run both then switch, and a scratch-DB equivalence replay. Command map included; downgrade is none, forward-only, stated plainly. --- docs/migrating-from-alembic.md | 121 +++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 docs/migrating-from-alembic.md diff --git a/docs/migrating-from-alembic.md b/docs/migrating-from-alembic.md new file mode 100644 index 0000000..853522b --- /dev/null +++ b/docs/migrating-from-alembic.md @@ -0,0 +1,121 @@ +# Migrating from alembic + +There is no automated conversion of alembic revisions, and none is +planned. Your alembic chain stays where it is. sqlpush starts a new +chain from today, and the two can coexist for as long as you want. + +## Command map + +| alembic | sqlpush | +| --- | --- | +| `alembic revision --autogenerate -m "..."` | `sqlpush revision "app.models:metadata" --ref-dsn ... -m "..."` | +| `alembic upgrade head` | `sqlpush migrate` | +| `alembic stamp head` | `sqlpush stamp` | +| `alembic current` / `alembic heads` | `SELECT * FROM public.sqlpush_versions` (plus the file numbering) | +| `alembic downgrade -1` | none. Forward only. | + +## Phase 1: observe parity first + +Before changing anything, point `check` at your alembic-managed +database: + +```console +$ export DATABASE_URL="postgresql://user:pass@host:5432/db" +$ sqlpush check "myapp.models:metadata" +$ echo $? +0 +``` + +`alembic_version` is pruned from every diff automatically, the same +treatment sqlpush gives its own registry table, so it never shows up +as drift. + +Exit `0` means your models match what alembic actually built. That is +the usual case and the green light to continue. Exit `2` or `3` means +there is real drift alembic never knew about: manual hotfixes, objects +created outside migrations, tables someone dropped by hand. Exit `3` +adds destructive drift, which means sqlpush wants to drop something to +make the database match the models. + +Legacy objects you deliberately keep are the normal cause. Accept them +with `--exclude`, which takes fnmatch patterns against table names and +can be repeated: + +```console +$ sqlpush check "myapp.models:metadata" --exclude "legacy_*" --exclude audit_2020 +``` + +Run `check` this way in CI for a week or a month. When it is green, +your models describe the real database. + +## Phase 2: baseline the chain + +With database and models in agreement, the new chain starts empty. +There is nothing to convert: the first `sqlpush revision` comes from +your next model change, generated against a reference DB sitting at +the chain head (see [the chain guide](the-chain.md) for the file +format and gates). + +Two things may belong in the chain before the first generated file: + +- Extensions. If your models use PostGIS or TimescaleDB, hand-author + `0000_extensions.sql` so fresh environments install them first. The + pattern is documented in [the chain guide](the-chain.md). +- Seed state. Same idea: a hand-authored `0000` file, with the SQL you + want every environment to run. + +If the database already has the extensions or seed data, `stamp` +registers those files without executing them. + +## Phase 3: run both, then switch + +For a while, keep `alembic upgrade` as the applier and add +`sqlpush check` to CI next to it. The check gate is exit `0`. Both +tools can look at the same database: alembic never sees sqlpush's +registry, and sqlpush prunes `alembic_version` from every diff +forever. + +When you trust the parity signal, switch over. New model changes go +through `sqlpush revision` and `sqlpush migrate`. Stop running alembic +whenever it is convenient; its version table can stay where it is, +untouched, because it is pruned from every future diff. The old chain +becomes an archive. + +## Phase 4: prove the chain on a scratch DB + +Before you retire alembic, prove that a fresh replay of the files +rebuilds the schema. Take a schema-only dump of the baseline (before +any chain file ran) and restore it into a scratch database, then run +the chain head-to-toe: + +```console +$ pg_dump --schema-only "$PROD_DSN" | psql "$SCRATCH_DSN" +$ sqlpush migrate --dsn "$SCRATCH_DSN" +applied: 4, skipped: 0, blocked: 0, partial_failure: False +$ sqlpush check "myapp.models:metadata" --dsn "$SCRATCH_DSN" +$ echo $? +0 +``` + +Exit `0` is the proof: the files alone rebuilt everything the models +describe. If the dump already contains a `public.sqlpush_versions` +table (because it was taken after the chain started running), drop +that table on the scratch copy first, or every file will skip as +already applied. `stamp` is for the opposite situation: adopting a +database whose chain already ran. + +## Honest differences + +- Forward only. There is no `downgrade`. To revert a change, revert + the model change and generate a new file. The models are the source + of truth, and a downgrade chain is a second history to maintain. +- No offline mode. alembic can render SQL without touching a database + (`alembic upgrade --sql`). Every sqlpush verb diffs against a live + database. The review surface is the files themselves, which are + plain SQL you can read before running. +- No branches. alembic has `branch_labels` and merge revisions. The + chain is strictly linear: lexicographic filename order is the whole + contract, and `revision` numbers files `max + 1`. + +If something in your alembic workflow has no sqlpush equivalent, open +a discussion. The gap may be a feature. From 9877a4f9ef0552ac7c6ebb38c370fc4ac22b7add Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:52:10 +0200 Subject: [PATCH 03/11] docs(readme): six verbs on the front page, full exit codes, inherited-DB story Surface the whole tool: a "When you want files: the chain" section after the push-style pitch (three chain verbs + link to the guide), the comparison row now honest (files optional), and guides links for the chain, alembic and migra. Exit codes table covers all six verbs; a compact per-verb flags table folds in the lock/timeout/concurrency knobs. New "An inherited database" section explains first-check drift and the --exclude / --allow-destructive escape hatches. Scoping paragraph under How it works (extension schemas, public registry, alembic_version pruning). asyncpg translation note in the lifespan section. Roadmap heading de-versioned; pitch softened to "no migration files required". --- README.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 9a756fa..18f8e06 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ **Prisma `db push` for SQLAlchemy.** Apply your models (SQLAlchemy, SQLModel, anything built on `MetaData`) to a live PostgreSQL / TimescaleDB -database directly, no migration files. sqlpush +database directly, no migration files required. sqlpush diffs your models against the real schema, classifies every operation by risk (safe / risky / destructive), and applies the plan atomically. Drift checks exit with codes your CI can gate on. @@ -28,9 +28,10 @@ re-encode what the models say, drift from them, and pile up forever. sqlpush closes the loop the way Prisma's `db push` does for its schema language, but for the SQLAlchemy ecosystem (SQLModel included): -- **No migration files, ever.** The diff *is* the migration: computed fresh +- **No migration files required.** The diff *is* the migration: computed fresh from models vs. live database on every run, via alembic's autogenerate - engine used as a library. + engine used as a library. Files exist as a second workflow when you want + them (see below). - **Risk-aware by default.** Every operation is classified `safe` / `risky` / `destructive`. Destructive ops (drops) are **blocked until `--allow-destructive`**: nothing executes at all while any is present. @@ -46,6 +47,20 @@ language, but for the SQLAlchemy ecosystem (SQLModel included): PostgreSQL only, by design. +## When you want files: the chain + +The push workflow has no files because most of the time you don't need +them. When you do, sqlpush has a second workflow built on the same +diff engine: the chain. `revision` writes the next numbered SQL file +from your models against a reference DB, `migrate` replays pending +files with gates and checksums, and `stamp` adopts an existing +database without executing anything. + +The files are plain SQL you can review, edit before first apply, and +run under `psql`. Schema change and data backfill ship as one file. +The [chain guide](docs/the-chain.md) covers the format, the gates and +the workflows. + ## Install ```console @@ -95,6 +110,16 @@ $ echo $? In CI, check drift and fail loudly (see exit codes below). Limit scope with repeated `--schema` / `--exclude` options. +## An inherited database + +The first `check` against a database with history often reports drift: +hand-built indexes, audit tables, that column someone added at 2am. If +any of the drift looks destructive, `check` exits `3` and `push` +blocks. That is the tool refusing to silently drop your legacy +objects. Two escape hatches: `--exclude` accepts objects you choose to +keep (fnmatch patterns, repeatable), and `--allow-destructive` accepts +the drops when you really do want them. + ## Exit codes | verb | 0 | 1 | 2 | 3 | @@ -102,13 +127,38 @@ repeated `--schema` / `--exclude` options. | `diff` | always | | | | | `check` | clean | | drift | destructive drift | | `push` | applied | destructive blocked | error (incl. partial failure) | | +| `revision` | file written | error (empty drift refuses) | | | +| `migrate` | clean | blocked or partial failure | | | +| `stamp` | registered | blocked or refused | | | + +Failures print a typed error on stderr, never a traceback. `push --safe-only` runs only safe operations and skips the rest informationally (exit `0`). Indexes on existing tables build `CONCURRENTLY` by default (opt out with `--no-concurrently`); a failed `CREATE INDEX CONCURRENTLY` marks the run as partial failure (exit `2`) instead of silently half-applying, and leaves an INVALID index — drop it -(`DROP INDEX CONCURRENTLY`) and re-push. +(`DROP INDEX CONCURRENTLY`) and re-push. `stamp` refuses a file whose +checksum no longer matches the registry; `--force` accepts the new +content. + +The knobs, per verb: + +| verb | flags | +| --- | --- | +| `push` | `--allow-destructive` `--safe-only` `--no-lock` `--lock-timeout` `--advisory-wait` `--no-concurrently` `--statement-timeout` | +| `revision` | `--ref-dsn` (required) `-m/--message` `--no-concurrently` `--dir` | +| `migrate` | `--allow-destructive` `--advisory-wait` `--lock-timeout` `--statement-timeout` `--dir` | +| `stamp` | `--force` `--dir` | + +Every verb except `revision` takes `--dsn` (or `$DATABASE_URL`). +`revision` requires `--ref-dsn`, with no env fallback: the reference +DB is a different database from the push target. `diff`, `check`, +`push` and `revision` also take repeatable `--schema` / `--exclude`. +Timeouts are seconds; a `lock_timeout` bounds how long a statement +waits on a lock before failing, `statement_timeout` bounds each +statement's runtime, and an exhausted `advisory-wait` raises instead +of hanging on a stuck lock holder. ## FastAPI / SQLModel: replace `create_all` @@ -123,7 +173,10 @@ async def lifespan(app): yield ``` -Push in the deploy pipeline, check at startup. +Push in the deploy pipeline, check at startup. asyncpg URLs work too: +a DSN or `AsyncEngine` spelling `postgresql+asyncpg` is translated to +the psycopg driver automatically, and asyncpg is never required in the +sqlpush process. ## How it works @@ -151,13 +204,20 @@ flowchart LR - **Typed errors**: only `SqlpushError` / `ConnectFailed` / `MetadataImportError` escape the API, never raw driver exceptions. +Scoping: `--schema` restricts the diff to named schemas (default: the +session's real `search_path`). Extension-owned schemas never enter +scope automatically, and schemas you pass explicitly are never +filtered. The chain's registry table (`public.sqlpush_versions`) +always lives in `public` and is pruned from every diff, so `check` +after `migrate` is clean. `alembic_version` gets the same treatment. + ## Comparison An honest view of the neighborhood (stars as of 2026-08): | | migration files | source of truth | risk gate | CI drift exit codes | TimescaleDB | | --- | --- | --- | --- | --- | --- | -| **sqlpush** | none (the diff is the migration) | SQLAlchemy `MetaData` | classified safe/risky/destructive, destructive blocked by default | `check` 0/2/3 | `@hypertable` directives | +| **sqlpush** | optional: push needs none; the chain has reviewable, checksummed files | SQLAlchemy `MetaData` | classified safe/risky/destructive, destructive blocked by default | `check` 0/2/3 | `@hypertable` directives | | [alembic](https://github.com/sqlalchemy/alembic) (4.4k★) | yes | migration scripts (autogenerate assists) | no | no | no | | [atlas](https://github.com/ariga/atlas) (8.7k★) | optional (HCL) | HCL / SQL (ORMs via providers) | lint policies | yes | no | | [prisma `db push`](https://www.prisma.io/docs/orm/reference/prisma-cli-reference) (47k★) | none | Prisma schema (Node/TS) | no | no | no | @@ -167,8 +227,9 @@ sqlpush is narrower than atlas and younger than alembic, deliberately. It is one tool for one job: keep a PostgreSQL schema in lockstep with SQLAlchemy models, safely enough to run from CI. -Coming from [migra](https://github.com/djrobstep/migra) (now -deprecated)? There is a [migration guide](docs/migrating-from-migra.md). +Guides: [the chain](docs/the-chain.md) (file format, gates, backfills), +[migrating from alembic](docs/migrating-from-alembic.md), and +[migrating from migra](docs/migrating-from-migra.md) (deprecated). ## Design notes @@ -180,7 +241,7 @@ deprecated)? There is a [migration guide](docs/migrating-from-migra.md). tooling; additive changes only within a version (operations now carry a `concurrent` boolean). -## Roadmap (0.1.x) +## Roadmap - jsonschema-validated `--json` output From dcafc05e633b88908885caeeb86d7d09fa92ab45 Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:52:26 +0200 Subject: [PATCH 04/11] docs(changelog): self-contained limitation wording + docs entry Replace the internal spec reference in the released 0.5.0 Known limitations section with wording that stands alone (the trade-off is named, not cited). Add the [Unreleased] entry for the two guides and the README's six-verb documentation surface. --- CHANGELOG.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d5823c..638bbba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,19 @@ the project follows [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Added + +- Documentation: a chain guide (`docs/the-chain.md` — the three verbs, + file format, migrate's gates, CONCURRENTLY replay semantics, the + data-backfill workflow, the hand-authored `0000_extensions.sql` + pattern, and an honest-limitations list) and an alembic migration + guide (`docs/migrating-from-alembic.md` — parity check first, the + empty-chain baseline, dual-run switchover, and a scratch-DB + equivalence proof; no automated revision conversion, stated up + front). The README now documents all six verbs: the full exit-code + table, a per-verb flags reference, the inherited-database story, and + schema-scoping facts. + ## [0.5.0] - 2026-09-02 ### Added @@ -66,8 +79,10 @@ the project follows [Semantic Versioning](https://semver.org/). ### Known limitations - Generated chain files replay per-op on their `-- op N [label]` - delimiters; hand-edits bypass that tokenization (pinned chain spec - §7). A label-less body containing CONCURRENTLY routes whole to the + delimiters; hand-edits bypass that tokenization (a deliberate + trade-off: generated files carry the op labels the replay splits on; + hand-edits cannot rely on them). A label-less body containing + CONCURRENTLY routes whole to the autocommit lane statement-by-statement, and lines starting `--` are stripped from per-op parsing — dollar-quoted bodies containing `--` lines are only safe in the whole-text fast path. From 406a4748d7c4c0e82b310b6ed1f2a95437064082 Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:55:00 +0200 Subject: [PATCH 05/11] docs: punctuation cleanup from the humanizer pass --- CHANGELOG.md | 4 ++-- README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 638bbba..58cf792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,11 +8,11 @@ the project follows [Semantic Versioning](https://semver.org/). ### Added -- Documentation: a chain guide (`docs/the-chain.md` — the three verbs, +- Documentation: a chain guide (`docs/the-chain.md`: the three verbs, file format, migrate's gates, CONCURRENTLY replay semantics, the data-backfill workflow, the hand-authored `0000_extensions.sql` pattern, and an honest-limitations list) and an alembic migration - guide (`docs/migrating-from-alembic.md` — parity check first, the + guide (`docs/migrating-from-alembic.md`: parity check first, the empty-chain baseline, dual-run switchover, and a scratch-DB equivalence proof; no automated revision conversion, stated up front). The README now documents all six verbs: the full exit-code diff --git a/README.md b/README.md index 18f8e06..a769111 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Failures print a typed error on stderr, never a traceback. informationally (exit `0`). Indexes on existing tables build `CONCURRENTLY` by default (opt out with `--no-concurrently`); a failed `CREATE INDEX CONCURRENTLY` marks the run as partial failure (exit `2`) -instead of silently half-applying, and leaves an INVALID index — drop it +instead of silently half-applying, and leaves an INVALID index: drop it (`DROP INDEX CONCURRENTLY`) and re-push. `stamp` refuses a file whose checksum no longer matches the registry; `--force` accepts the new content. From 3441822e44a8eb51fb22689bb6587926041686e9 Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 22:59:13 +0200 Subject: [PATCH 06/11] docs: checksum and stamp-override wording (review nits) --- docs/the-chain.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/the-chain.md b/docs/the-chain.md index e992a7f..4115ed4 100644 --- a/docs/the-chain.md +++ b/docs/the-chain.md @@ -70,7 +70,8 @@ The structure rides on legal SQL comments, so a chain file runs directly under `psql`. There is no sqlpush-specific syntax anywhere in the body. -The checksum is sha256 over the whole file, newline-normalized. Files +The checksum is sha256 over the whole file, newline-normalized, with +trailing whitespace ignored. Files run in lexicographic filename order; the `NNNN_` prefix keeps that order meaningful, and `revision` numbers the next file max+1. There is no merge graph, just one linear chain. A missing header, an unknown @@ -82,8 +83,8 @@ refused, never guessed at. - The destructive gate reads the header. A file marked `risk=DESTRUCTIVE` is blocked until you pass `--allow-destructive`. - The checksum gate catches edits. A file that changed after it was - applied is refused, with an error naming the file. `stamp --force` - is the only override. + applied is refused, with an error naming the file. The escape is + `stamp --force`, which re-records the new content as applied. - Strict order. A blocked file stops the chain; nothing after it runs. - One lock for everyone. `migrate` takes the same advisory lock as `push`, keyed to the database, so chain workers and pushers take From d40b7aa0c87e2211bc568e4fbc23639aedfbfdcd Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:09:03 +0200 Subject: [PATCH 07/11] docs(readme): lead with the create_all retirement story --- CHANGELOG.md | 5 +++- README.md | 75 +++++++++++++++++++++++++++++++++++----------------- 2 files changed, 55 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58cf792..a451551 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,10 @@ the project follows [Semantic Versioning](https://semver.org/). equivalence proof; no automated revision conversion, stated up front). The README now documents all six verbs: the full exit-code table, a per-verb flags reference, the inherited-database story, and - schema-scoping facts. + schema-scoping facts. The front page leads with the `create_all` + retirement story (the tutorial lifespan against the one-line + `aensure_schema(..., mode="check")` boot guard), and the PyPI + keywords now include `fastapi` and `sqlmodel`. ## [0.5.0] - 2026-09-02 diff --git a/README.md b/README.md index a769111..de8b0da 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,12 @@ [![Python](https://img.shields.io/pypi/pyversions/sqlpush?style=for-the-badge)](https://pypi.org/project/sqlpush/) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge)](LICENSE) -**Prisma `db push` for SQLAlchemy.** Apply your models (SQLAlchemy, -SQLModel, anything built on `MetaData`) to a live PostgreSQL / TimescaleDB -database directly, no migration files required. sqlpush -diffs your models against the real schema, classifies every operation by -risk (safe / risky / destructive), and applies the plan atomically. Drift -checks exit with codes your CI can gate on. +**Prisma `db push` for SQLAlchemy.** Your models are the migration. +sqlpush diffs them (SQLAlchemy, SQLModel, anything built on `MetaData`) +against the live PostgreSQL / TimescaleDB database, classifies every +operation by risk (safe / risky / destructive), and applies the plan +atomically. No migration files to write, no `upgrade` step to forget. +Drift checks exit with codes your CI can gate on. ```console sqlpush diff "myapp.models:metadata" # see the SQL, ordered by risk @@ -110,6 +110,51 @@ $ echo $? In CI, check drift and fail loudly (see exit codes below). Limit scope with repeated `--schema` / `--exclude` options. +## FastAPI: retire `create_all()` + +Most FastAPI + SQLModel apps ship the lifespan the tutorials teach: + +```python +@asynccontextmanager +async def lifespan(app): + async with engine.begin() as conn: + await conn.run_sync(SQLModel.metadata.create_all) + yield +``` + +`create_all` creates tables that are missing. That is all it ever +does. Add a column to a model and the database never hears about it; +an index on an existing table, a type change, a drop: nothing. +Production drifts from the models in silence, so every real change +still rides the alembic treadmill: autogenerate, review, upgrade, and +two histories to keep in agreement forever. + +The sqlpush lifespan is one line: + +```python +from contextlib import asynccontextmanager +from sqlpush import aensure_schema + + +@asynccontextmanager +async def lifespan(app): + await aensure_schema(SQLModel.metadata, engine, mode="check") + yield +``` + +`mode="check"` verifies the models against the database at startup and +raises when they disagree: the app refuses to boot against a schema it +does not match, which beats failing on the first query at 3am. The +schema change itself comes from wherever you put it: `sqlpush push` in +the deploy pipeline (destructive ops gated), or +`ensure_schema(..., mode="push")` when you want the API to apply it. + +asyncpg URLs work too: a DSN or `AsyncEngine` spelling +`postgresql+asyncpg` is translated to the psycopg driver automatically, +and asyncpg is never required in the sqlpush process. + +Push in the deploy pipeline, check at boot. + ## An inherited database The first `check` against a database with history often reports drift: @@ -160,24 +205,6 @@ waits on a lock before failing, `statement_timeout` bounds each statement's runtime, and an exhausted `advisory-wait` raises instead of hanging on a stuck lock holder. -## FastAPI / SQLModel: replace `create_all` - -```python -from contextlib import asynccontextmanager -from sqlpush import aensure_schema - - -@asynccontextmanager -async def lifespan(app): - await aensure_schema(SQLModel.metadata, engine, mode="check") - yield -``` - -Push in the deploy pipeline, check at startup. asyncpg URLs work too: -a DSN or `AsyncEngine` spelling `postgresql+asyncpg` is translated to -the psycopg driver automatically, and asyncpg is never required in the -sqlpush process. - ## How it works ```mermaid From e58101453002084e9c17d099f3d6e686af455509 Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:09:03 +0200 Subject: [PATCH 08/11] chore(meta): fastapi and sqlmodel discovery keywords --- pyproject.toml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pyproject.toml b/pyproject.toml index f36d8de..359242a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,6 +10,8 @@ requires-python = ">=3.10" license = "MIT" keywords = [ "sqlalchemy", + "sqlmodel", + "fastapi", "alembic", "postgresql", "timescaledb", From 6741218c8c40e0ba25ae2a44e99bbfa2ba472dfc Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:10:45 +0200 Subject: [PATCH 09/11] docs(readme): async spelling in the lifespan push note --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index de8b0da..823c17c 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ raises when they disagree: the app refuses to boot against a schema it does not match, which beats failing on the first query at 3am. The schema change itself comes from wherever you put it: `sqlpush push` in the deploy pipeline (destructive ops gated), or -`ensure_schema(..., mode="push")` when you want the API to apply it. +`aensure_schema(..., mode="push")` when you want the API to apply it. asyncpg URLs work too: a DSN or `AsyncEngine` spelling `postgresql+asyncpg` is translated to the psycopg driver automatically, From 7c97417482c7a4739f1e0589e18d5337c7fe58a4 Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:16:12 +0200 Subject: [PATCH 10/11] docs(readme): drop redundant license section (badge covers it) --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 823c17c..e47fb84 100644 --- a/README.md +++ b/README.md @@ -271,7 +271,3 @@ Guides: [the chain](docs/the-chain.md) (file format, gates, backfills), ## Roadmap - jsonschema-validated `--json` output - -## License - -[MIT](LICENSE) · © 2026 Juan Miguel Contreras From 46a3da1d0f06cf6b4046a8fe14ee2147991a332f Mon Sep 17 00:00:00 2001 From: Juan Miguel Contreras <19253629+juanmicl@users.noreply.github.com> Date: Wed, 2 Sep 2026 23:20:20 +0200 Subject: [PATCH 11/11] docs(readme): full-pass review improvements (order, dedup, real render output) --- README.md | 99 +++++++++++++++++++++++++++---------------------------- 1 file changed, 48 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index e47fb84..8c0f047 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,22 @@ sqlpush check "myapp.models:metadata" # CI gate: exit 0/2/3 sqlpush push "myapp.models:metadata" # apply (destructive gated) ``` -If you've ever run `Base.metadata.create_all()` in production and known it -was wrong, then sighed at the migration-script treadmill when you reached -for alembic: sqlpush is for you. +If you've run `Base.metadata.create_all()` in production and known it +was wrong, sqlpush is for you. + +## Install + +```console +pip install sqlpush +``` + +Or from source: + +```console +git clone https://github.com/juanmicl/sqlpush && cd sqlpush && uv sync +``` + +Python 3.10 or newer. PostgreSQL only. ## Why @@ -35,43 +48,20 @@ language, but for the SQLAlchemy ecosystem (SQLModel included): - **Risk-aware by default.** Every operation is classified `safe` / `risky` / `destructive`. Destructive ops (drops) are **blocked until `--allow-destructive`**: nothing executes at all while any is present. -- **Drift detection built for CI.** `check` plans once and exits `0` clean / - `2` drift / `3` destructive drift, scriptable without parsing output. - `--json` emits a stable versioned contract. -- **Safe under concurrency.** An advisory lock (keyed to the database, not - the DSN) coordinates workers: one pusher at a time, losers wait bounded +- **Drift detection built for CI.** `check` plans once and reports + through its exit code, no output parsing; `--json` emits a stable + versioned contract. +- **Safe under concurrency.** An advisory lock coordinates workers: one + pusher at a time, losers wait bounded and re-verify, so deploy pipelines can race without corrupting anything. - **Hypertables without hand-written SQL.** Decorate a model with `@hypertable` and the `create_hypertable` directive is planned state-aware: idempotent pushes, clean checks, no false drift. -PostgreSQL only, by design. - -## When you want files: the chain - -The push workflow has no files because most of the time you don't need -them. When you do, sqlpush has a second workflow built on the same -diff engine: the chain. `revision` writes the next numbered SQL file -from your models against a reference DB, `migrate` replays pending -files with gates and checksums, and `stamp` adopts an existing -database without executing anything. - -The files are plain SQL you can review, edit before first apply, and -run under `psql`. Schema change and data backfill ship as one file. -The [chain guide](docs/the-chain.md) covers the format, the gates and -the workflows. - -## Install - -```console -pip install sqlpush -``` - -Or from source: +If you know alembic: sqlpush is its autogenerate engine, productized +into apply and check verbs, with no revision scripts to maintain. -```console -git clone https://github.com/juanmicl/sqlpush && cd sqlpush && uv sync -``` +PostgreSQL only, by design. ## The 30-second tour @@ -83,14 +73,13 @@ $ export DATABASE_URL="postgresql+psycopg://user:pass@host:5432/db" $ sqlpush diff "myapp.models:metadata" -- safe - CREATE TABLE hero ( - id SERIAL NOT NULL PRIMARY KEY, - name VARCHAR(50) NOT NULL + id SERIAL NOT NULL, + name VARCHAR(50) NOT NULL, + PRIMARY KEY (id) ); -- risky - CREATE INDEX ix_hero_name ON hero (name); ``` @@ -158,13 +147,26 @@ Push in the deploy pipeline, check at boot. ## An inherited database The first `check` against a database with history often reports drift: -hand-built indexes, audit tables, that column someone added at 2am. If +hand-built indexes, audit tables, a column someone added by hand. If any of the drift looks destructive, `check` exits `3` and `push` blocks. That is the tool refusing to silently drop your legacy objects. Two escape hatches: `--exclude` accepts objects you choose to keep (fnmatch patterns, repeatable), and `--allow-destructive` accepts the drops when you really do want them. +## When you want files: the chain + +Most changes never need a file. When one does, sqlpush has a second +workflow built on the same diff engine: the chain. `revision` writes +the next numbered SQL file from your models against a reference DB, +`migrate` replays pending files with gates and checksums, and `stamp` +adopts an existing database without executing anything. + +The files are plain SQL you can review, edit before first apply, and +run under `psql`. Schema change and data backfill ship as one file. +The [chain guide](docs/the-chain.md) covers the format, the gates and +the workflows. + ## Exit codes | verb | 0 | 1 | 2 | 3 | @@ -218,16 +220,14 @@ flowchart LR apply --> report["report"] ``` -- **Diff engine** scopes reflection to your target schemas (default: the - session's real `search_path`) and prunes system catalogs (TimescaleDB - internals included) before reflection even starts. +- **Diff engine** scopes reflection to your target schemas and prunes + system catalogs (TimescaleDB internals included) before reflection + even starts. - **Classifier** maps each operation to a risk class; unknown operations are `risky`, never silently safe. -- **Executor** splits the plan: existing-table indexes render - `CREATE INDEX CONCURRENTLY` and run one-per-transaction on autocommit - (`--no-concurrently` opts out; indexes on tables the same plan creates - stay in the atomic transaction), everything else applies in a single - atomic transaction with a bounded `lock_timeout`. +- **Executor** splits the plan: concurrent index builds run one per + transaction on autocommit, everything else applies in a single atomic + transaction with a bounded `lock_timeout`. - **Typed errors**: only `SqlpushError` / `ConnectFailed` / `MetadataImportError` escape the API, never raw driver exceptions. @@ -257,6 +257,7 @@ SQLAlchemy models, safely enough to run from CI. Guides: [the chain](docs/the-chain.md) (file format, gates, backfills), [migrating from alembic](docs/migrating-from-alembic.md), and [migrating from migra](docs/migrating-from-migra.md) (deprecated). +Changes land in the [CHANGELOG](CHANGELOG.md). ## Design notes @@ -267,7 +268,3 @@ Guides: [the chain](docs/the-chain.md) (file format, gates, backfills), - `--json` output is a versioned contract (`"version": 1`) meant for tooling; additive changes only within a version (operations now carry a `concurrent` boolean). - -## Roadmap - -- jsonschema-validated `--json` output