Run DB migrations at startup with advisory locks; seeders become dev-only#373
Open
runleveldev wants to merge 4 commits into
Open
Run DB migrations at startup with advisory locks; seeders become dev-only#373runleveldev wants to merge 4 commits into
runleveldev wants to merge 4 commits into
Conversation
…locks The server now applies all pending database migrations on startup before listening, so a deployment no longer depends on a one-shot init service to migrate. The run is serialized across processes by an engine-appropriate advisory lock (pg_advisory_lock on PostgreSQL, GET_LOCK on MySQL; SQLite needs none), so only one process migrates at a time. On failure the process exits non-zero (systemd Restart=on-failure retries) rather than serving an out-of-date schema. - Add utils/migrate.js: programmatic runner using umzug (the same v2 API sequelize-cli uses, so it shares the SequelizeMeta ledger) wrapped in the advisory lock. - Wire runMigrations() into server.js before app.listen(); fail fast on error. - Depend on umzug directly as a production dependency and move sequelize-cli to devDependencies (production runs migrations via the app, not the CLI). - Convert the four production seeders (groups, wazuh/sssd env vars) into idempotent migrations so required data is applied automatically and tracked in SequelizeMeta. They are timestamped to run after all schema migrations (preserving the old "seeders run last" ordering) and remain idempotent so databases that already ran them as seeders are not duplicated. - Drop the two obsolete push-notification seeders (their keys are removed by 20260604000002-remove-push-notification-settings). - Keep seeders/ for test/development data only. - container-creator-init.service no longer runs db:migrate/db:seed:all; it only provisions the PostgreSQL role/database and credentials. - Document the new behavior in the README.
Now that seeders are development-only (production data lives in migrations), move the local-dev provisioning out of bin/dev-bootstrap.js into a proper Sequelize seeder. It runs automatically via `make dev` (npm run db:migrate -> sequelize-cli db:seed:all) instead of a separate `node bin/dev-bootstrap.js` step. Behavior is preserved: it idempotently creates a localhost Site, a local-dummy Node (nodeType: dummy), and a localhost ExternalDomain, and is a NO-OP when any Site already exists. Seeders use the default `none` storage (they re-run on every db:seed:all), so the existing-site guard keeps it safe to re-run. It also no-ops when NODE_ENV=production so dev-only data never seeds production. - Add seeders/20260616000000-seed-dev-environment.js (queryInterface-based). - Remove bin/dev-bootstrap.js and its `make dev` invocation; update the Makefile comment to note seeding now happens via db:seed:all.
The README had grown to ~665 lines of largely inaccurate detail: manual clone/npm/systemd install steps that no longer match how the app is deployed (it ships as an OCI image and as deb/rpm/apk packages), MySQL/utf8mb4 and SESSION_SECRET instructions that don't apply (production is PostgreSQL; session secrets are auto-generated), plus stale API-route, schema, and job-runner testing dumps. Replace it with a concise component reference: - Development is just `make dev` (SQLite + dummy hypervisor, no .env/Proxmox). - Production merely notes the OCI image and the deb/rpm/apk package options, deferring to the Installation Guide. - Keep accurate, current notes on configuration (example.env), startup migrations/seeders, the OpenAPI/Swagger entry point, and the directory layout, with links to the canonical docs.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes the Manager apply Sequelize migrations automatically at process startup, coordinating concurrent instances via database advisory locks, and shifts seeders to development-only by expressing “always-present” production data as idempotent migrations.
Changes:
- Add a programmatic Umzug-based migration runner with advisory-lock serialization, and run it before the HTTP server starts listening.
- Rework development bootstrap into a Sequelize seeder; remove obsolete push-notification seeders and delete the old
bin/dev-bootstrap.jspath. - Update packaging/docs and init service behavior so deployments no longer rely on an init unit to run
sequelize-cli db:migrate/db:seed:all.
Reviewed changes
Copilot reviewed 13 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| images/manager/container-creator-init.service | Stops running migrations/seeders in the init unit; only writes DB env to /etc/default/container-creator. |
| create-a-container/utils/migrate.js | New Umzug-based migration runner with Postgres/MySQL advisory locks. |
| create-a-container/server.js | Runs migrations at startup and exits non-zero on failure. |
| create-a-container/seeders/20260616000000-seed-dev-environment.js | New dev-only seeder replacing the old dev bootstrap script. |
| create-a-container/seeders/20260313000000-push-notification-api-key.js | Removes obsolete seeder. |
| create-a-container/seeders/20260120165612-push-notification-settings.js | Removes obsolete seeder. |
| create-a-container/seeders/.gitkeep | Retains seeders directory placeholder. |
| create-a-container/README.md | Large cleanup; updates dev/prod workflow docs to match new migration/seed behavior. |
| create-a-container/package.json | Moves sequelize-cli to devDependencies and adds umzug as a production dependency. |
| create-a-container/package-lock.json | Lockfile update reflecting dependency changes. |
| create-a-container/migrations/20260616000003-seed-sssd-access-filter-and-user-attrs.js | Updates commentary around seed migration approach. |
| create-a-container/migrations/20260616000002-seed-sssd-env-vars.js | Comment/doc adjustment for seeded env vars. |
| create-a-container/migrations/20260616000001-seed-wazuh-env-vars.js | Comment/doc adjustment for seeded env vars. |
| create-a-container/migrations/20260616000000-seed-groups.js | Minor signature/comment tweaks in seed migration. |
| create-a-container/Makefile | Removes bin/dev-bootstrap.js from make dev; relies on npm run db:migrate (which seeds). |
| create-a-container/bin/dev-bootstrap.js | Deletes the old dev bootstrap script. |
Files not reviewed (1)
- create-a-container/package-lock.json: Generated file
- migrate.js: release the dedicated advisory-lock connection back to the pool if acquiring the lock throws (Postgres pg_advisory_lock / MySQL GET_LOCK), so a permission/network error during acquisition no longer leaks a connection across startup retries. - job-runner.js: run migrations on startup too. Both services are advisory-lock serialized and migrations are idempotent, so whichever starts first migrates; this removes the first-boot race where the runner could connect to an un-migrated schema and crash-loop (its systemd unit does not order after a migrated server). - seed-dev-environment.js: quote identifiers/literals via the dialect-aware queryGenerator instead of hard-coded Postgres double quotes, so the dev seeder also works under DATABASE_DIALECT=mysql.
7d3ea9b to
ac83b6c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the Manager apply its own database migrations at startup (coordinated by a database advisory lock so only one process migrates at a time), and reworks seeders to be development-only. Production data that must always exist is now expressed as migrations.
This removes the deployment's reliance on a one-shot init service to run
sequelize db:migrate/db:seed:all, and makessequelize-clia dev-only authoring tool.Changes
Startup migrations + advisory locks (
aed1691)utils/migrate.js: a programmatic runner using umzug (the same v2 APIsequelize-cliuses internally, so it shares theSequelizeMetaledger and skips already-applied migrations). The run is wrapped in an engine-appropriate advisory lock:pg_advisory_lock/pg_advisory_unlock(held on one dedicated session connection for the whole batch)GET_LOCK/RELEASE_LOCKserver.jscallsrunMigrations(sequelize)at the start ofmain(), beforeapp.listen(). On failure the process exits non-zero (so systemdRestart=on-failureretries) rather than serving an out-of-date schema.umzugis now a production dependency;sequelize-climoved to devDependencies.20260604000002-remove-push-notification-settings).images/manager/container-creator-init.serviceno longer runsdb:migrate/db:seed:all; it only provisions the PostgreSQL role/database and writes credentials to/etc/default/container-creator.dev-bootstrap→ dev seeder (ededa0c)bin/dev-bootstrap.jsinto a proper Sequelize seeder (seeders/20260616000000-seed-dev-environment.js) that runs viamake dev(npm run db:migrate→sequelize-cli db:seed:all).localhostSite, alocal-dummyNode (nodeType: dummy), and alocalhostExternalDomain; no-ops if any Site already exists and whenNODE_ENV=production.README cleanup (
c6aec84)create-a-container/README.mdfrom ~665 → ~120 lines, removing stale/inaccurate manual install, MySQL, andSESSION_SECRETinstructions. Development is now justmake dev; production points to the OCI image andmake deb/rpm/apkpackages, deferring to the Installation Guide.Validation
Verified manually against SQLite (no automated test suite in this component):
default_container_env_varsseeded; 0 push-notification keys.server.jsboot: migrate → "Server running".make devseeder: creates the local dev environment on a fresh DB, no-ops on re-run and underNODE_ENV=production.Notes
@vscode/sqlite3in production), though that specific issue is already resolved onmain.