Clone integration test databases from a migrated template and run them in parallel

[ancongui]

What changes

The PostgreSQL integration suite spent most of its time preparing databases, not testing. Every test created a
database, ran all 30 migrations on it, created its login roles, and kept them all on the server. This change
keeps the same per-test isolation and removes that cost:

• Template database. Tests that need the current schema (migration_settings, and everything built on it:
  settings, access_db, authenticated_client, …) clone weave_b1_template_<hash> with
  CREATE DATABASE … TEMPLATE … (about 45 ms). The template is migrated once by the real migrate(), which
  also runs check_schema. Its name is a hash of the migration files, the schema version, the default
  operations policy the migrations record, the module that defines migrate() and the Alembic version, so
  changing any of them builds a new template. A session advisory lock lets one worker build it while the
  others wait. It is marked IS_TEMPLATE true ALLOW_CONNECTIONS false only after a complete migration, so a
  failed build is never cloned.
  A migration failure while building it is reported as itself, not as an unavailable backend.
• Shared servers. Each process records in the template's comment when it last used it (at its first clone,
  then at most every 10 minutes). A build drops another weave_b1_template_* only when nobody is connected to
  it and nobody has used it for 6 hours, so checkouts with different migrations can share one server.
  Templates without that comment, built before this change, count as unused.
• Cleanup. Databases and login roles a test creates are dropped when it ends (DROP DATABASE … WITH (FORCE) first, so no role keeps a privilege, then DROP ROLE). WEAVE_TEST_KEEP_DATABASES=1 keeps them.
  When WEAVE_TEST_RESOURCE_MANIFEST is set, each resource is recorded exactly as before and kept, so the
  manifest still lists resources that exist. Every drop is attempted even when one fails, and one warning
  names all the databases and roles left behind. The roles that three tests create themselves are now
  recorded too (test_incident_migration, test_schedule_migration, and the external database of
  connectors/test_postgresql). Tests record what they create through a retain fixture instead of
  from conftest import retain_resource, which resolved to the live AI conftest under a bare pytest. The
  template stays on the server between runs; it holds no test data.
• Parallel runs. pytest-xdist joins the dev group (the worker lockfiles record the root project's groups,
  so they change by one line each). Fixtures were already worker-safe (uuid4 names, one
  database per test); the template build is the only shared step and it is locked.
• Docs. CONTRIBUTING.md explains the fixtures, when the template is built, which templates a build
  drops, the environment variables and the parallel command, and says the durability settings are optional
  because they made no measurable difference.

scripts/check.py still runs the integration stage serially; it gets the template and cleanup speed-up
without any change to the gate.

How migrations stay covered

• Tests about migrations keep running them for real on an empty database: empty_settings is unchanged
  (test_boot schema rejection, test_access audit forward migration, test_email migration RLS,
  test_incident_migration, test_schedule_migration).
• test_explicit_migration_is_idempotent now uses a new fresh_migration_settings fixture, which runs
  migrate() on an empty database inside the test before running it a second time, as it did before.
• The template itself is built by the production migrate() with its schema check, once per migration set.

Timings

Measured on the colima-weave-tests VM (4 CPUs, 8 GiB, PostgreSQL 17) with only PostgreSQL configured.
Another project's PostgreSQL shared the VM and used up to one full CPU during several runs, so single runs
vary by about 20%.

Subset of 106 tests (test_access, test_boot, test_human_files, test_human_tasks, test_leases,
test_runs):

Run	Wall time	Setup per test	Call per test
main, serial	484.6 s	4.12 s	0.42 s
this branch, serial	108.1–121.9 s	0.69–0.81 s	0.24–0.27 s
this branch, -n 4	44.0 s
this branch, -n 8	40.0 s

Whole suite (676 tests collected):

Run	Wall time	Median setup
main, serial	2044 s (34 min), see below	2.72 s
this branch, serial	1149 s (19 min)	0.87 s
this branch, -n 2 --dist loadfile	825 s
this branch, -n 4 --dist loadfile	355–426 s (four runs)
this branch, -n 4 --dist load	428 s
this branch, -n 8 --dist loadfile	268 s

On main the serial run never finished cleanly here: after about 560 retained databases (15 GB of data, 50k
files per checkpoint) the server crashed with a segmentation fault during a test's migration and the next 57
tests failed with "the database system is in recovery mode". Those 9 test files were rerun on a fresh server
(463 s) to complete the baseline.

Turning off fsync, synchronous_commit and full_page_writes made no measurable difference once databases
are cloned and dropped (-n 4 --dist loadfile: 355 s on a default server, 383 s on a tuned one).

Verification

• Whole suite, serial, against the main baseline: 614 passed, 4 failed, 57 errors, 1 skipped on both. The 57
  errors and 2 of the failures need backends this run did not configure (Kafka, Keycloak, built images, the
  release receipt). test_concurrent_publication_and_lost_response fails on main and on this branch alike.
• One timing race swapped places between the two serial runs (test_resolution_receipt_revision_and_audit
  failed, test_two_replicas_join_once passed). Both are concurrent mutations of one project that expect the
  first transaction to finish inside the 250 ms lock_timeout of the project admission lock; a slower holder
  gives the other request 429 WV-OPERATION-CAPACITY. Run alone, both fail now and then on main too
  (test_business_failure_and_sibling_complete_concurrently[failure]: 1 of 5 on each).
• With -n 4, two to three tests of that same group fail per run
  (test_business_failure_and_sibling_complete_concurrently[failure], test_resolution_receipt_revision_and_audit,
  test_timeout_races_signal_once, test_timely_signal_survives_sibling_suspension_until_recovery_or_overall_timeout[mixed_signal_overall]):
  more CPU contention makes the 250 ms window easier to miss. No failure comes from shared database state.
  This is why the gate stays serial.
• test_due_expiry_wins_concurrent_completion (429 instead of 409 when test_human_tasks.py runs before
  test_human_files.py) passed 3 of 3 runs in that order on this branch and 2 of 2 on main; it belongs to the
  same timing group and did not fail in any parallel run.
• After every run the server held no test database and no test role, only the template.
  WEAVE_TEST_KEEP_DATABASES=1 and WEAVE_TEST_RESOURCE_MANIFEST were checked to keep databases and roles, and
  the manifest records the roles of the migration tests.
• make check (all stages), ruff check, ruff format --check and mypy pass. Final whole-suite run with
  -n 4 --dist loadfile: 612 passed, 6 failed, 57 errors, 1 skipped in 426 s; the two extra failures are
  the timing tests named above.
• Review follow-ups, on a fresh default server: the migration subset serially (test_template_cleanup,
  test_resource_cleanup, test_boot, test_incident_migration, test_schedule_migration, test_email,
  test_access, providers/test_teams, and the test_operations test that creates its own login): 61 passed
  in 78 s. Then, with a marked template without a use comment and a recently used one seeded and the current
  template dropped, -n 4 --dist loadfile over test_template_cleanup, test_resource_cleanup, test_boot,
  test_access, test_runs, test_leases, test_human_tasks, test_incident_migration,
  test_schedule_migration, test_operations and test_email: 124 passed in 74 s. The DDL log showed one
  template build, the unmarked template dropped and the recently used one kept; no test database or role
  was left. pytest --collect-only tests/integration collects 679 tests; pytest --collect-only tests
  reports no ImportError (its 8 "import file mismatch" errors come from test files sharing a basename and
  happen on main too). make check, ruff check, ruff format --check and mypy pass.
• Not exercised live: the connectors/test_postgresql.py fixture, which needs a release receipt for its port.