Refresh table metadata when psycodict announces schema changes

[roed-math]

Companion PR to roed314/psycodict#111 (LISTEN/NOTIFY), showing what the consumer side looks like: the website picks up schema changes without restarting every worker. psycodict#111 has since merged and shipped, so nothing here is waiting on it: requirements.txt asks for released psycodict (psycodict[pgbinary]>=1.0.0rc1,<2), CI resolves 1.0.0rc2, and every version in that range provides both halves of the API. The refresher still tolerates an older psycodict, logging one line and staying a no-op, so an environment that has not upgraded keeps working. Still a draft pending psycodict 1.0.0 final (only release candidates are out so far).

What it does. Each web worker owns a SchemaRefresher (new module lmfdb/schema_refresh.py), driven by a before_request hook. The refresher keeps a db.listener() subscribed to the psycodict_schema channel; each request does a non-blocking poll(), a socket read on an idle dedicated connection. When notifications have arrived, it calls db.refresh_tables() (psycodict#99, already merged) once per batch, so added/dropped/renamed columns and tables become visible before the request is handled. Log lines record each refresh with the table names from the notification payloads. Only that steady-state poll is non-blocking: establishing a listener and refreshing metadata do talk to the database and can block, they just happen rarely. What check() does guarantee unconditionally is that no exception of its own reaches the request.

Policies this PR chooses — the things psycodict#111 deliberately leaves to the application:

• No background thread. Sync workers are single-threaded; polling at request boundaries avoids refresh-during-request races and costs nothing measurable. An idle worker lags until its next request, which is harmless: no requests means no queries to fail. A non-blocking lock keeps two callers from polling or refreshing at the same time, but it does not serialize refresh_tables() against queries running in other requests, so it is not on its own enough for threaded or gevent workers; the design assumes the single-threaded workers we actually run, where a request boundary is a moment with nothing in flight.
• Reconnect with catch-up. On listener failure: drop it, back off 30s, resubscribe — and do a full refresh_tables() on every (re)subscription, because notifications sent while unsubscribed are lost (LISTEN only delivers what is sent after it). The same catch-up covers the window between worker start and first subscription, and a failed refresh drops the listener so the next resubscription retries it.
• Whole-catalog refresh. The payload (a table name) is used for logging and for collapsing bursts into one refresh; refresh_tables() re-reads everything anyway, which handles create/drop/rename uniformly. Per-table refresh via the payload is an easy later optimization if the full refresh ever gets expensive.
• Fork-safety. Under gunicorn --preload, an inherited listener is abandoned (not closed — its socket is shared with the parent) and each worker builds its own.

Cost. One extra idle PG connection per worker, one non-blocking poll per request, one catch-up refresh per worker per (re)subscription; refreshes otherwise happen only when the schema actually changes.

Tests. lmfdb/tests/test_schema_refresh.py unit-tests the control flow with stub db/listener objects (subscribe + catch-up, foreign-channel filtering, duplicate-payload collapse, subscription-failure backoff and recovery, lost-listener resubscribe, failed-refresh retry, hot-standby permanent-disable, fork handling) and passes against any psycodict version; the live NOTIFY→LISTEN round trip is covered by psycodict's own tests in LMFDB#111. Filtering and dedupe are asserted on the refresh reason the refresher computes, not merely on a refresh count, so dropping the channel filter or the sorted(set(...)) collapse fails the suite (checked by mutating each in turn). The file is registered in matrix_includes.json under both proddb and devmirror, and the test-file count guard in python-package.yml is bumped to 45. Verified locally: 9 passed under sage -python -m pytest, pyflakes/pylint/ruff clean.

Deployment topology (verified live). devmirror is a physical hot standby (PG 18.1, pg_is_in_recovery() = true): a server in recovery refuses LISTEN outright (cannot execute LISTEN during recovery, SQLSTATE 25006), and NOTIFY is not WAL-logged, so notifications cannot reach websites pointing at it no matter what the sync does — no trigger or event trigger on devmirror can help (triggers don't fire during WAL replay, and a standby cannot NOTIFY). The refresher therefore treats SQLSTATE 25006 at subscribe time as permanent and disables itself for the life of the process, so development copies of the website (which read devmirror) keep today's behavior — restart to pick up schema changes — with a single info log line instead of a retry loop. Production/beta webservers, whose schema changes are applied by psycodict-driven processes on the primaries they read from, get the live refresh.

We've decided not to pursue automatic refresh for dev copies, since prod/beta don't need it. For the record, the workable design would be a replicated schema-generation marker: _notify_schema_change additionally bumps a one-row meta_schema_gen table in the same transaction on the primary. The row replicates through WAL in commit order — so by the time a website sees the bump, the schema change itself has already replayed on the standby (a guarantee primary-side LISTEN cannot give) — and a website that cannot LISTEN polls that single-row SELECT at most every few seconds. That is a small psycodict follow-up plus ~15 lines here; the refresh-on-UndefinedColumn fallback discussed in psycodict#99 remains a complementary option.

🤖 Generated with Claude Code

[roed-math]

Addressed the review: the two blocking CI fixes, plus the test and documentation accuracy items.

Required, and the reason every shard was red

• matrix_includes.json: lmfdb/tests/test_schema_refresh.py added to the paired proddb/devmirror entries that already carry lmfdb/tests. It now appears exactly twice; folders already listed tests, so it is untouched.
• python-package.yml: test-file count guard 44 -> 45. Every shard died at that guard before reaching tests or lint, so the previous run's failures carried no signal about the code underneath.

Test accuracy (item 3)

The old batch did mix in a foreign channel, but only asserted a refresh count, so it would have passed with the channel filter deleted; and it never repeated a payload, so it never exercised the deduplication the description claims. Now:

• test_other_channels_do_not_refresh: a batch made up entirely of another channel's notifications, asserting the refresh count does not move.
• test_batch_collapses_duplicates_and_filters_channels: a burst repeating two tables alongside a foreign notification, asserting on the reason string the refresher computes (via a small RecordingRefresher subclass) rather than on a count, so each table appears once, in sorted order, and the foreign payload appears nowhere.

Confirmed by mutation: deleting the channel == SCHEMA_CHANNEL filter fails both new tests, and replacing sorted({...}) with a plain list fails the second. Still unit-only, no live PostgreSQL.

Documentation (items 4, 5, 6)

• The "never blocks" guarantee was too broad. The class docstring now says that only the steady-state poll of an established listener is non-blocking, that check() never lets an exception of its own reach the request, and that establishing a listener and refreshing metadata do talk to the database and may block. The before_request docstring in app.py made the same claim in miniature and got the same correction.
• The lock comment and the "no background thread" bullet now say what the lock actually buys: it keeps two callers off the same listener, but does not serialize refresh_tables() against queries running in other requests, so it is not on its own enough for threaded or gevent workers. The design assumes the single-threaded sync workers we run, where a request boundary is a moment with nothing in flight.
• Old-psycodict wording: "logs once and disables itself" -> "logs once and remains a no-op", which is what the code does. Documentation fix only; _disabled stays reserved for the hot-standby case, where retrying really is pointless.

Description (item 7)

Rewritten: psycodict#111 merged 2026-07-22 and shipped in 1.0.0rc1, and requirements.txt asks for released psycodict (psycodict[pgbinary]>=1.0.0rc1,<2) rather than git master, so the "not yet merged / installs from git master / try it with a branch install" framing is gone. The threaded/gevent claim and the blanket non-blocking claim are corrected there too. Left as a draft since only release candidates of psycodict 1.0.0 exist so far; happy to mark it ready if you would rather not wait for the final release.

No behavior changed anywhere: the two workflow files, the two docstring/comment passages, the app.py docstring, and the tests.

Local: 9 passed under sage -python -m pytest lmfdb/tests/test_schema_refresh.py, pyflakes/pylint/ruff clean. find lmfdb \( -name 'test_*.py' -o -name '*_test.py' \) | wc -l reports 45, and the matrix parses with the new file scheduled exactly once per database target.

CI is queued rather than running at the moment, so I have not yet seen a shard get past the guard on the hosted runners; that is the thing to watch on this push, since every shard previously stopped at checking that we didn't miss any test files.

[roed314]

GPT signed off.