The nightly pg_dump of workwell_spike failed: run 32807396702.
Treat this as a production incident, not a backup problem. This job is the only always-on
process that opens a real connection to the real database, so it is our de-facto database
health check. The 15-minute self-heal reconciler probes /actuator/health, which is
deliberately DB-free and cannot detect a database outage — do not "fix" that by adding a
query to it, or the reconciler becomes a compute-pinning loop (see docs/DEPLOY.md →
"Database compute cost").
Most likely causes, in order:
- Database unreachable — check the Neon console. A
402 in the run log means the compute
quota is exhausted and every DB-backed route is already returning internal_error.
- Rotated/expired
DATABASE_URL_TWH or the backup IAM credentials.
pg_dump version drift against the server.
Quick check — if this returns 500s, the live site is down:
curl -s -o /dev/null -w '%{http_code}\n' https://twh-api-ts.os.mieweb.org/api/versionThis issue closes automatically on the next successful backup.
The nightly
pg_dumpofworkwell_spikefailed: run 32807396702.Treat this as a production incident, not a backup problem. This job is the only always-on
process that opens a real connection to the real database, so it is our de-facto database
health check. The 15-minute self-heal reconciler probes
/actuator/health, which isdeliberately DB-free and cannot detect a database outage — do not "fix" that by adding a
query to it, or the reconciler becomes a compute-pinning loop (see
docs/DEPLOY.md→"Database compute cost").
Most likely causes, in order:
402in the run log means the computequota is exhausted and every DB-backed route is already returning
internal_error.DATABASE_URL_TWHor the backup IAM credentials.pg_dumpversion drift against the server.Quick check — if this returns 500s, the live site is down:
curl -s -o /dev/null -w '%{http_code}\n' https://twh-api-ts.os.mieweb.org/api/versionThis issue closes automatically on the next successful backup.