Found during the production-readiness investigation.
Symptom / failure scenario: the ingestor's reconnect loop (apps/engine/src/replication.rs:42-56) retries pg::connect with a flat 500ms sleep and resumes peeking — ensure_slot runs only in setup_postgres at boot. After a failover to a promoted replica (or a dropped/invalidated slot), the slot doesn't exist: the loop errors forever. If the process is instead rebooted, ensure_slot silently creates a fresh slot at the new head LSN, losing every change in the gap — shapes are silently missing data with no signal.
There is also no timeline/system-id detection (upstream persists {pg_system_identifier, timeline_id} and purges all shapes on mismatch), and no exponential backoff.
Fix direction: persist system-id/timeline and compare at (re)connect → on mismatch or slot-gone, purge all shapes and force client refetch rather than continuing; exponential backoff with jitter; export a retained-WAL gauge. Reference: sync-service timeline.ex, connection/manager.ex:452-500, db_connection_error.ex.
🤖 Generated with Claude Code
Found during the production-readiness investigation.
Symptom / failure scenario: the ingestor's reconnect loop (
apps/engine/src/replication.rs:42-56) retriespg::connectwith a flat 500ms sleep and resumes peeking —ensure_slotruns only insetup_postgresat boot. After a failover to a promoted replica (or a dropped/invalidated slot), the slot doesn't exist: the loop errors forever. If the process is instead rebooted,ensure_slotsilently creates a fresh slot at the new head LSN, losing every change in the gap — shapes are silently missing data with no signal.There is also no timeline/system-id detection (upstream persists
{pg_system_identifier, timeline_id}and purges all shapes on mismatch), and no exponential backoff.Fix direction: persist system-id/timeline and compare at (re)connect → on mismatch or slot-gone, purge all shapes and force client refetch rather than continuing; exponential backoff with jitter; export a retained-WAL gauge. Reference: sync-service
timeline.ex,connection/manager.ex:452-500,db_connection_error.ex.🤖 Generated with Claude Code