Skip to content

server: prepare safe multi-node api failover - #124

Draft
morajabi wants to merge 13 commits into
mainfrom
codex/multi-node-ha
Draft

morajabi wants to merge 13 commits into
mainfrom
codex/multi-node-ha

Conversation

@morajabi

@morajabi morajabi commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Prepare independently runnable API origins for Fly → Nuremberg → Finland failover. Depends on the database optimization work in #123; production routing remains unchanged.

Fix two gaps exposed by multi-node review: clear caches again when broker subscriptions recover, and recheck webhook claim/access authority after DNS before opening an external POST. Preserve at-least-once delivery and completion fencing. Idle standbys also avoid periodic discovery transactions when they have no connected users, and explicit-chat history drops one redundant SELECT while retaining its read-only snapshot and authorization checks.

Add actual two-process V3 qualification using the real TypeScript client: process-local temporary-key replacement on a new origin, brokerless durable catch-up, revocation on both origins, survival of a separate session, and continued service after one origin drains. Add true 100-chat user-bucket replay and revocation benchmarks to cover the tuned access queries before enabling broader preparation.

Provide local-only Cloudflare pool templates, Coolify node profiles and a sanitized metadata checker. Release digests remain intentionally unbound; the checker does not certify production readiness. Existing Redis/Valkey remains optional; PostgreSQL owns durable state. Full-role nodes run jobs even before ingress, and production activation remains separate.

Validation: paired MCP artifact build, server typecheck and lint passed. Focused suites passed: 33 history tests, 29 repair tests, 24 webhook tests, 11 cache tests, 3 user-replay tests, 6 profile tests, and the two-process V3 test (21 assertions). Independent reviews covered cache/admission races, webhook ownership and routing/TLS configuration. A clean-source 20-sample matrix at nominal added RTT6 measured access→all-custom history 162.81→99.63ms, full 100-chat user replay253.67→162.73ms, and revoked replay242.59→162.33ms. Calibration was7.2–7.7ms. Per-statement probes confirmed auto5custom/5generic versus all-custom10custom/0generic; custom planning cost2.3–2.6ms versus generic reuse0.05–0.08ms in EXPLAIN observations. Small synthetic tables do not qualify production skew, CPU or tail behavior; broad preparation stays opt-in. Hosted CI passed at 8c8f90a74: complete server tests/typecheck/lint, both native architecture images and runtime smoke, and all selected integration checks. No schema, production deployment, DNS/LB or customer-worker changes.

@morajabi
morajabi changed the base branch from codex/db-roundtrips to main October 15, 2026 17:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant