Skip to content

server: synchronize qualified standby before fly releases - #125

Closed
morajabi wants to merge 5 commits into
codex/multi-node-hafrom
codex/standby-release-sync
Closed

morajabi wants to merge 5 commits into
codex/multi-node-hafrom
codex/standby-release-sync

Conversation

@morajabi

@morajabi morajabi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Ordinary manual Fly releases currently leave the warm ARM standby stale. Require a healthy standby preflight before DDL, synchronize the qualified ARM image before Fly replacement, and verify exact standby image/runtime intent and direct trusted TLS readiness before and after Fly. Publication-only and the separately protected first-API bootstrap retain their existing behavior.

Fixed-resource consumers validate original manifest/layer/rootfs identity and resource bounds, preserve runtime configuration, journal uncertain writes, and refuse blind redispatch or stale/concurrent releases. A restricted controller relay keeps its bearer on Finland. An idempotent hash-pinned installer writes only public helpers/dedicated directories, rejects conflicts, and never creates credentials/keys or starts/restarts services. A strict read-only probe branches before release journal handling and explicitly cannot qualify a release. Runtime credential/key installation and real rollout acceptance remain separate gates; no serving-container deployment ran as part of this PR.

Validation: final head cfb0c6f0f has 24/24 integration checks green, including 29 consumer/probe and 5 installer fixtures plus workflow contracts. The separate four-job Server Tests run at 175961748 passed, including real digest-preserving ARM archive/rootfs validation; registry setup/publication steps were skipped. Runtime/build tooling is unchanged; the expanded runbook changes documentation in the exported context, so no exact full-context or final-head labeled-image qualification is claimed. Earlier 1,949-file equality proof applies only to 175961748→8ef6b173f. No local Docker or production failure injection.

Stacked on codex/multi-node-ha at f497aacb7; do not merge the HA base or activate production through this draft. The runbook retains separately reviewed independent emergency retained-image deployment and distinguishes routine paired releases.

@morajabi

Copy link
Copy Markdown
Contributor Author

Closing as superseded; branch kept for reference. The paired-release design gated every Fly release on GitHub Actions, Fly, Nuremberg and the Finland controller all being healthy, and once merged, STANDBY_SYNC_ENABLED=false would block releases rather than fall back to Fly-only. The agreed replacement (Secret Sauce .context/2026-10-17-ha-decisions/final-ops-design.md, W1): Fly releases stay independent; Nuremberg follows via inline-ops nbg sync (inline-private#6/#7, used for the b5a221a release); /readyz reports the serving revision (#126); and the Finland observer alerts when the origins differ for 45+ minutes (inline-private#9).

@morajabi morajabi closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant