Skip to content

docs(availability): put today-vs-after availability at the top #612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swares merged 1 commit into from
Oct 3, 2026
Merged

docs(availability): put today-vs-after availability at the top #612

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions docs/AVAILABILITY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,24 @@ have drifted apart; this document reconciles them.
| VMs | Every host or guest reboot is a GitLab outage | Host maintenance: < 1 s pause. Guest reboot: still an outage **[HW-LIMIT]** |
| Bare metal | n150-1/2 never patched; weekly avoidable reboots | All patched; reboots rare (Livepatch); unavoidable residue is the NAS **[HW-LIMIT]** |

### At a glance: availability today and after the plan

Planned events only (patching, upgrades, image updates), estimated from the manifests
and playbooks under the assumptions in §5. Unplanned failures are excluded. Phase 0 of
the plan replaces these estimates with load-tested measurements.

| Service | Today | After Phases 1–5 | Remaining cause |
|---|---|---|---|
| Ingress | ~99.995 % | **≥ 99.999 %** | — |
| SSO | ~99.99 % | **≥ 99.999 %** | — |
| AI gateway | ~99.97 % | **≥ 99.999 %** gateway; backends at half capacity during agent maintenance | One NPU node down = half capacity **[HW-LIMIT]** |
| Immich (web/API) | ~99.95–99.97 % | **≥ 99.99 %** | Originals unavailable during H4 reboots **[HW-LIMIT]** |
| Home Assistant | ~99.97 % | **~99.99 %** | Single-instance by design; ~30–90 s per HA release |
| MQTT | ~99.97 % | **≥ 99.99 %** | Session takeover on broker failover |
| GitLab | — | Host maintenance non-disruptive | Guest reboots 3–8 min **[HW-LIMIT]** |
| NAS (NFS) | ~99.99 % | **~99.995 %** (rarer reboots) | Single storage node **[HW-LIMIT]** |
| LAN DNS | degraded weekly | **no stalls** | — |

---

## 2. What "uninterrupted" means here
Expand Down Expand Up @@ -129,7 +147,7 @@ home-assistant, lldap, minio, zot, semaphore, and all four Postgres instances
| rpi5 (Vault) | Auto-unseal; ESO serves cached secrets | 0 s user-visible; ESO sync errors 1–2 min |
| octopi (Pi-hole primary) | LAN clients that list it first wait the resolver timeout (~5 s on glibc) per lookup before failing over | 1–2 min of slow lookups, no hard failure. Cluster unaffected (octopi is last) |
| opi-zero2w-2 (MQTT) | Home Assistant connects only to `.188`; the bridge to `-4` replicates topics but is **not** client failover | **1–2 min MQTT outage; QoS-0 messages lost** |
| xu3-1 | Build agent | Irrelevant to service |
| xu3-1 | No service role (not a build agent; BACKLOG §2.16) | Irrelevant to service |

### 4.5 Pi-hole application update (`update-non-apt.yml -t pihole`)
- **Only octopi is updated.** The play header and `UPDATES.md` say "secondary first",
Expand Down Expand Up @@ -222,7 +240,7 @@ because the volume cannot follow the pod.

Ordered so each phase is useful on its own and later phases build on earlier ones.
Every item is a GitOps PR or an Ansible change run with `--check` first, per CLAUDE.md.
**Decisions marked ◆ need your sign-off before any code is written** (§9).
**Decisions marked ◆ need your sign-off before any code is written** (§8).

### Phase 0 — Measure first (½ day, no risk)

Expand Down Expand Up @@ -327,29 +345,11 @@ here is a proposal for you, not something to be performed.
- **Network and power.** One router/switch, no documented UPS. These dominate
*unplanned* availability more than anything above. **[HW-LIMIT]** — a UPS for the
rack and the switch is the single highest-value hardware purchase for this goal.
- **xu3-1** build agent — not on any served path.

---

## 8. Projected availability after the plan

Same assumptions as §5 (planned events only).

| Service | Today | After Phases 1–5 | Remaining cause |
|---|---|---|---|
| Ingress | ~99.995 % | **≥ 99.999 %** | — |
| SSO | ~99.99 % | **≥ 99.999 %** | — |
| AI gateway | ~99.97 % | **≥ 99.999 %** gateway; backends at half capacity during agent maintenance | One NPU node down = half capacity **[HW-LIMIT]** |
| Immich (web/API) | ~99.95–99.97 % | **≥ 99.99 %** | Originals unavailable during H4 reboots **[HW-LIMIT]** |
| Home Assistant | ~99.97 % | **~99.99 %** | Single-instance by design; ~30–90 s per HA release |
| MQTT | ~99.97 % | **≥ 99.99 %** | Session takeover on broker failover |
| GitLab | — | Host maintenance non-disruptive | Guest reboots 3–8 min **[HW-LIMIT]** |
| NAS (NFS) | ~99.99 % | **~99.995 %** (rarer reboots) | Single storage node **[HW-LIMIT]** |
| LAN DNS | degraded weekly | **no stalls** | — |
- **xu3-1** — no service role (not a build agent; BACKLOG §2.16), not on any served path.

---

## 9. Decisions needed before any code
## 8. Decisions needed before any code

1. **S1/S2/S3 — storage stack.** CNPG + Longhorn + Garage is the recommendation. The
alternative is CNPG only (Phase 2 for databases) and accepting `local-path` for
Expand All @@ -366,7 +366,7 @@ alone should remove most of the *request-visible* errors without touching storag

---

## 10. How to prove it
## 9. How to prove it

The claim is true for a layer when **M2's load test, run during that layer's
maintenance process, meets §2's criteria**:
Expand Down
Loading