diff --git a/BACKLOG.md b/BACKLOG.md index 0253978..83595fa 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -4,9 +4,14 @@ Swept from every `.md` file and all code/config in the repo. The sections it replaced have been collapsed to pointers: `README.md`, `docs/OVERVIEW.md`, `docs/services.md`, -and the dated `TODO-2026-07-14.md` / `TODO-2026-07-23.md` / `TODO-2026-08-03.md` files, +and the dated `archive/TODO-2026-07-14.md` / `archive/TODO-2026-07-23.md` / `archive/TODO-2026-08-03.md` files, which are now historical session notes — keep them for the reasoning, not the tasks. +**Moved 2026-10-03.** The dated TODO files now live in `archive/`, unedited, so every +`file:line` citation below still resolves at its new path. `README.md` was restructured +the same day, so `README.md:` citations in this file refer to commit `6cb781f`, +the last commit before that change. + The reason for consolidating: those six lists disagreed. Offsite backup was marked **DONE** in three of them while the unit had never copied a byte, the Vault root token was listed as a required credential in three more after being deliberately eliminated, @@ -324,7 +329,7 @@ Note the doc's list still says "Vault root token" — as of 08-07 there delibera isn't one. See §6.3. ### 1.3 ~~Finish the offsite tier~~ — **DONE, verified running 08-15** -`TODO-2026-08-03.md:274`, `ansible/playbooks/backup-offsite.yml:59` +`archive/TODO-2026-08-03.md:274`, `ansible/playbooks/backup-offsite.yml:59` **SEED COMPLETE 2026-08-11 02:19 UTC.** ~3.1 days wall clock, 44m35s CPU, ~204 GiB. Retention applied on the first run, keeping 32 snapshots across five groups @@ -894,7 +899,7 @@ So the chain from the envelope is: unseal with item 5 → root shell on the rpi5 config → generate-root → full admin. It works. It just needs host root as well as the envelope, which in a rebuild you would have anyway. -**And it was documented twice.** `TODO-2026-08-03.md:727-814` carries a section titled +**And it was documented twice.** `archive/TODO-2026-08-03.md:727-814` carries a section titled *"Why `generate-root` was blocked — and the actual fix"*: the identical 403 with no token, the `enable_unauthenticated_access` remedy, and at `:760` the full list of families — **`"rekey"`, `"generate-root"`, `"generate-operation-token"`**. The `rekey` family is @@ -1771,7 +1776,7 @@ The playbook was deleted rather than fixed; see §9. It preseeded slapd and boun header said "never hardcode it". It had also been dead since 2026-07-18. ### 2.5 ~~Vault unseal keys sit beside the sealed data~~ — **claim corrected 08-15; storage kept, deliberately** -`ansible/templates/vault-unseal.sh.j2:6`, `TODO-2026-08-03.md:871-876` +`ansible/templates/vault-unseal.sh.j2:6`, `archive/TODO-2026-08-03.md:871-876` `docs/SECURITY.md` described the shares as "Offline, physically secure". They are on the RPi5 at `0400`, next to the raft store they unseal, so root there is equivalent to full @@ -1991,7 +1996,7 @@ for. All three were written from reading the repo. All three corrections came fr measuring the running system. ### 2.8 Optional: tighten `admin` so it cannot self-escalate -`ansible/files/vault-policies/admin.hcl`, `TODO-2026-08-03.md:310` +`ansible/files/vault-policies/admin.hcl`, `archive/TODO-2026-08-03.md:310` `admin` holds `create` plus `sudo` on `sys/policies/acl/*` and `auth/*`, so a token carrying it can write a policy granting anything and mint a token for it — root at @@ -3257,7 +3262,7 @@ package manager said, and warns by name if a host process holds 9100 on a cluste cannot reach the LAN. ### 3.7 Alloy restart recency filter -`TODO-2026-08-03.md:335` — benign historical restarts keep surfacing. +`archive/TODO-2026-08-03.md:335` — benign historical restarts keep surfacing. ### 3.8 `PolicyViolation` events fire continuously against dead ReplicaSets @@ -6019,17 +6024,20 @@ them. - [ ] **`Makefile:2` still lists `ldap` in `.PHONY`** though the target was removed on 08-09 and the file carries a comment explaining the removal. Harmless; finish the job. -- [ ] **`FETCH_HEAD` is tracked in git** — a git internal committed by accident. -- [ ] **Duplicated, drifted CI config under `ansible/`** — `ansible/.gitlab-ci.yml`, +- [x] **`FETCH_HEAD` is tracked in git** — a git internal committed by accident. + Removed 2026-10-03. +- [x] **Duplicated, drifted CI config under `ansible/`** — `ansible/.gitlab-ci.yml`, `ansible/.github/workflows/*`, `ansible/.claude/settings.json`. Only the root copies - are read; these are dead weight diverging silently. + are read; these are dead weight diverging silently. Moved to + `archive/ansible-stale-ci/` 2026-10-03. - [ ] **Mixed ArgoCD `repoURL` schemes** — 12 files use `git@github.com:`, but `gitops/apps/gitlab-runner.yaml:18,22` uses `https://`. Argo treats these as two repos needing two credentials. - [ ] **`.gitignore` path bug in `ansible/`** (C7) — and note this working copy lives in `Downloads`, so `LAB-URLS.md` and similar travel with any zip or sync of the folder. -- [ ] **Archive `TODO-2026-07-14.md`** (25/26 done) and collapse `README.md`'s TODO to a - pointer at this file. +- [x] **Archive `TODO-2026-07-14.md`** (25/26 done) and collapse `README.md`'s TODO to a + pointer at this file. Done 2026-10-03: all three dated TODO files moved to `archive/` + unedited, and the README section is now a pointer. ### 7.y This file's own status was wrong — **SWEPT 2026-09-20** @@ -6455,7 +6463,7 @@ work, invisible for months, in a section titled *small, live, cheap*. state (`backup-etcd.sh.j2:14` mentions the `state.db` rename) and delete. - [ ] `lldap-backup` init container failed once and self-healed; a retry loop around - `pg_dump` would make it deterministic (`TODO-2026-08-03.md:447-452`). + `pg_dump` would make it deterministic (`archive/TODO-2026-08-03.md:447-452`). - [x] **Re-sweep Ansible for `no_log` registers consumed later** — done 08-09. All 12 files carrying `no_log` were checked. Results, so this need not be redone: diff --git a/FETCH_HEAD b/FETCH_HEAD deleted file mode 100644 index e69de29..0000000 diff --git a/README.md b/README.md index 8c951a6..f3d724f 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,16 @@ -# Home Lab — DevOps Environment +# HomeLab — GitOps platform on a 14-host x86 + ARM64 fleet -A GitOps-managed home lab built around an **Odroid-H4 Ultra** NAS core and a -**3-node HA k3s cluster** (H4 + two N150 mini PCs), with an ARM64 inference fleet -(two Orange Pi 5 Pros). Infrastructure is defined as code: **Ansible** stands hosts up, -**Argo CD** runs everything inside the cluster from this git repo, and a two-tier storage -model (hot NVMe + two cold SATA RAID 1 mirrors) keeps data safe. +A production-style platform run at home: a **3-node HA k3s cluster**, everything in it +reconciled by **Argo CD** from this repo, hosts built by **Ansible** and **OpenTofu**, +secrets from **HashiCorp Vault**, policy enforced by **Kyverno**, and a full +**Prometheus / Grafana / Loki** observability stack. An ARM64 tier serves local AI +inference on NPUs behind an OpenAI-compatible gateway. Backups go to two RAID 1 mirrors +and offsite to Cloudflare R2, and **restores are drilled, not assumed**. + +**Cloud counterpart:** [HomeLab-aws](https://github.com/swares/HomeLab-aws) builds an +EKS cluster from nothing, runs a slice of the same stack (Argo CD, Kyverno, LiteLLM) with +IRSA and an ALB, and destroys it every night for about $1 a session. It is deliberately +detachable: nothing in this repo depends on it. ```mermaid flowchart TB @@ -36,10 +42,25 @@ flowchart TB class BK c; ``` +## What this demonstrates + +| Skill | Where to look | +|---|---| +| **GitOps end to end.** Every change is a PR. Argo CD app-of-apps plus a git-directory ApplicationSet, `selfHeal` and `prune` on, rollback is `git revert` | `gitops/`, `docs/UPDATES.md` | +| **CI guardrails before merge.** YAML lint, kubeconform schema validation, ApplicationSet collision checks, and OPA/conftest policies run on every PR | `.github/workflows/validate.yml`, `ci/policies/` | +| **Policy as code in the cluster.** Kyverno in Enforce mode: no `:latest`, no privileged pods, resource limits required | `gitops/workloads/kyverno/` | +| **Secrets management.** Vault (KV v2, policies in git, no standing root token) feeding External Secrets Operator; Ansible Vault for host secrets | `ansible/files/vault-policies/`, `docs/SECURITY.md` | +| **Identity.** Authelia OIDC single sign-on backed by lldap; cert-manager private CA for TLS | `docs/SSO.md` | +| **Observability.** kube-prometheus-stack, Loki, Alloy on every node, blackbox probes, Alertmanager | `gitops/apps/monitoring.yaml`, `gitops/workloads/monitoring/` | +| **Infrastructure as code.** Ansible for 14 hosts across x86 and ARM64, OpenTofu with remote state, Packer images, nightly drift checks, Renovate dependency PRs | `ansible/`, `tofu/`, `packer/`, `renovate.json` | +| **Backup and disaster recovery.** restic to two local mirrors and offsite R2, etcd and Vault snapshots, and restore drills that actually restored data | `docs/BACKUP-RESTORE.md` | +| **Incident response.** A written incident record with root cause, fix, and proof by unattended reboot | `docs/INCIDENT-2026-08-23-h4-boot.md` | +| **AI platform engineering.** LiteLLM gateway routing to Ollama, NPU-native RKLLama, and Whisper, plus an adapter that makes edge devices an inference backend | `docs/AI-INFERENCE.md`, `docs/AI-ROUTING.md` | + ## Why it's shaped this way - **k3s, not full Kubernetes** — the H4 is also the NAS. k3s runs as a single systemd - service alongside Samba/NFS and leaves most of the box free. Traefik is the default + service alongside the NFS server and leaves most of the box free. Traefik is the default ingress; workloads use `networking.k8s.io/v1 Ingress`, not OpenShift Routes. - **Argo CD, not imperative ops** — change the cluster by editing git and opening PRs. Argo reconciles with `selfHeal` on, so drift reverts and rollback is `git revert`. @@ -56,6 +77,9 @@ flowchart TB | `gitops/` | What Argo deploys — `bootstrap/` (app-of-apps), `apps/`, `workloads/` | | `docs/` | Architecture, hardware, runbook, security, AI inference, service catalog, updates | | `scripts/` | One-shot helpers (`enable-winrm.ps1`, `lab-check.sh`, flannel FDB service) | +| `ci/` | OPA/conftest policies run by CI against every workload manifest | +| `tofu/`, `packer/` | OpenTofu (VMs, DNS) and Packer image builds | +| `archive/` | Superseded files kept for reference; nothing here is live | | `CLAUDE.md` | Operating rules — read before touching anything | ## What's running @@ -81,54 +105,25 @@ flowchart TB | cert-manager | `cert-manager` | TLS via `lab-ca` ClusterIssuer (self-signed root CA) | | Argo CD | `argocd` | GitOps controller — selfHeal + prune on all apps | -### Host services (outside k3s) - -| Service | Host | IP | -|---------|------|----| -| HashiCorp Vault | rpi5 | 192.168.1.128 | -| Mosquitto MQTT (primary) | opi-zero2w-2 | 192.168.1.188 | -| Mosquitto MQTT (secondary, HA bridge) | opi-zero2w-4 | 192.168.1.99 | -| Pi-hole DNS (primary, v6.4.3) | octopi (RPi 3B #2) | 192.168.1.148 | -| Pi-hole DNS (secondary, v6) | rpi4b (RPi 4B) | 192.168.1.116 | -| dnsmasq DNS (tertiary fallback) | opi-zero2w-1 | 192.168.1.184 | -| Samba / NFS (NAS) | H4 (host) | 192.168.1.160 | -| GitLab CE | gitlab-1 VM on n150-1 | 192.168.1.50 | - -### Ingress endpoints - -| URL | Service | -|-----|---------| -| `argocd.apps.lab.home.arpa` | Argo CD | -| `grafana.apps.lab.home.arpa` | Grafana | -| `immich.apps.lab.home.arpa` | Immich | -| `ai.apps.lab.home.arpa` | LiteLLM gateway | -| `ha.apps.lab.home.arpa` | Home Assistant | -| `authelia.apps.lab.home.arpa` | Authelia (SSO) | -| `lldap.apps.lab.home.arpa` | lldap (LDAP directory UI) | -| `semaphore.apps.lab.home.arpa` | Semaphore (Ansible UI) | -| `minio.apps.lab.home.arpa` | Minio (S3 API) | -| `minio-console.apps.lab.home.arpa` | Minio console | -| `stt.apps.lab.home.arpa` | Whisper STT | -| `*.apps.lab.home.arpa` | → 192.168.1.201 (kube-vip service VIP for Traefik) | - -## Fleet - -| Host | Ansible name | IP | Role | -|------|--------------|----|------| -| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS (smbd/NFS) | -| N150 mini PC #1 | n150-1 | 192.168.1.42 | k3s server + KVM hypervisor (Ubuntu 24.04) | -| N150 mini PC #2 | n150-2 | 192.168.1.21 | k3s server + KVM hypervisor (Ubuntu 24.04) | -| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, RKLLama/Ollama inference, NPU | -| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, RKLLama/Ollama inference, NPU | -| Raspberry Pi 5 | rpi5 | 192.168.1.128 | HashiCorp Vault | -| Raspberry Pi 4B | rpi4b | 192.168.1.116 | Pi-hole secondary DNS (v6, Bookworm) | -| RPi 3B #2 (octopi) | octopi | 192.168.1.148 | Pi-hole primary DNS (v6.4.3, Bookworm) | -| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed) | -| OPi Zero 2W #1 | opi-zero2w-1 | 192.168.1.184 | dnsmasq DNS tertiary fallback | -| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT primary broker | -| OPi Zero 2W #3 | opi-zero2w-3 | 192.168.1.217 | dnsmasq — configured, but NOT in `lab_dns_servers` (see docs/HARDWARE.md) | -| OPi Zero 2W #4 | opi-zero2w-4 | 192.168.1.99 | MQTT secondary broker (HA bridge) | -| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent | +### Host services, endpoints and fleet + +Outside the cluster, the fleet runs HashiCorp Vault (Raspberry Pi 5), redundant Pi-hole +DNS with a dnsmasq fallback, an HA Mosquitto MQTT pair, the NFS server on the H4, and GitLab +CE in a KVM VM. Every service is exposed at `*.apps.lab.home.arpa` through a kube-vip +VIP in front of Traefik. + +| Host class | Count | Role | +|---|---|---| +| Odroid-H4 Ultra (x86) | 1 | k3s server + NAS | +| N150 mini PC (x86) | 3 | two k3s servers + KVM hypervisors, one Windows host (WinRM) | +| Orange Pi 5 Pro (ARM64, RK3588 NPU) | 2 | k3s agents, LLM inference | +| Raspberry Pi 5 / 4B / 3B | 3 | Vault, primary and secondary DNS | +| Orange Pi Zero 2W | 4 | MQTT pair, fallback DNS | +| Odroid XU3 | 1 | Legacy board on Ubuntu 16.04; retire or rebuild pending (`BACKLOG.md` §2.16) | + +Full detail lives in one place each, so it can't drift between copies: +[docs/HARDWARE.md](docs/HARDWARE.md) owns hosts and IPs, +[docs/services.md](docs/services.md) owns the service catalog and endpoints. ## Quickstart (fresh bootstrap) @@ -182,38 +177,28 @@ See [docs/SECURITY.md](docs/SECURITY.md) for the full security model. - **Never** `mkfs`/`wipefs` the cold disks (`/dev/md0`, `/dev/md1`) - **Never** run `restic forget`/`prune` by hand — retention is handled by backup timers only -- **Never** stop `smbd`, `nfs`, `backup-nas`, or `backup-etcd` +- **Never** stop `nfs-server`, `backup-nas`, or `backup-etcd`. There is no Samba on the H4; NFS is the only export path - Before any hot-tier storage change: confirm last backup succeeded -## TODO +## Open work **All open work lives in [`BACKLOG.md`](BACKLOG.md).** It is the single list, swept from every document and from the code, and ordered by what happens if an item is ignored. -This section used to carry its own list. It drifted: it claimed offsite restic backup -was done (it had never copied a byte), and it duplicated items that also appeared in -`docs/OVERVIEW.md`, `docs/services.md`, `docs/STANDUP.md` and three dated `TODO-*.md` -files, each with a different idea of what was outstanding. One list or none. - -### Done ✅ -- [x] octopi flashed to Bookworm, Pi-hole v6.4.3 running (2026-07-13) -- [x] RPi 4B: Pi-hole v6 secondary DNS live at 192.168.1.116 (2026-07-02) -- [x] n150-1/n150-2 joined as k3s server nodes, kube-vip VIP 192.168.1.200 (2026-07-02) -- [x] lldap migrated from ldap-1 VM to k3s Deployment in `lldap` namespace (2026-07-04) -- [x] Authelia → PostgreSQL backend (2026-07-03) -- [x] Immich library → NFS ReadWriteMany PV (schedules on any node) (2026-07-04) -- [x] Shared NFS storage between n150-1/n150-2 for VM live migration (2026-07-03) -- [x] Monitoring stack migrated to n150-1 (2026-07-04) -- [x] zswap on n150-1/n150-2 (zstd, zsmalloc, 20%) (2026-07-03) -- [x] MQTT HA: opi-zero2w-4 secondary broker with bidirectional bridge (2026-07-10) -- [x] Kyverno 3 ClusterPolicies in Enforce mode (2026-07-14) -- [x] ArgoCD notifications + git-directory ApplicationSet (2026-07-14) -- [x] Semaphore Ansible UI live (2026-07-18) -- [x] OpenTofu state in Minio; gitlab-1 VM codified (2026-07-18) -- [x] Offsite restic replication to Cloudflare R2 (`homelab-nas`) — wired 2026-08-07. - Note: the `backup-offsite.timer` line previously here was **false**. The unit - existed but `offsite_restic_repo` was never set, so it exited 0 nightly and - reported PASSED without copying anything. See `BACKLOG.md` §1.3. -- [x] restic repository password rotated on both local repos (2026-08-07) -- [x] Vault: no standing root token; policies under git (2026-08-07) +It exists because the lists drifted. An earlier TODO here claimed offsite restic backup +was done when it had never copied a byte: the unit existed, but `offsite_restic_repo` was +never set, so it exited 0 nightly and reported PASSED. It was fixed and verified on +2026-08-07 (`BACKLOG.md` §1.3). A green status over unfinished work is the failure this +repo now checks for. + +The dated `TODO-2026-*.md` files that preceded `BACKLOG.md` are kept, unedited, in +[`archive/`](archive/) because `BACKLOG.md` cites them by line number. + +## Related repositories + +| Repo | What it is | +|---|---| +| [HomeLab-aws](https://github.com/swares/HomeLab-aws) | Ephemeral EKS sandbox: OpenTofu, IRSA, ALB controller, ordered nightly teardown | +| [My_M5Stack_Core_Framework](https://github.com/swares/My_M5Stack_Core_Framework) | Edge sensor and inference firmware; its adapter runs in this cluster | +| [HostMon](https://github.com/swares/HostMon) | ESP32-S3 network-monitoring appliance that alerts into the edge tier | diff --git a/archive/README.md b/archive/README.md new file mode 100644 index 0000000..157f482 --- /dev/null +++ b/archive/README.md @@ -0,0 +1,17 @@ +# archive/ + +Superseded files kept for reference. **Nothing here is live.** No playbook, workflow, +or Argo Application reads from this folder. + +Moved here on 2026-10-03 to clear the repo root. The files themselves are unedited. + +| File | What it was | Why it's kept | +|---|---|---| +| `TODO-2026-07-14.md`, `TODO-2026-07-23.md`, `TODO-2026-08-03.md` | Dated session task lists, replaced by [`../BACKLOG.md`](../BACKLOG.md) on 2026-08-07 | `BACKLOG.md` cites them by `file:line` as evidence. Moving a file keeps its line numbers, so those citations still hold at the new path. **Do not edit these files**, or the citations break | +| `m5fw-contrib-README.md` | Build and apply procedure for contributing the adapter code to the M5Stack framework repo | Historical; the live adapter docs are `gitops/workloads/ai-gateway/m5stack-adapter/README.md` | +| `protocol.py` | An older copy of the M5Stack device-protocol client | The canonical client lives in the framework repo, and the deployed copy is `gitops/workloads/ai-gateway/m5stack-adapter/protocol.py` | +| `sample-speech-1m.wav` | A one-minute speech sample, apparently for Whisper STT testing | Kept in case a test still expects it (5 MB) | +| `ansible-stale-ci/` | `ansible/.github/`, `ansible/.gitlab-ci.yml`, `ansible/.claude/` | Drifted duplicates of the root copies. Neither GitHub nor GitLab ever read them from that location (`BACKLOG.md` §7.1) | + +The empty, accidentally committed `FETCH_HEAD` was deleted rather than archived. It was +a git internal file, not content. diff --git a/TODO-2026-07-14.md b/archive/TODO-2026-07-14.md similarity index 100% rename from TODO-2026-07-14.md rename to archive/TODO-2026-07-14.md diff --git a/TODO-2026-07-23.md b/archive/TODO-2026-07-23.md similarity index 100% rename from TODO-2026-07-23.md rename to archive/TODO-2026-07-23.md diff --git a/TODO-2026-08-03.md b/archive/TODO-2026-08-03.md similarity index 100% rename from TODO-2026-08-03.md rename to archive/TODO-2026-08-03.md diff --git a/ansible/.claude/settings.json b/archive/ansible-stale-ci/.claude/settings.json similarity index 100% rename from ansible/.claude/settings.json rename to archive/ansible-stale-ci/.claude/settings.json diff --git a/ansible/.github/workflows/scheduled-updates.yml b/archive/ansible-stale-ci/.github/workflows/scheduled-updates.yml similarity index 100% rename from ansible/.github/workflows/scheduled-updates.yml rename to archive/ansible-stale-ci/.github/workflows/scheduled-updates.yml diff --git a/ansible/.github/workflows/sync-check.yml b/archive/ansible-stale-ci/.github/workflows/sync-check.yml similarity index 100% rename from ansible/.github/workflows/sync-check.yml rename to archive/ansible-stale-ci/.github/workflows/sync-check.yml diff --git a/ansible/.github/workflows/validate.yml b/archive/ansible-stale-ci/.github/workflows/validate.yml similarity index 100% rename from ansible/.github/workflows/validate.yml rename to archive/ansible-stale-ci/.github/workflows/validate.yml diff --git a/ansible/.gitlab-ci.yml b/archive/ansible-stale-ci/.gitlab-ci.yml similarity index 100% rename from ansible/.gitlab-ci.yml rename to archive/ansible-stale-ci/.gitlab-ci.yml diff --git a/m5fw-contrib-README.md b/archive/m5fw-contrib-README.md similarity index 100% rename from m5fw-contrib-README.md rename to archive/m5fw-contrib-README.md diff --git a/protocol.py b/archive/protocol.py similarity index 100% rename from protocol.py rename to archive/protocol.py diff --git a/sample-speech-1m.wav b/archive/sample-speech-1m.wav similarity index 100% rename from sample-speech-1m.wav rename to archive/sample-speech-1m.wav