docs: showcase README; move root clutter to archive/

BACKLOG.md

@@ -4,9 +4,14 @@
 
 Swept from every `.md` file and all code/config in the repo. The sections it replaced
 have been collapsed to pointers: `README.md`, `docs/OVERVIEW.md`, `docs/services.md`,
-and the dated `TODO-2026-07-14.md` / `TODO-2026-07-23.md` / `TODO-2026-08-03.md` files,
+and the dated `archive/TODO-2026-07-14.md` / `archive/TODO-2026-07-23.md` / `archive/TODO-2026-08-03.md` files,
 which are now historical session notes — keep them for the reasoning, not the tasks.
 
+**Moved 2026-10-03.** The dated TODO files now live in `archive/`, unedited, so every
+`file:line` citation below still resolves at its new path. `README.md` was restructured
+the same day, so `README.md:<line>` citations in this file refer to commit `6cb781f`,
+the last commit before that change.
+
 The reason for consolidating: those six lists disagreed. Offsite backup was marked
 **DONE** in three of them while the unit had never copied a byte, the Vault root token
 was listed as a required credential in three more after being deliberately eliminated,
@@ -324,7 +329,7 @@ Note the doc's list still says "Vault root token" — as of 08-07 there delibera
 isn't one. See §6.3.
 
 ### 1.3 ~~Finish the offsite tier~~ — **DONE, verified running 08-15**
-`TODO-2026-08-03.md:274`, `ansible/playbooks/backup-offsite.yml:59`
+`archive/TODO-2026-08-03.md:274`, `ansible/playbooks/backup-offsite.yml:59`
 
 **SEED COMPLETE 2026-08-11 02:19 UTC.** ~3.1 days wall clock, 44m35s CPU, ~204 GiB.
 Retention applied on the first run, keeping 32 snapshots across five groups
@@ -894,7 +899,7 @@ So the chain from the envelope is: unseal with item 5 → root shell on the rpi5
 config → generate-root → full admin. It works. It just needs host root as well as the
 envelope, which in a rebuild you would have anyway.
 
-**And it was documented twice.** `TODO-2026-08-03.md:727-814` carries a section titled
+**And it was documented twice.** `archive/TODO-2026-08-03.md:727-814` carries a section titled
 *"Why `generate-root` was blocked — and the actual fix"*: the identical 403 with no token,
 the `enable_unauthenticated_access` remedy, and at `:760` the full list of families —
 **`"rekey"`, `"generate-root"`, `"generate-operation-token"`**. The `rekey` family is
@@ -1771,7 +1776,7 @@ The playbook was deleted rather than fixed; see §9. It preseeded slapd and boun
 header said "never hardcode it". It had also been dead since 2026-07-18.
 
 ### 2.5 ~~Vault unseal keys sit beside the sealed data~~ — **claim corrected 08-15; storage kept, deliberately**
-`ansible/templates/vault-unseal.sh.j2:6`, `TODO-2026-08-03.md:871-876`
+`ansible/templates/vault-unseal.sh.j2:6`, `archive/TODO-2026-08-03.md:871-876`
 
 `docs/SECURITY.md` described the shares as "Offline, physically secure". They are on the
 RPi5 at `0400`, next to the raft store they unseal, so root there is equivalent to full
@@ -1991,7 +1996,7 @@ for. All three were written from reading the repo. All three corrections came fr
 measuring the running system.
 
 ### 2.8 Optional: tighten `admin` so it cannot self-escalate
-`ansible/files/vault-policies/admin.hcl`, `TODO-2026-08-03.md:310`
+`ansible/files/vault-policies/admin.hcl`, `archive/TODO-2026-08-03.md:310`
 
 `admin` holds `create` plus `sudo` on `sys/policies/acl/*` and `auth/*`, so a token
 carrying it can write a policy granting anything and mint a token for it — root at
@@ -3257,7 +3262,7 @@ package manager said, and warns by name if a host process holds 9100 on a cluste
 cannot reach the LAN.
 
 ### 3.7 Alloy restart recency filter
-`TODO-2026-08-03.md:335` — benign historical restarts keep surfacing.
+`archive/TODO-2026-08-03.md:335` — benign historical restarts keep surfacing.
 
 ### 3.8 `PolicyViolation` events fire continuously against dead ReplicaSets
 
@@ -6019,17 +6024,20 @@ them.
 - [ ] **`Makefile:2` still lists `ldap` in `.PHONY`** though the target was removed on
   08-09 and the file carries a comment explaining the removal. Harmless; finish the job.
 
-- [ ] **`FETCH_HEAD` is tracked in git** — a git internal committed by accident.
-- [ ] **Duplicated, drifted CI config under `ansible/`** — `ansible/.gitlab-ci.yml`,
+- [x] **`FETCH_HEAD` is tracked in git** — a git internal committed by accident.
+  Removed 2026-10-03.
+- [x] **Duplicated, drifted CI config under `ansible/`** — `ansible/.gitlab-ci.yml`,
   `ansible/.github/workflows/*`, `ansible/.claude/settings.json`. Only the root copies
-  are read; these are dead weight diverging silently.
+  are read; these are dead weight diverging silently. Moved to
+  `archive/ansible-stale-ci/` 2026-10-03.
 - [ ] **Mixed ArgoCD `repoURL` schemes** — 12 files use `git@github.com:`, but
   `gitops/apps/gitlab-runner.yaml:18,22` uses `https://`. Argo treats these as two
   repos needing two credentials.
 - [ ] **`.gitignore` path bug in `ansible/`** (C7) — and note this working copy lives in
   `Downloads`, so `LAB-URLS.md` and similar travel with any zip or sync of the folder.
-- [ ] **Archive `TODO-2026-07-14.md`** (25/26 done) and collapse `README.md`'s TODO to a
-  pointer at this file.
+- [x] **Archive `TODO-2026-07-14.md`** (25/26 done) and collapse `README.md`'s TODO to a
+  pointer at this file. Done 2026-10-03: all three dated TODO files moved to `archive/`
+  unedited, and the README section is now a pointer.
 
 ### 7.y This file's own status was wrong — **SWEPT 2026-09-20**
 
@@ -6455,7 +6463,7 @@ work, invisible for months, in a section titled *small, live, cheap*.
   state (`backup-etcd.sh.j2:14` mentions the `state.db` rename) and delete.
 
 - [ ] `lldap-backup` init container failed once and self-healed; a retry loop around
-  `pg_dump` would make it deterministic (`TODO-2026-08-03.md:447-452`).
+  `pg_dump` would make it deterministic (`archive/TODO-2026-08-03.md:447-452`).
 - [x] **Re-sweep Ansible for `no_log` registers consumed later** — done 08-09.
   All 12 files carrying `no_log` were checked. Results, so this need not be redone:
 

README.md

@@ -1,10 +1,16 @@
-# Home Lab — DevOps Environment
+# HomeLab — GitOps platform on a 14-host x86 + ARM64 fleet
 
-A GitOps-managed home lab built around an **Odroid-H4 Ultra** NAS core and a
-**3-node HA k3s cluster** (H4 + two N150 mini PCs), with an ARM64 inference fleet
-(two Orange Pi 5 Pros). Infrastructure is defined as code: **Ansible** stands hosts up,
-**Argo CD** runs everything inside the cluster from this git repo, and a two-tier storage
-model (hot NVMe + two cold SATA RAID 1 mirrors) keeps data safe.
+A production-style platform run at home: a **3-node HA k3s cluster**, everything in it
+reconciled by **Argo CD** from this repo, hosts built by **Ansible** and **OpenTofu**,
+secrets from **HashiCorp Vault**, policy enforced by **Kyverno**, and a full
+**Prometheus / Grafana / Loki** observability stack. An ARM64 tier serves local AI
+inference on NPUs behind an OpenAI-compatible gateway. Backups go to two RAID 1 mirrors
+and offsite to Cloudflare R2, and **restores are drilled, not assumed**.
+
+**Cloud counterpart:** [HomeLab-aws](https://github.com/swares/HomeLab-aws) builds an
+EKS cluster from nothing, runs a slice of the same stack (Argo CD, Kyverno, LiteLLM) with
+IRSA and an ALB, and destroys it every night for about $1 a session. It is deliberately
+detachable: nothing in this repo depends on it.
 
 ```mermaid
 flowchart TB
@@ -36,10 +42,25 @@ flowchart TB
     class BK c;
 ```
 
+## What this demonstrates
+
+| Skill | Where to look |
+|---|---|
+| **GitOps end to end.** Every change is a PR. Argo CD app-of-apps plus a git-directory ApplicationSet, `selfHeal` and `prune` on, rollback is `git revert` | `gitops/`, `docs/UPDATES.md` |
+| **CI guardrails before merge.** YAML lint, kubeconform schema validation, ApplicationSet collision checks, and OPA/conftest policies run on every PR | `.github/workflows/validate.yml`, `ci/policies/` |
+| **Policy as code in the cluster.** Kyverno in Enforce mode: no `:latest`, no privileged pods, resource limits required | `gitops/workloads/kyverno/` |
+| **Secrets management.** Vault (KV v2, policies in git, no standing root token) feeding External Secrets Operator; Ansible Vault for host secrets | `ansible/files/vault-policies/`, `docs/SECURITY.md` |
+| **Identity.** Authelia OIDC single sign-on backed by lldap; cert-manager private CA for TLS | `docs/SSO.md` |
+| **Observability.** kube-prometheus-stack, Loki, Alloy on every node, blackbox probes, Alertmanager | `gitops/apps/monitoring.yaml`, `gitops/workloads/monitoring/` |
+| **Infrastructure as code.** Ansible for 14 hosts across x86 and ARM64, OpenTofu with remote state, Packer images, nightly drift checks, Renovate dependency PRs | `ansible/`, `tofu/`, `packer/`, `renovate.json` |
+| **Backup and disaster recovery.** restic to two local mirrors and offsite R2, etcd and Vault snapshots, and restore drills that actually restored data | `docs/BACKUP-RESTORE.md` |
+| **Incident response.** A written incident record with root cause, fix, and proof by unattended reboot | `docs/INCIDENT-2026-08-23-h4-boot.md` |
+| **AI platform engineering.** LiteLLM gateway routing to Ollama, NPU-native RKLLama, and Whisper, plus an adapter that makes edge devices an inference backend | `docs/AI-INFERENCE.md`, `docs/AI-ROUTING.md` |
+
 ## Why it's shaped this way
 
 - **k3s, not full Kubernetes** — the H4 is also the NAS. k3s runs as a single systemd
-  service alongside Samba/NFS and leaves most of the box free. Traefik is the default
+  service alongside the NFS server and leaves most of the box free. Traefik is the default
   ingress; workloads use `networking.k8s.io/v1 Ingress`, not OpenShift Routes.
 - **Argo CD, not imperative ops** — change the cluster by editing git and opening PRs.
   Argo reconciles with `selfHeal` on, so drift reverts and rollback is `git revert`.
@@ -56,6 +77,9 @@ flowchart TB
 | `gitops/` | What Argo deploys — `bootstrap/` (app-of-apps), `apps/`, `workloads/` |
 | `docs/` | Architecture, hardware, runbook, security, AI inference, service catalog, updates |
 | `scripts/` | One-shot helpers (`enable-winrm.ps1`, `lab-check.sh`, flannel FDB service) |
+| `ci/` | OPA/conftest policies run by CI against every workload manifest |
+| `tofu/`, `packer/` | OpenTofu (VMs, DNS) and Packer image builds |
+| `archive/` | Superseded files kept for reference; nothing here is live |
 | `CLAUDE.md` | Operating rules — read before touching anything |
 
 ## What's running
@@ -81,54 +105,25 @@ flowchart TB
 | cert-manager | `cert-manager` | TLS via `lab-ca` ClusterIssuer (self-signed root CA) |
 | Argo CD | `argocd` | GitOps controller — selfHeal + prune on all apps |
 
-### Host services (outside k3s)
-
-| Service | Host | IP |
-|---------|------|----|
-| HashiCorp Vault | rpi5 | 192.168.1.128 |
-| Mosquitto MQTT (primary) | opi-zero2w-2 | 192.168.1.188 |
-| Mosquitto MQTT (secondary, HA bridge) | opi-zero2w-4 | 192.168.1.99 |
-| Pi-hole DNS (primary, v6.4.3) | octopi (RPi 3B #2) | 192.168.1.148 |
-| Pi-hole DNS (secondary, v6) | rpi4b (RPi 4B) | 192.168.1.116 |
-| dnsmasq DNS (tertiary fallback) | opi-zero2w-1 | 192.168.1.184 |
-| Samba / NFS (NAS) | H4 (host) | 192.168.1.160 |
-| GitLab CE | gitlab-1 VM on n150-1 | 192.168.1.50 |
-
-### Ingress endpoints
-
-| URL | Service |
-|-----|---------|
-| `argocd.apps.lab.home.arpa` | Argo CD |
-| `grafana.apps.lab.home.arpa` | Grafana |
-| `immich.apps.lab.home.arpa` | Immich |
-| `ai.apps.lab.home.arpa` | LiteLLM gateway |
-| `ha.apps.lab.home.arpa` | Home Assistant |
-| `authelia.apps.lab.home.arpa` | Authelia (SSO) |
-| `lldap.apps.lab.home.arpa` | lldap (LDAP directory UI) |
-| `semaphore.apps.lab.home.arpa` | Semaphore (Ansible UI) |
-| `minio.apps.lab.home.arpa` | Minio (S3 API) |
-| `minio-console.apps.lab.home.arpa` | Minio console |
-| `stt.apps.lab.home.arpa` | Whisper STT |
-| `*.apps.lab.home.arpa` | → 192.168.1.201 (kube-vip service VIP for Traefik) |
-
-## Fleet
-
-| Host | Ansible name | IP | Role |
-|------|--------------|----|------|
-| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS (smbd/NFS) |
-| N150 mini PC #1 | n150-1 | 192.168.1.42 | k3s server + KVM hypervisor (Ubuntu 24.04) |
-| N150 mini PC #2 | n150-2 | 192.168.1.21 | k3s server + KVM hypervisor (Ubuntu 24.04) |
-| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, RKLLama/Ollama inference, NPU |
-| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, RKLLama/Ollama inference, NPU |
-| Raspberry Pi 5 | rpi5 | 192.168.1.128 | HashiCorp Vault |
-| Raspberry Pi 4B | rpi4b | 192.168.1.116 | Pi-hole secondary DNS (v6, Bookworm) |
-| RPi 3B #2 (octopi) | octopi | 192.168.1.148 | Pi-hole primary DNS (v6.4.3, Bookworm) |
-| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed) |
-| OPi Zero 2W #1 | opi-zero2w-1 | 192.168.1.184 | dnsmasq DNS tertiary fallback |
-| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT primary broker |
-| OPi Zero 2W #3 | opi-zero2w-3 | 192.168.1.217 | dnsmasq — configured, but NOT in `lab_dns_servers` (see docs/HARDWARE.md) |
-| OPi Zero 2W #4 | opi-zero2w-4 | 192.168.1.99 | MQTT secondary broker (HA bridge) |
-| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent |
+### Host services, endpoints and fleet
+
+Outside the cluster, the fleet runs HashiCorp Vault (Raspberry Pi 5), redundant Pi-hole
+DNS with a dnsmasq fallback, an HA Mosquitto MQTT pair, the NFS server on the H4, and GitLab
+CE in a KVM VM. Every service is exposed at `*.apps.lab.home.arpa` through a kube-vip
+VIP in front of Traefik.
+
+| Host class | Count | Role |
+|---|---|---|
+| Odroid-H4 Ultra (x86) | 1 | k3s server + NAS |
+| N150 mini PC (x86) | 3 | two k3s servers + KVM hypervisors, one Windows host (WinRM) |
+| Orange Pi 5 Pro (ARM64, RK3588 NPU) | 2 | k3s agents, LLM inference |
+| Raspberry Pi 5 / 4B / 3B | 3 | Vault, primary and secondary DNS |
+| Orange Pi Zero 2W | 4 | MQTT pair, fallback DNS |
+| Odroid XU3 | 1 | Legacy board on Ubuntu 16.04; retire or rebuild pending (`BACKLOG.md` §2.16) |
+
+Full detail lives in one place each, so it can't drift between copies:
+[docs/HARDWARE.md](docs/HARDWARE.md) owns hosts and IPs,
+[docs/services.md](docs/services.md) owns the service catalog and endpoints.
 
 ## Quickstart (fresh bootstrap)
 
@@ -182,38 +177,28 @@ See [docs/SECURITY.md](docs/SECURITY.md) for the full security model.
 
 - **Never** `mkfs`/`wipefs` the cold disks (`/dev/md0`, `/dev/md1`)
 - **Never** run `restic forget`/`prune` by hand — retention is handled by backup timers only
-- **Never** stop `smbd`, `nfs`, `backup-nas`, or `backup-etcd`
+- **Never** stop `nfs-server`, `backup-nas`, or `backup-etcd`. There is no Samba on the H4; NFS is the only export path
 - Before any hot-tier storage change: confirm last backup succeeded
 
-## TODO
+## Open work
 
 **All open work lives in [`BACKLOG.md`](BACKLOG.md).** It is the single list, swept
 from every document and from the code, and ordered by what happens if an item is
 ignored.
 
-This section used to carry its own list. It drifted: it claimed offsite restic backup
-was done (it had never copied a byte), and it duplicated items that also appeared in
-`docs/OVERVIEW.md`, `docs/services.md`, `docs/STANDUP.md` and three dated `TODO-*.md`
-files, each with a different idea of what was outstanding. One list or none.
-
-### Done ✅
-- [x] octopi flashed to Bookworm, Pi-hole v6.4.3 running (2026-07-13)
-- [x] RPi 4B: Pi-hole v6 secondary DNS live at 192.168.1.116 (2026-07-02)
-- [x] n150-1/n150-2 joined as k3s server nodes, kube-vip VIP 192.168.1.200 (2026-07-02)
-- [x] lldap migrated from ldap-1 VM to k3s Deployment in `lldap` namespace (2026-07-04)
-- [x] Authelia → PostgreSQL backend (2026-07-03)
-- [x] Immich library → NFS ReadWriteMany PV (schedules on any node) (2026-07-04)
-- [x] Shared NFS storage between n150-1/n150-2 for VM live migration (2026-07-03)
-- [x] Monitoring stack migrated to n150-1 (2026-07-04)
-- [x] zswap on n150-1/n150-2 (zstd, zsmalloc, 20%) (2026-07-03)
-- [x] MQTT HA: opi-zero2w-4 secondary broker with bidirectional bridge (2026-07-10)
-- [x] Kyverno 3 ClusterPolicies in Enforce mode (2026-07-14)
-- [x] ArgoCD notifications + git-directory ApplicationSet (2026-07-14)
-- [x] Semaphore Ansible UI live (2026-07-18)
-- [x] OpenTofu state in Minio; gitlab-1 VM codified (2026-07-18)
-- [x] Offsite restic replication to Cloudflare R2 (`homelab-nas`) — wired 2026-08-07.
-      Note: the `backup-offsite.timer` line previously here was **false**. The unit
-      existed but `offsite_restic_repo` was never set, so it exited 0 nightly and
-      reported PASSED without copying anything. See `BACKLOG.md` §1.3.
-- [x] restic repository password rotated on both local repos (2026-08-07)
-- [x] Vault: no standing root token; policies under git (2026-08-07)
+It exists because the lists drifted. An earlier TODO here claimed offsite restic backup
+was done when it had never copied a byte: the unit existed, but `offsite_restic_repo` was
+never set, so it exited 0 nightly and reported PASSED. It was fixed and verified on
+2026-08-07 (`BACKLOG.md` §1.3). A green status over unfinished work is the failure this
+repo now checks for.
+
+The dated `TODO-2026-*.md` files that preceded `BACKLOG.md` are kept, unedited, in
+[`archive/`](archive/) because `BACKLOG.md` cites them by line number.
+
+## Related repositories
+
+| Repo | What it is |
+|---|---|
+| [HomeLab-aws](https://github.com/swares/HomeLab-aws) | Ephemeral EKS sandbox: OpenTofu, IRSA, ALB controller, ordered nightly teardown |
+| [My_M5Stack_Core_Framework](https://github.com/swares/My_M5Stack_Core_Framework) | Edge sensor and inference firmware; its adapter runs in this cluster |
+| [HostMon](https://github.com/swares/HostMon) | ESP32-S3 network-monitoring appliance that alerts into the edge tier |

archive/README.md

@@ -0,0 +1,17 @@
+# archive/
+
+Superseded files kept for reference. **Nothing here is live.** No playbook, workflow,
+or Argo Application reads from this folder.
+
+Moved here on 2026-10-03 to clear the repo root. The files themselves are unedited.
+
+| File | What it was | Why it's kept |
+|---|---|---|
+| `TODO-2026-07-14.md`, `TODO-2026-07-23.md`, `TODO-2026-08-03.md` | Dated session task lists, replaced by [`../BACKLOG.md`](../BACKLOG.md) on 2026-08-07 | `BACKLOG.md` cites them by `file:line` as evidence. Moving a file keeps its line numbers, so those citations still hold at the new path. **Do not edit these files**, or the citations break |
+| `m5fw-contrib-README.md` | Build and apply procedure for contributing the adapter code to the M5Stack framework repo | Historical; the live adapter docs are `gitops/workloads/ai-gateway/m5stack-adapter/README.md` |
+| `protocol.py` | An older copy of the M5Stack device-protocol client | The canonical client lives in the framework repo, and the deployed copy is `gitops/workloads/ai-gateway/m5stack-adapter/protocol.py` |
+| `sample-speech-1m.wav` | A one-minute speech sample, apparently for Whisper STT testing | Kept in case a test still expects it (5 MB) |
+| `ansible-stale-ci/` | `ansible/.github/`, `ansible/.gitlab-ci.yml`, `ansible/.claude/` | Drifted duplicates of the root copies. Neither GitHub nor GitLab ever read them from that location (`BACKLOG.md` §7.1) |
+
+The empty, accidentally committed `FETCH_HEAD` was deleted rather than archived. It was
+a git internal file, not content.