ci: add macOS arm64 runners to the wavekat-ci pool

.github/workflows/preview.yml

@@ -44,7 +44,12 @@ jobs:
         run: |
           OUTPUT=$(npx wrangler pages deploy ./dist --project-name=wavekat-com --branch="${{ github.head_ref || github.ref_name }}" 2>&1)
           echo "$OUTPUT"
-          ALIAS=$(echo "$OUTPUT" | grep -oP '(?<=Deployment alias URL: )https://\S+')
+          # POSIX sed rather than `grep -oP`: portable across the runner
+          # pool, and it does not fail the step under `set -e` when the
+          # deploy output carries no alias URL. See docs/06.
+          ALIAS=$(printf '%s\n' "$OUTPUT" \
+            | sed -nE 's|.*Deployment alias URL: (https://[^[:space:]]+).*|\1|p' \
+            | tail -n 1)
           echo "url=${ALIAS}" >> "$GITHUB_OUTPUT"
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}

CLAUDE.md

@@ -55,6 +55,18 @@ This repo uses **release-please**. Since GitHub squash-merges use the PR title a
 - **Deployment**: Cloudflare Pages (consistent with rest of org)
 - **Domain**: `wavekat.com` — DNS to be pointed at Cloudflare Pages once site is ready
 
+## CI runs on a shared, org-wide, mixed-arch runner pool
+
+All workflows use `runs-on: [self-hosted, wavekat-ci]`. Two things about that label are easy to get wrong:
+
+**It is a pool, not a machine.** A Linux x86-64 workstation and a Mac mini (running the runners as Docker containers, so they report `Linux`/`ARM64`) both carry it, and a job lands on either non-deterministically. Setup lives in `scripts/setup-gha-runners.sh` (Linux, systemd), `scripts/setup-gha-runners-docker.sh` (Linux, containers) and `scripts/setup-gha-runners-macos.sh` (macOS, Docker Desktop); the full story — including why the Mac runs Linux containers rather than a native macOS runner — is `docs/06-self-hosted-runners.md`.
+
+**It is org-wide, and this repo is not its main consumer.** Seven repos ride on `wavekat-ci` (`wavekat.com`, `wavekat-voice`, `wavekat-platform`, `wavekat-asr`, `wavekat-cli`, `wavekat-lab`, `wavekat-platform-client`). Never change what the label points at — or assume a new host is safe — based on this repo's workflows alone; `wavekat-voice` and `wavekat-asr` build sherpa-onnx/ONNX native code and are the arch-sensitive ones. No shipped artifact is built on `wavekat-ci` (installers use GitHub-hosted runners), so the blast radius of a bad host is red CI, not a bad release.
+
+Since every runner is Ubuntu 24.04, GNU shell is fine — but **arch must never be assumed**. Anything that downloads a prebuilt binary or pins a target triple has to resolve arch at runtime (`uname -m`, `dpkg --print-architecture`), and `npm ci` needs both `linux-x64` and `linux-arm64` optional deps in `package-lock.json`, so never regenerate the lockfile with `--no-optional`.
+
+To pin a job to one host, add the runner's automatic arch label: `runs-on: [self-hosted, wavekat-ci, X64]` or `..., ARM64]`.
+
 ## SEO & GEO — every new page must be both
 
 This site is optimized for classic search (SEO) **and** generative answer engines (GEO — being quoted by ChatGPT, Perplexity, Google AI Overviews, Claude). The two overlap but aren't identical: SEO wants crawlable, well-described, linkable pages; GEO wants self-contained, factual, extractable passages an LLM can lift verbatim. Build for both on every page.

docs/06-self-hosted-runners.md

@@ -0,0 +1,188 @@
+# 06 — Self-hosted runners: adding the Mac mini
+
+Every workflow in this repo runs on `runs-on: [self-hosted, wavekat-ci]`. Until
+now that label existed on one machine — a Linux x86-64 workstation. This doc
+adds a Mac mini (Apple Silicon) to the same pool, running the runners as Docker
+Desktop containers rather than natively.
+
+It also records the part that turned out to matter more than the setup itself:
+**`wavekat-ci` is an org-wide label, and seven repos ride on it.** Changing what
+that label points at is not a wavekat.com decision.
+
+## 1. Why containers and not a native macOS runner
+
+The first instinct — install the runner natively with `svc.sh` and launchd — is
+wrong here, for a reason that only shows up when you read the other repos'
+workflows. `wavekat-voice/.github/workflows/ci.yml` does this, three times:
+
+```yaml
+run: sudo apt-get update && sudo apt-get install -y --no-install-recommends libasound2-dev cmake
+```
+
+A native macOS runner has no `apt-get`. Every `wavekat-ci` job in the org's most
+important repo would have failed the moment it landed on the Mac — not subtly,
+immediately. The same jobs also assume GNU userland throughout, and macOS ships
+BSD (`grep -oP`, `sed -i`, `readlink -f`, `date -d`, `sha256sum`, `xargs -r`,
+and bash 3.2 all differ).
+
+So the choice isn't "Docker vs raw shell on macOS". It's:
+
+> Should the Mac mini be an **ARM Linux** CI host, or a **macOS** CI host?
+
+macOS cannot run macOS containers — Docker Desktop runs Linux containers in a
+VM — so containers mean the Mac contributes *Linux/arm64* capacity. That is what
+we want: nothing in the pool's workload needs macOS, and a uniform Ubuntu 24.04
+userland everywhere means a `run:` block can never work on one host and fail on
+the other.
+
+The runner image (`scripts/docker/`) was already arch-portable —
+`dpkg --print-architecture` selects the runner tarball, and the `gh` apt line is
+arch-templated — so it builds natively on Apple Silicon with **no changes and no
+Rosetta**.
+
+## 2. Setting up the Mac mini
+
+Requires Docker Desktop installed and running.
+
+```sh
+brew install gh && gh auth login       # an account with wavekat org admin
+
+git clone git@github.com-wavekat:wavekat/wavekat.com.git
+cd wavekat.com
+./scripts/setup-gha-runners-macos.sh
+```
+
+Tear down with `./scripts/uninstall-gha-runners-macos.sh`. Both take the same
+env vars as the Linux scripts (`RUNNER_ORG`, `RUNNER_COUNT`, `RUNNER_PREFIX`,
+`RUNNER_LABELS`, `RUNNER_IMAGE`, `RUNNER_TOKEN`).
+
+Differences from `setup-gha-runners-docker.sh` (the Linux container script):
+
+- **Supervision is Docker's own `--restart unless-stopped`**, not systemd.
+  Docker Desktop restarts containers with a restart policy when it starts, so
+  there is no launchd plist to write.
+- **Two manual settings, or the runners die at the next reboot:**
+  Docker Desktop → Settings → General → *Start Docker Desktop when you sign in*,
+  and System Settings → Users & Groups → *Automatic login*. Docker Desktop is a
+  GUI app; without a login session there is no daemon and no runners.
+- **The script checks VM memory against `RUNNER_COUNT`** and warns under ~2 GB
+  per runner. Docker Desktop's default allocation is thin for four parallel
+  Rust/Astro builds, and the failure mode is an OOM-killed build rather than a
+  clear error. Raise it in Settings → Resources, or lower `RUNNER_COUNT`.
+- **Keep `_work` on the named volume** (the script does). A host bind mount
+  would put every `npm ci` and `cargo build` through virtiofs — this is the
+  usual reason people conclude Docker on Mac is slow.
+
+Docker Desktop's disk image is also a fixed size. Four runners with warm cargo
+targets, pnpm stores and Playwright browsers will grow into it; if builds start
+failing on "no space left on device", raise the disk limit in Settings →
+Resources before blaming anything else.
+
+## 3. Who else uses `wavekat-ci` — check before you flip the label
+
+The label is org-wide. As of this change, **7 of the org's 27 repos** run jobs
+on it:
+
+| Repo | Workflows on `wavekat-ci` | What those jobs do | Arch-sensitive? |
+|------|---------------------------|--------------------|-----------------|
+| `wavekat.com` | 4 (6 jobs) | Astro build, link/meta checks, CF Pages deploy | No |
+| `wavekat-voice` | 6 of 7 | Rust check/clippy/test, `make sidecar` (sherpa-onnx via CMake), pnpm + Playwright | **Yes** |
+| `wavekat-platform` | 3 | ci, release, db-migrate | Unverified |
+| `wavekat-asr` | 2 of 2 | `cargo test --features sherpa-onnx` | **Yes** |
+| `wavekat-cli` | 3 | release-plz tail + publish (the cross-target build matrix uses GitHub-hosted runners) | No |
+| `wavekat-lab` | 6 of 7 | ci, Common Voice sync/deploy, ONNX publish | Some |
+| `wavekat-platform-client` | 2 | ci, release-plz | No |
+
+Not on the label (they use GitHub-hosted runners): `wavekat-core`,
+`wavekat-vad`, `wavekat-turn`, `wavekat-tts`, `wavekat-flow`, `wavekat-brand`.
+
+One reassurance from that audit: **no shipped artifact is built on `wavekat-ci`.**
+`wavekat-voice/release.yml` builds installers on `macos-latest`,
+`ubuntu-latest`, `windows-latest` and `windows-11-arm`; its only `wavekat-ci`
+job is `trigger-site-rebuild`, which is a `curl -X POST`. Likewise
+`wavekat-cli/release.yml` cross-builds its targets on GitHub-hosted runners. So
+an arm64 runner joining the pool cannot cause a wrong-architecture binary to
+reach users — the blast radius is CI going red, not a bad release.
+
+### The real risk: ONNX on aarch64
+
+`wavekat-asr` runs `cargo test --workspace --features sherpa-onnx`, and
+`wavekat-voice`'s `sidecar` job builds the same stack via CMake.
+`wavekat-voice/release.yml` describes the daemon as linking "a prebuilt
+sherpa-onnx static lib" — and prebuilt native libs are exactly the thing that is
+often published for `x86_64-unknown-linux-gnu` and not for
+`aarch64-unknown-linux-gnu`.
+
+**Verify this before putting `wavekat-ci` on the Mac containers**, because a
+failure here is silent-until-merge: PRs in two repos start failing about half
+the time, on whichever runs happen to land on arm64.
+
+`wavekat-lab/cv-runner-provision.yml` also hardcodes
+`actions-runner-linux-x64-…` in a provisioning script. That one is provisioning
+a *different* machine, so it is probably fine — but it is the same class of
+assumption and worth a read.
+
+### Recommended rollout
+
+Register the Mac with a distinct label first and opt repos in one at a time:
+
+```sh
+RUNNER_LABELS=wavekat-ci-arm64,mac-mini ./scripts/setup-gha-runners-macos.sh
+```
+
+Point one low-risk repo at it (`wavekat.com` is the obvious candidate — pure
+Node, no native deps), let it run for a few days, then either widen the label to
+`wavekat-ci` or keep the split permanently and pin the ONNX repos to x86-64 with
+the runner's automatic `X64` label:
+
+```yaml
+runs-on: [self-hosted, wavekat-ci, X64]
+```
+
+Both hosts carry automatic labels you can pin against:
+
+| Host | Automatic labels | Labels we add |
+|------|------------------|---------------|
+| Linux workstation | `self-hosted`, `Linux`, `X64` | `wavekat-ci`, `<hostname>` |
+| Mac mini (containers) | `self-hosted`, `Linux`, `ARM64` | `wavekat-ci`, `<hostname>` |
+
+Note the Mac's containers report `Linux`, not `macOS` — the runner sees the
+container, not the host.
+
+## 4. What still has to stay portable
+
+Userland is now uniform (Ubuntu 24.04 everywhere), so the BSD-vs-GNU trap list
+no longer gates CI. Two things do:
+
+- **Architecture.** Anything that downloads a prebuilt binary, pins a target
+  triple, or compiles native code must resolve arch at runtime rather than
+  assuming x86-64. `uname -m` / `dpkg --print-architecture`, not a hardcoded
+  `x64` in a URL.
+- **Lockfile optional deps.** `npm ci` needs both `linux-x64` and `linux-arm64`
+  optional packages present. This repo's `package-lock.json` has both for every
+  native dep (esbuild, rolldown, oxide, resvg, sharp, workerd, and
+  `@astrojs/compiler-binding-linux-{x64,arm64}-gnu`). Regenerating the lockfile
+  with `--no-optional`, or on a platform-filtered install, silently drops them
+  and breaks one host while the other stays green.
+
+The `sed` in `preview.yml` (replacing a `grep -oP`) is kept: it is correct on
+GNU too, and it no longer fails the step under `set -e` when no alias URL is
+found. The BSD notes only become load-bearing again if someone adds a *native*
+macOS runner later.
+
+## 5. Operating it
+
+```sh
+docker ps --filter name=gha-runner        # what's up
+docker logs -f gha-runner-1               # live logs
+docker exec -it gha-runner-1 bash         # shell inside a runner
+open https://github.com/organizations/wavekat/settings/actions/runners
+```
+
+Self-hosted runners do **not** get a clean machine per job. The named volume
+persists between runs: `actions/checkout` cleans the repo, but the cargo target
+dir, pnpm store, npm cache and Playwright browsers do not — deliberately, since
+that warmth is why these runners are fast. The sharp edge is that a job failing
+on one host and not the other is usually stale state, not code. The reset is
+`docker rm -f gha-runner-N && docker volume rm gha-runner-N`, then re-run the
+setup script.

scripts/setup-gha-runners-macos.sh

@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#
+# Install N self-hosted GitHub Actions runners on a macOS host (Apple
+# Silicon or Intel) as Docker Desktop containers, and register them with
+# the `wavekat` org.
+#
+# This is the macOS twin of setup-gha-runners-docker.sh. It reuses the
+# SAME image (scripts/docker), which is already arch-portable, and
+# registers with the SAME `wavekat-ci` label — so the Mac joins the
+# existing pool and `runs-on: [self-hosted, wavekat-ci]` jobs land on
+# whichever host is idle. No workflow changes needed.
+#
+# Note what this means: macOS cannot run macOS containers, so Docker
+# Desktop runs these on Linux/arm64 inside its VM. The Mac contributes
+# *Linux* CI capacity. That is the point — nothing this repo builds
+# needs macOS, and a uniform Ubuntu userland everywhere means a `run:`
+# block can never work on one host and fail on the other.
+#
+# Supervision is Docker's own `--restart unless-stopped`, not launchd:
+# Docker Desktop restarts the containers when it starts, so as long as
+# Docker Desktop launches at login the runners come back after a reboot.
+#
+# Usage (run on the Mac, with Docker Desktop running):
+#
+#   # Easiest: let the script fetch a registration token via gh CLI.
+#   #   (`brew install gh && gh auth login` as a wavekat org admin)
+#   ./setup-gha-runners-macos.sh
+#
+#   # Or pass a token explicitly (valid 1h, can register multiple runners):
+#   RUNNER_TOKEN=AAAA... ./setup-gha-runners-macos.sh
+#
+#   # Override defaults:
+#   RUNNER_COUNT=2 RUNNER_PREFIX=mac-mini RUNNER_LABELS=wavekat-ci,mac-mini \
+#     ./setup-gha-runners-macos.sh
+#
+# Re-running is safe: existing containers are torn down, their volumes
+# wiped, and the runners re-registered with a fresh token.
+
+set -euo pipefail
+
+ORG="${RUNNER_ORG:-wavekat}"
+COUNT="${RUNNER_COUNT:-4}"
+PREFIX="${RUNNER_PREFIX:-$(hostname -s)}"
+RUNNER_LABELS="${RUNNER_LABELS:-wavekat-ci,${PREFIX}}"
+IMAGE="${RUNNER_IMAGE:-wavekat/gha-runner:latest}"
+RUNNER_VERSION="${RUNNER_VERSION:-}" # empty = the Dockerfile's default
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DOCKER_CONTEXT="${SCRIPT_DIR}/docker"
+
+log()  { printf '\033[1;36m==>\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m!!\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[1;31mxx\033[0m %s\n' "$*" >&2; exit 1; }
+
+[[ "$(uname -s)" == "Darwin" ]] || die "this script targets macOS (got $(uname -s)) — use setup-gha-runners-docker.sh on Linux"
+[[ -d "${DOCKER_CONTEXT}" ]] || die "missing docker context at ${DOCKER_CONTEXT}"
+
+# 1. Find the Docker CLI. Docker Desktop symlinks into /usr/local/bin,
+#    but a non-login shell may not have its own bin dir on PATH.
+DOCKER=""
+for candidate in docker /usr/local/bin/docker "${HOME}/.docker/bin/docker" /opt/homebrew/bin/docker; do
+  if command -v "${candidate}" >/dev/null 2>&1; then DOCKER="${candidate}"; break; fi
+done
+[[ -n "${DOCKER}" ]] || die "docker CLI not found — install Docker Desktop from https://docker.com/products/docker-desktop"
+
+if ! "${DOCKER}" info >/dev/null 2>&1; then
+  die "Docker Desktop is not running (or the daemon is unreachable). Launch Docker Desktop and re-run."
+fi
+
+# 2. Sanity-check the VM's memory against the runner count. Docker
+#    Desktop's default (often 8 GB) is thin for four parallel Astro
+#    builds; the failure mode is an OOM-killed build, not a clear error.
+MEM_BYTES="$("${DOCKER}" info --format '{{.MemTotal}}' 2>/dev/null || echo 0)"
+if [[ "${MEM_BYTES}" -gt 0 ]]; then
+  MEM_GB=$(( MEM_BYTES / 1024 / 1024 / 1024 ))
+  log "Docker Desktop VM memory: ${MEM_GB} GB for ${COUNT} runner(s)"
+  if [[ $(( MEM_GB / COUNT )) -lt 2 ]]; then
+    warn "under ~2 GB per runner — builds may be OOM-killed."
+    warn "Raise it in Docker Desktop → Settings → Resources → Memory,"
+    warn "or lower RUNNER_COUNT (RUNNER_COUNT=2 ./setup-gha-runners-macos.sh)."
+  fi
+fi
+
+# 3. Build the runner image. The Dockerfile resolves the runner tarball
+#    per-arch (dpkg --print-architecture), so this builds natively on
+#    Apple Silicon with no changes and no Rosetta.
+log "building runner image ${IMAGE} (native $(uname -m))"
+if [[ -n "${RUNNER_VERSION}" ]]; then
+  "${DOCKER}" build --build-arg "RUNNER_VERSION=${RUNNER_VERSION}" -t "${IMAGE}" "${DOCKER_CONTEXT}"
+else
+  "${DOCKER}" build -t "${IMAGE}" "${DOCKER_CONTEXT}"
+fi
+
+# 4. Fetch a registration token (one token can register multiple runners
+#    within its 1h validity window).
+get_token() {
+  if [[ -n "${RUNNER_TOKEN:-}" ]]; then
+    printf '%s' "${RUNNER_TOKEN}"
+    return
+  fi
+  if ! command -v gh >/dev/null 2>&1; then
+    cat >&2 <<'EOF'
+
+No RUNNER_TOKEN set, and `gh` CLI is not installed.
+
+Install gh with Homebrew and re-run, or fetch a token elsewhere:
+
+  brew install gh
+  gh auth login        # use an account with wavekat org admin
+
+  # or, from any machine with gh authed as a wavekat admin:
+  gh api -X POST /orgs/wavekat/actions/runners/registration-token --jq .token
+  RUNNER_TOKEN=<token> ./setup-gha-runners-macos.sh
+EOF
+    exit 1
+  fi
+  gh api -X POST "/orgs/${ORG}/actions/runners/registration-token" --jq .token \
+    || die "failed to fetch registration token (is gh authed as a wavekat admin?)"
+}
+
+TOKEN="$(get_token)"
+[[ -n "${TOKEN}" ]] || die "got empty registration token"
+
+# 5. (Re)create N runners. One container each, with its own named volume
+#    so registration survives restarts and Docker Desktop upgrades.
+#
+#    `--restart unless-stopped` is the whole supervision story on macOS:
+#    there is no systemd, and Docker Desktop restores containers with a
+#    restart policy when it starts. No launchd plist needed.
+for i in $(seq 1 "${COUNT}"); do
+  NAME="${PREFIX}-${i}"
+  CONTAINER="gha-runner-${i}"
+  log "configuring runner ${NAME} (container ${CONTAINER})"
+
+  # Tear down any previous instance and wipe its volume, so the fresh
+  # registration token is applied cleanly instead of the entrypoint
+  # short-circuiting on a stale .runner file.
+  "${DOCKER}" rm -f "${CONTAINER}" >/dev/null 2>&1 || true
+  "${DOCKER}" volume rm "${CONTAINER}" >/dev/null 2>&1 || true
+
+  "${DOCKER}" run -d \
+    --name "${CONTAINER}" \
+    --hostname "${CONTAINER}" \
+    --restart unless-stopped \
+    -v "${CONTAINER}:/home/runner/runner" \
+    -e "RUNNER_ORG=${ORG}" \
+    -e "RUNNER_NAME=${NAME}" \
+    -e "RUNNER_LABELS=${RUNNER_LABELS}" \
+    -e "RUNNER_TOKEN=${TOKEN}" \
+    "${IMAGE}" >/dev/null
+done
+
+log "done — ${COUNT} runner(s) registered to ${ORG} with labels: ${RUNNER_LABELS}"
+echo
+warn "One manual step, or the runners will not survive a reboot:"
+warn "  Docker Desktop → Settings → General → 'Start Docker Desktop when you sign in'"
+warn "  System Settings → Users & Groups → Automatic login → the runner user"
+echo
+log "check status:     ${DOCKER} ps --filter name=gha-runner"
+log "live logs:        ${DOCKER} logs -f gha-runner-1"
+log "container shell:  ${DOCKER} exec -it gha-runner-1 bash"
+log "org view:         https://github.com/organizations/${ORG}/settings/actions/runners"