feat(chopper): run hermes in a microvm

[devusb]

Brings hermes back after #104, this time from the official container image inside a microvm guest rather than as a host-native NixOS service.

Why the image and not the module

Upstream keeps Nix and NixOS at Tier 2 — "maintained in-tree only as a best effort", annotated "frequently breaks" — while the Docker image is Tier 1. Their Nix page says as much directly: "For production stability, use standard installation paths (Docker or FHS environments) instead."

The practical difference is dependencies. Native Nix mode seals everything at build time: no runtime pip install, no runtime npm install, MCP servers resolved when the derivation builds. That is the friction the old hermes.nix was fighting with its programs.nix-ld.libraries block. The image ships Python 3.13 + uv, Node 26 + npx, Playwright, git, ripgrep and ffmpeg, so npx- and uvx-based MCP servers just run.

The microvm supplies the isolation the host-native module would otherwise have provided.

Container mode was considered

The module's container.enable runs the Nix-built binary inside a persistent Ubuntu container, which would give declarative settings and mcpServers alongside an FHS userland — and it keeps its writable layer across restarts, rebuilds and reboots, which is better than what this PR does.

It loses on one hard constraint: backend.mode is the module's only route to hermes serve / hermes dashboard, and the docs state in two places that it "cannot be used with container.enable". No dashboard. It would also still be the Tier 2 Nix build, so it fixes the FHS half of the problem and keeps the packaging half.

Shape

Follows the guest pattern from #118 and #115 — 4 vcpu, 8G, its own tap and vsock CID, 192.168.20.51/23.

Container. oci-containers on podman, image pinned to a release tag rather than latest, since the image has no hermes update and upgrades are a re-pull.

--network=host is load-bearing. Hermes decides whether to engage its dashboard auth gate from its own bind address, and a non-loopback bind refuses to start without an auth provider. A podman port map would still leave it binding 0.0.0.0 inside the container; host networking lets it bind the guest's real loopback, where the gate stays off. Fine here because the guest does one thing.

Worth knowing: oci-containers passes --replace unconditionally, so the container's writable layer is discarded on every restart. Anything the agent apt installs is transient. Upstream's own guidance covers the common cases — npx/uvx re-fetch, and durable tools belong in /opt/data/.local/bin, which the image puts on PATH and which lives on the mounted volume. A derived image is the escape hatch if a specific package ever justifies one.

Access. The guest owns its tailnet identity and publishes the dashboard itself through services.tailscale.serve, so chopper does not proxy for it — no caddy vhost, no authentik forward auth. This is upstream's documented topology, not a workaround: "bind HERMES_DASHBOARD_HOST=127.0.0.1 and reach the dashboard over an SSH tunnel / Tailscale instead."

Keeping it off the public internet is deliberate. Per upstream, an unauthenticated exposed dashboard was the entry point for the June 2026 MCP-config persistence campaign, where scanners drove agents into planting SSH-key backdoors. Tailscale Services also cannot be funnelled today — tailscale serve takes --service, tailscale funnel does not — so the tailnet boundary is enforced rather than merely intended.

A hermes wrapper on the guest's PATH shells into the container, so tailscale SSH followed by hermes … works without remembering the podman invocation.

Naming. The guest is vm-hermes and the service is svc:hermes. Nodes and services share a MagicDNS namespace, and machine names resolves a collision by appending a suffix that persists "even if the conflicting machine's name is later changed" — so a guest called hermes risks permanently becoming hermes-1. The meaningful name belongs to the Service, which is the host-independent identity with its own TailVIP and can be advertised by several hosts. vm- matches the existing tap-interface convention, so the unit, state directory, tap and hostname are all vm-hermes.

Storage. Two volumes beyond the shared /var/lib, split by lifecycle: containers.img for podman's image store, which is disposable and regenerated by a pull, and hermes.img for /opt/data, which holds memory, skills, sessions and MCP tokens and is not. Reclaiming disk and backing up therefore have clean boundaries.

Before this can work

Two tailnet policy entries, since Services are gated on both reachability and advertisement:

{ "src": ["autogroup:member"], "dst": ["svc:hermes"], "ip": ["443"] },
"autoApprovers": { "services": { "svc:hermes": ["tag:server"] } },

The auto-approver keys off tag:server, which common.nix already advertises, so future guests get it for free.

After first boot the gateway crash-loops until hermes setup runs — that is the upstream flow, not a fault. Memory provider selection happens there too; holographic keeps its SQLite store in $HERMES_HOME, so it lands on hermes.img with everything else worth keeping.

Not included

No deploy-backup for /opt/data yet. It is a host service and the guest is a separate system, so wiring it in is its own piece of work — worth doing, since memory and skills are the part that would actually hurt to lose.

No declarative provider credentials. The codex provider authenticates over OAuth that hermes setup persists into /opt/data, so there is nothing to put in an env file yet; a sops secret can be added when there is.

Verification

Guest system and the full chopper closure both build. Generated podman run was checked for the network mode, env, volume and pinned tag; the generated serve config for svc:hermes on tcp:443; the guest hostname resolving to vm-hermes with host units named microvm@vm-hermes; root has no keys and mhelton has two, with %wheel NOPASSWD. Nothing is deployed — everything past the build is unverified until this lands on the host.