diff --git a/README.md b/README.md index e58417e..0caed6b 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,20 @@ flowchart LR CTID block **3000–3999** (declared in [`stack.yaml`](stack.yaml); members inherit its `defaults`). +> **`podman-host` (3006) overrides `node`.** The stack default is `desktop-01`, which is the +> on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)); 3006 is +> pinned to `hpe-01` because it hosts an always-on browser that a scheduled sweep has to reach. + | CTID | Member | Script | Channel | Status | |------|--------|--------|---------|--------| | 3000 | [forgejo](forgejo.lxc.yaml) | `ct/forgejo.sh` | stable | ready to deploy | | 3001 | [forgejo-runner](forgejo-runner.lxc.yaml) | `ct/forgejo-runner.sh` | **dev** (ProxmoxVED) | ready — in-dev script | | 3002 | [github-runner](github-runner.lxc.yaml) | `ct/github-runner.sh` | stable | ready (new, independent of CT 2005) | -| 3003 | woodpecker | — | — | **parked** — no community-script in either repo yet | +| 3003 | [shell](shell.lxc.yaml) | `ct/debian.sh` + `shell` provisioner | stable | deployed | +| 3004 | [cloudflared](cloudflared.lxc.yaml) | `ct/cloudflared.sh` | stable | deployed | +| 3005 | [newt](newt.lxc.yaml) | `ct/debian.sh` + `newt` provisioner | stable | deployed | +| 3006 | [podman-host](podman-host.lxc.yaml) | `ct/podman.sh` | stable | **new** ([#447](https://github.com/Chrison-Homelab/Homelab/issues/447)) — pinned to `hpe-01`, see [its README](podman-host/README.md) | +| — | woodpecker | — | — | **parked** — no community-script in either repo yet | ## Deploying diff --git a/podman-host.lxc.yaml b/podman-host.lxc.yaml new file mode 100644 index 0000000..5691062 --- /dev/null +++ b/podman-host.lxc.yaml @@ -0,0 +1,128 @@ +# DevOps podman host — the stack's rootless Podman + quadlet host (ADR-0009, #447). +# +# DELIBERATELY GENERIC, exactly like stacks/SmartHome/podman-host.lxc.yaml. This is +# *the DevOps stack's container host*, not "the browser box". The marketplace-tools +# Chromium is simply the first service to land here — chosen as the pilot because it +# is self-contained (one container, no NFS, all state in one volume) and nothing else +# depends on it, so the fiddly rootless plumbing proves itself where the blast radius +# is smallest. Further DevOps tooling joins as extra quadlets in podman-host/quadlets/ +# rather than as new one-app CTs. +# +# WHY A NEW CT AND NOT A QUADLET ON AN EXISTING HOST — #447 suggested landing this on +# the Media podman host (CT 5114). That option IS available (Homelab.Stacks.Media +# declares podman-host.lxc.yaml + podman-host/; it just looks absent locally when the +# submodule is not initialised), so this is a deliberate choice against it, not an +# absence of alternatives. +# +# The ADR-0009 rule quoted in SmartHome's host — "further SmartHome services join as +# extra quadlets rather than as new one-app Docker CTs" — is scoped to services joining +# THEIR OWN stack's host. It argues against one-app CTs inside a stack that already has +# a podman host; it does not argue for parking one stack's tooling on another stack's +# host. DevOps has no podman host, so creating one is the same act SmartHome (6004), +# Monitoring (4001) and Media (5114) each already performed — not a violation of it. +# The alternative homes were rejected on their merits: +# · Media 5114 — cross-stack. A browser holding a live Facebook session would share a +# host with Plex and both *arr fleets, coupling the lifecycle of a scraper to the +# media stack; a restart to fix a wedged Chrome would touch unrelated services. +# · Monitoring 4001 — stack.yaml says it is "deliberately NOT co-located with any +# other stack: this host watches everything else, so it needs its own lifecycle". +# It also holds Pulse's admin creds, API token and authentik client secret, and a +# browser carrying a live Facebook session does not belong in that blast radius +# in either direction. +# · SmartHome 6004 — sits on IoT VLAN 1040, firewalled off from 1010. +# So this establishes the DevOps stack's own podman host, the same role 4001 and 6004 +# play for their stacks. Net CT count is unchanged versus the alternative of a +# one-app CT, and the next piece of DevOps tooling costs zero CTs. +# +# ⚠ NODE OVERRIDE IS LOAD-BEARING. The DevOps stack default is `desktop-01`, which is +# the designated on-demand SLEEP node (#191) — a browser hosted there would be asleep +# precisely when a scheduled sweep wanted it. Pinned to hpe-01, which is an always-on +# sentinel with the measured headroom for it (see #447: 9.1Gi available, load 0.47). +# Longer term the natural home is node #4 once #334 unblocks; migrating means changing +# this one line. +# +# WHAT THE ENGINE DOES HERE (app: podman → PodmanProvisioner, superproject #284): +# ct/podman.sh creates the CT (rootful podman + a root podman.socket), then the +# provisioner converts it to the rootless quadlet model — masks that socket, adds +# the non-root `podman` user with an in-map subuid range + linger, makes +# network-online.target reachable, renders the quadlets below and drives +# `systemctl --user`. Read docs/plans/284-podman-platform.md before touching a +# quadlet: the eleven gotchas there are all load-bearing. +# +# EXPOSURE — INTERNAL ONLY, AND THIS ONE MATTERS MORE THAN USUAL. No tunnel ingress, +# no Pangolin route, no wildcard DNS. The container holds a logged-in Facebook +# session, and the Chrome DevTools protocol has NO AUTHENTICATION WHATSOEVER: anything +# that can reach its port gets full control of that session — read messages, post +# listings, change the account. The quadlet therefore publishes CDP on the CT's +# LOOPBACK only and is reached by SSHing INTO THIS CT and tunnelling. See +# podman-host/README.md for the exact invocation and why the obvious +# `ssh -L … root@hpe-01` is the wrong one. +# +# Validate against ../../Infrastructure/schema/shape.schema.json. + +apiVersion: homelab/v1 +kind: LXC + +metadata: + name: podman-host + stack: DevOps + tags: [podman, quadlet, container-host, adr-0009, browser-automation] + +spec: + app: podman # community-scripts ct/podman.sh + the `podman` provisioner + ctid: 3006 # in-block (3000–3999); 3000 forgejo · 3001/3002 runners · + # 3003 shell · 3004 cloudflared · 3005 newt + node: hpe-01 # OVERRIDES the DevOps default of desktop-01 — see the node + # note above; desktop-01 sleeps (#191), this must not. + cores: 2 # Chrome is the only real consumer; 2 is enough for a scraper + # driving one or two tabs, and leaves hpe-01's 4 cores alone. + memory: 4096 # MB — Chrome alone sits at 1–1.5 GB with a few tabs, plus the + # KasmVNC server, plus podman itself. 2048 was the first draft + # and is too tight once /dev/shm is actually used (see the + # quadlet's ShmSize): shm is tmpfs and counts against this. + disk: 12 # GB — the linuxserver image is ~2 GB of layers, the rest is + # the Chrome profile volume (cache grows; it is disposable). + + # DHCP reservation (#416) so the SSH target for the tunnel is stable. The DNS record + # rides the reservation, so name and lease move together and nothing points at a + # bare address again. + network: + reservation: + fixedIp: 10.10.204.36 + name: devops podman-host (CT 3006) + + features: + nesting: true # rootless podman needs nesting for its own nested userns + keyctl: true # containers keyring — settable only via the pct/SSH create path + fuse: true # fuse-overlayfs fallback storage driver + # NOTE: ct/podman.sh silently DROPS var_fuse; the provisioner + # reconciles features from this shape after create (#284). + + # node is set above; network/os/source/timezone otherwise inherited from stack.yaml + # (VLAN 1010, debian 13, unprivileged, onboot). + + config: + # Rootless user + subuid window. The default 10000:50000 fits inside an + # unprivileged LXC's own 65536-uid map — the provisioner verifies this against + # /proc/self/uid_map and fails loudly rather than letting podman break at first run. + user: podman + quadlets: podman-host/quadlets + + # Rootless API socket for local inspection (`podman ps` over the user socket). + userSocket: true + + # ⚠ COCKPIT DELIBERATELY OFF, unlike SmartHome's host. Cockpit would put a + # privileged web console on the same CT as a browser holding live account + # credentials, on a VLAN the workstation can already reach. The management value + # is small here — one container — and `pct exec` plus the user socket cover it. + # cockpit: false (omitted; the default) + + # podman secrets seeded from the gitignored secrets.env, add-only (rotation is an + # explicit operator action, never a silent converge side effect). + # + # These gate the KasmVNC web UI, which is the one interactive surface: it exists so + # the Facebook login can be done by hand once. Without auth, anyone on VLAN 1010 + # could open that UI and drive the logged-in session. Set both in secrets.env. + secrets: + chromium_kasm_user: CHROMIUM_KASM_USER + chromium_kasm_password: CHROMIUM_KASM_PASSWORD diff --git a/podman-host/README.md b/podman-host/README.md new file mode 100644 index 0000000..fe9fe06 --- /dev/null +++ b/podman-host/README.md @@ -0,0 +1,115 @@ +# DevOps podman host (CT 3006) + +The DevOps stack's rootless Podman + quadlet host, per [ADR-0009](../../../docs/adr/ADR-0009-container-runtime.md) +and [#447](https://github.com/Chrison-Homelab/Homelab/issues/447). Pinned to **hpe-01** — +the stack default `desktop-01` is the on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)), +which would be asleep exactly when a scheduled sweep wanted the browser. + +| Quadlet | What it is | Ports | +|---|---|---| +| `chromium` | Persistent Chromium with a CDP endpoint, for `~/marketplace-tools` | CDP `127.0.0.1:9222` (CT loopback) · KasmVNC `:3000` / `:3001` | + +Further DevOps tooling joins as **extra quadlets in `quadlets/`**, not as new one-app CTs. +There is deliberately no `.network` unit: a single container needs no by-name DNS, and one +gets added the moment a second container has to resolve the first (see the Monitoring +host's `monitoring.network` for why it becomes mandatory then). + +## Deploy + +```bash +set -a && . ./secrets.env && set +a # CHROMIUM_KASM_USER + CHROMIUM_KASM_PASSWORD +./build.sh Preview --stack DevOps # dry-run +./build.sh Deploy --stack DevOps # live apply +``` + +Config is **not** baked into the guest — the quadlets are rendered onto the host and folded +into the managed marker, so editing one re-converges and restarts the unit that consumes it. + +## One-off: log Chromium into Facebook + +The KasmVNC UI exists for exactly this. It is the only interactive surface. + +1. Open **`http://:3000`** (or `https://:3001`, self-signed) and authenticate + with `CHROMIUM_KASM_USER` / `CHROMIUM_KASM_PASSWORD` from `secrets.env`. + The shape reserves `10.10.204.36`, but a freshly-created CT keeps its original lease + until it renews — check the current address with `pct exec 3006 -- hostname -I`. +2. Log into Facebook by hand, and complete any 2FA challenge. +3. That's it — the session persists in the `/config` volume across restarts. + +> ⚠ **Do not copy `~/marketplace-tools/profile/` into the volume.** Moving a session onto a +> new device fingerprint is very likely to trip Facebook's new-device check, and can cost a +> checkpoint or a temporary lockout on the account. Logging in fresh takes two minutes. + +Trade Me needs no login for scraping. (Note that this also means the container cannot add +things to a Trade Me *watchlist* — that has bitten us before, when clicks were reported as +successful against a profile that was only ever logged into Facebook.) + +## Using it from the workstation + +`~/marketplace-tools/lib.js` honours **`CDP_URL`** (default `http://127.0.0.1:9222`), so every +script — `scan.js`, `watch.js`, `detail.js`, `search.js`, `trademe.js` — picks this up with no +code change. + +```bash +node ~/marketplace-tools/cdp-tunnel.js & # 127.0.0.1:9222 -> CT 3006 +CDP_URL=http://127.0.0.1:9222 node ~/marketplace-tools/scan.js +``` + +### ⚠ `ssh -L` does not work here, and cannot + +Three facts rule it out, and each one on its own is enough: + +1. **Chrome refuses to bind DevTools off localhost.** Modern Chrome (verified on 151) ignores + `--remote-debugging-address` — the flag appears on the cmdline and Chrome binds + `127.0.0.1:9222` anyway. So CDP lives on the *container's* loopback and cannot be published, + proxied or bound to the LAN even deliberately. This is why the quadlet uses `Network=host`: + it makes Chrome's own localhost bind land on the CT's loopback. +2. **This CT has no sshd.** It is a rootless-podman host driven entirely over `pct exec`, so + `ssh -L 9222:127.0.0.1:9222 root@` has nothing to connect to. `authorizedKeys` is a + `shell`-provisioner feature, not part of the `podman` app. +3. **`ssh -L … root@hpe-01` forwards to the NODE's loopback**, where nothing is listening. + +So `cdp-tunnel.js` listens on the workstation's loopback and, per connection, splices stdio +through `ssh pct exec 3006 -- nc 127.0.0.1 9222`. Nothing new listens in the CT, the LAN +never sees 9222, and no credential is added anywhere. Knobs: `NODE_HOST`, `CTID`, `PORT`. + +> If `cdp-tunnel.js` reports `EADDRINUSE`, a local Chrome from `launch.js` still holds 9222. +> Close it, or use `PORT=9333` and point `CDP_URL` at the same port. + +## Why loopback-only is not paranoia + +The DevTools protocol has **no authentication of any kind** — no token, no password, no +ACL. Anything that can open a TCP connection to 9222 gets full control of a browser holding +a live Facebook session: reading DMs, posting and editing listings, changing account +settings. Port 9222 is equivalent to the account password, and should be treated that way. + +The KasmVNC UI is the deliberate exception — it must be reachable to be useful — which is +why `CUSTOM_USER`/`PASSWORD` are non-optional. That basic auth is the only thing between +VLAN 1010 and the session. + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| `connectOverCDP` hangs or refuses | `cdp-tunnel.js` not running, or you tried `ssh -L` — see above for why that cannot work. | +| **403 on `/json/version`**, everything else healthy | Chrome rejects DevTools requests whose `Host` header is not `localhost`/a bare IP (DNS-rebinding defence). `cdp-tunnel.js` presents `localhost:9222`, which is fine — but a reverse proxy on a hostname would be refused. | +| Tabs crash, blank screenshots, "Target closed" mid-navigation | `/dev/shm` too small. Reads like a scraper bug; it isn't. `ShmSize=1g` in the quadlet. | +| Container exits immediately at startup | Chrome's sandbox vs. the nested userns — `PodmanArgs=--security-opt seccomp=unconfined` is required. Last resort `CHROME_CLI=--no-sandbox`, which disables the renderer sandbox. | +| **`Unit chromium.service not found`** | Quadlet rejected the file over ONE unsupported key and generated nothing. `SecurityOpt=` is not a quadlet key (use `PodmanArgs=--security-opt …`). Diagnose in one second with `/usr/libexec/podman/quadlet -dryrun -user` as the `podman` user. | +| CDP silent but container healthy, Chrome flags correct, curl works *inside* the container | Chrome bound its own loopback and the publish forwarded to an interface nothing listens on. Needs `Network=host`. | +| Unit absent after a CT reboot | Missing `[Install] WantedBy=default.target` (gotcha 6). Note also that quadlets do not start for ~92s after boot because `network-online.target` is never reached — gotcha 7, handled by the provisioner. | +| Trade Me returns `ERR_BLOCKED_BY_RESPONSE` | Rate limiting, not a container fault. The scripts back off 30/60/90s; pace multi-page runs with `DELAY=6000`. | +| Selectors break, pages look mobile | Viewport too narrow — both sites are responsive and switch DOM. `CUSTOM_WIDTH/HEIGHT` are set to 1920×1080 for this reason. | + +## What this unblocks + +A sweep that runs **on the homelab** rather than the laptop, which is the actual point: + +```bash +node ~/marketplace-tools/scan.js nodes >> /var/log/marketplace-scan.log +``` + +Driven by a systemd timer on the node instead of a human remembering. The "check back +before the auction closes" case has been impossible until now: a cloud agent cannot reach +the laptop, and Trade Me is a client-rendered SPA, so plain HTTP fetches return no live bid +data. diff --git a/podman-host/quadlets/chromium.container b/podman-host/quadlets/chromium.container new file mode 100644 index 0000000..41d00dc --- /dev/null +++ b/podman-host/quadlets/chromium.container @@ -0,0 +1,142 @@ +# Chromium for marketplace-tools (#447) — a persistent, always-on browser with a CDP +# endpoint, so the Facebook Marketplace / Trade Me scraping in ~/marketplace-tools stops +# depending on a headed Chrome window on the MacBook (which dies whenever the window is +# closed, and can never run unattended). +# +# HEADED CHROMIUM UNDER A VIRTUAL DISPLAY, NOT TRUE HEADLESS — this is a deliberate +# choice, not laziness. Facebook and Trade Me both fingerprint for headless, and Trade Me +# is ALREADY rate-limiting us (ERR_BLOCKED_BY_RESPONSE, handled with 30/60/90s backoff in +# trademe.js and detail.js). linuxserver/chromium runs a real Chromium under KasmVNC, +# which also gives us the one thing a headless container cannot: a way to do the +# interactive Facebook login by hand, once. +# +# ⚠ SECURITY — READ BEFORE CHANGING THE NETWORKING. +# The Chrome DevTools protocol has NO AUTHENTICATION OF ANY KIND. There is no token, no +# password, no ACL. Anything that can open a TCP connection to the CDP port gets full +# control of this browser, and this browser holds a logged-in Facebook session: reading +# DMs, posting and editing listings, changing account settings. Treat port 9222 as +# equivalent to the account password. +# · CDP is on the CT's LOOPBACK only, because Chrome itself refuses to bind DevTools +# anywhere else (see the Network=host block). Reached by an SSH tunnel INTO THIS CT +# — see ../README.md, and note the obvious node-level tunnel is the wrong one. +# · The KasmVNC UI is on the CT's 3000/3001 and IS reachable on VLAN 1010, because it +# has to be for the one-time Facebook login. It is gated by CUSTOM_USER/PASSWORD, +# and that basic auth is the only thing between the VLAN and the session. +# · No Pangolin route, no wildcard DNS, no tunnel ingress. Ever. + +[Unit] +Description=Chromium (marketplace-tools CDP browser) +After=network-online.target +Wants=network-online.target + +[Container] +Image=lscr.io/linuxserver/chromium:latest +ContainerName=chromium +AutoUpdate=registry + +# ── Networking: HOST, and this is load-bearing ──────────────────────────────────────── +# ⚠ DO NOT "FIX" THIS BACK TO PublishPort. That was the first design and it does not work, +# for a reason that is invisible from the outside: +# +# MODERN CHROME IGNORES --remote-debugging-address. Verified on Chrome 151 in this very +# image: given `--remote-debugging-port=9222 --remote-debugging-address=0.0.0.0`, the +# flags appear on the process cmdline and in CHROME_CLI, and Chrome then binds +# 127.0.0.1:9222 ANYWAY — DevTools off-localhost was locked down years ago. `ss` inside +# the container shows 127.0.0.1:9222, not 0.0.0.0:9222. +# +# With a bridge network, `PublishPort=127.0.0.1:9222:9222` therefore forwards to the +# container's EXTERNAL interface, where nothing listens. The failure is maximally +# confusing: the CT shows a healthy listener on 127.0.0.1:9222 (pasta), the container is +# up, Chrome has the right flags, curl INSIDE the container returns /json/version +# perfectly — and the tunnel still gets an empty reply. +# +# Host networking makes Chrome's own localhost bind land directly on the CT's loopback, +# which is exactly the topology we wanted. It is also a STRONGER guarantee than the +# publish ever was: CDP is now loopback-only because CHROME binds it that way and refuses +# to do otherwise, not because of an argument someone could delete. +# +# Consequence: published ports do not apply under host networking, so KasmVNC is on the +# CT's OWN 3000/3001 (not the 3010/3011 an earlier draft mapped). It is still reachable on +# VLAN 1010 and still gated by CUSTOM_USER/PASSWORD — that basic auth remains the only +# thing between the VLAN and the session. +Network=host + +# ── Chrome's shared memory ──────────────────────────────────────────────────────────── +# Chrome hammers /dev/shm and the container default (64 MB) does NOT produce a clean +# error — it produces tab crashes, blank screenshots and "Target closed" mid-navigation, +# which read like scraper bugs. Same reason authentik's quadlets set 512m. 1g is ample +# for a couple of tabs. This is tmpfs, so it counts against the CT's 4096 MB. +ShmSize=1g + +# ── Chrome's sandbox vs. nested userns ──────────────────────────────────────────────── +# Chrome's own sandbox needs syscalls that podman's default seccomp profile blocks, and +# we are already three namespaces deep (LXC → rootless podman → container). Without this +# Chromium exits immediately at startup. This is what linuxserver document for the image. +# If it STILL refuses to start, the next step is CHROME_CLI=--no-sandbox — but that +# disables Chrome's renderer sandbox entirely, so try it only to confirm the diagnosis +# and prefer keeping the sandbox on. +# ⚠ THIS MUST GO THROUGH PodmanArgs. `SecurityOpt=` is NOT a quadlet key — quadlet has +# SeccompProfile= (which takes a PATH, so it cannot express "unconfined"), and an +# unsupported key makes the generator reject THE WHOLE FILE, producing no unit at all. +# The failure mode is silent and misleading: `systemctl --user start chromium.service` +# reports "Unit chromium.service not found", which reads as a rendering or linger +# problem rather than a syntax error. Verify with: +# /usr/libexec/podman/quadlet -dryrun -user +PodmanArgs=--security-opt seccomp=unconfined + +# ── Chrome flags ────────────────────────────────────────────────────────────────────── +# --remote-debugging-address=0.0.0.0 IS KEPT DELIBERATELY, even though Chrome 151 ignores +# it (see the Network=host block above for the evidence). It costs nothing, it documents +# the intent, and if a future Chrome ever honours it again the setup still works. What we +# rely on is Chrome's localhost bind landing on the CT loopback via host networking. +# +# ⚠ AND CHROME VALIDATES THE HOST HEADER. It rejects DevTools requests whose Host is not +# localhost or a bare IP — a defence against DNS rebinding. This is why the access path +# MUST be an SSH tunnel presenting `localhost:9222` and not, say, a reverse proxy on a +# hostname: Playwright's connectOverCDP fetches /json/version first, and that fetch is +# what would get refused. A 403 on /json/version with everything else looking healthy is +# this, not a networking fault. +# +# ⚠ QUOTED, AND IT MUST BE — same trap as pulse.container's OIDC_SCOPES. systemd's +# `Environment=` splits an UNQUOTED value on whitespace, so writing this bare would set +# CHROME_CLI=--remote-debugging-port=9222 and discard the rest as a malformed second +# assignment. Verified in the generated ExecStart: it must appear as ONE quoted value. +Environment="CHROME_CLI=--remote-debugging-port=9222 --remote-debugging-address=0.0.0.0" + +# ── KasmVNC auth ────────────────────────────────────────────────────────────────────── +# Seeded from secrets.env via the shape's `secrets:` map — see ../../podman-host.lxc.yaml. +Secret=chromium_kasm_user,type=env,target=CUSTOM_USER +Secret=chromium_kasm_password,type=env,target=PASSWORD + +Environment=PUID=1000 +Environment=PGID=1000 +Environment=TZ=Pacific/Auckland +# Window size the scrapers were written against — Marketplace and Trade Me are both +# responsive, and a narrow viewport switches them to mobile layouts with different DOM, +# which breaks the selectors rather than merely looking odd. +Environment=CUSTOM_WIDTH=1920 +Environment=CUSTOM_HEIGHT=1080 + +# ── State ───────────────────────────────────────────────────────────────────────────── +# The Chrome profile lives under /config, so this volume is what makes the Facebook +# session survive a restart — the equivalent of marketplace-tools/profile/ today. +# +# ⚠ DO NOT COPY THE LAPTOP'S profile/ DIRECTORY IN. Moving a session to a new device +# fingerprint is very likely to trip Facebook's new-device check and can cost a +# checkpoint/lockout on the account. Log in fresh through the KasmVNC UI instead — it is +# a two-minute job, once. +Volume=/home/podman/devops/data/chromium:/config + +[Service] +# Gotcha 9 in docs/plans/284-podman-platform.md: podman does NOT create a missing +# bind-mount source, it fails the unit. Create it before the container starts. +ExecStartPre=/usr/bin/mkdir -p /home/podman/devops/data/chromium +Restart=always +RestartSec=10 +# Chrome under KasmVNC is slow to first paint on a cold image pull. +TimeoutStartSec=300 + +[Install] +# Gotcha 6: without this the unit does not come back after a CT reboot, and the failure +# is silent — the container is simply absent until someone starts it by hand. +WantedBy=default.target