From 9d35b1ef93905185285c72aec7b5c19d838f06ad Mon Sep 17 00:00:00 2001 From: Chrison Simtian Date: Thu, 27 Aug 2026 20:30:38 +1200 Subject: [PATCH 1/2] feat(podman-host): DevOps podman host + Chromium quadlet for marketplace-tools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #447 (superproject). Establishes the DevOps stack's rootless podman + quadlet host (CT 3006) and lands a persistent Chromium with a CDP endpoint as its first quadlet, so the marketplace-tools scraping stops depending on a headed Chrome window on the MacBook — which dies whenever the window is closed and can never run unattended. Placement: #447 suggested a quadlet on the Media host (5114). That is available, so this is a deliberate choice against it. ADR-0009's "join as extra quadlets" rule is scoped to services joining their OWN stack's host; it argues against one-app CTs where a stack already has a podman host, not for parking one stack's tooling on another's. DevOps had no podman host, so creating one is the same act SmartHome (6004), Monitoring (4001) and Media (5114) each performed. Media was rejected as cross-stack coupling (a wedged Chrome restart would touch Plex and the *arr fleets); Monitoring because it deliberately keeps its own lifecycle and holds Pulse's admin creds and authentik client secret; SmartHome for being on IoT VLAN 1040. Node is pinned to hpe-01, overriding the DevOps default of desktop-01 — that is the on-demand sleep node (#191) and would be asleep exactly when a scheduled sweep needed the browser. Security posture is the reason for most of the quadlet's shape. The DevTools protocol has no authentication of any kind, and this browser holds a live Facebook session, so CDP is published on the CT's loopback only and reached by an SSH tunnel terminating inside the CT. The README spells out why the obvious `ssh -L … root@hpe-01` is the wrong tunnel, and why publishing on the CT's LAN address to make it "just work" must not be done. Cockpit is deliberately off here, unlike SmartHome's host. The KasmVNC UI is the one interactive surface — needed for the one-time Facebook login — so it is gated by CUSTOM_USER/PASSWORD from secrets.env. Gotchas encoded from docs/plans/284-podman-platform.md and hard-won elsewhere: - CHROME_CLI is QUOTED. systemd's Environment= splits unquoted values on whitespace, which would silently drop --remote-debugging-address and leave CDP bound to the container's own loopback where podman cannot forward to it — failing as a connection error that looks like a network fault. Same trap as pulse.container's OIDC_SCOPES. - ShmSize=1g. Chrome's default 64MB /dev/shm does not error cleanly; it produces tab crashes and "Target closed" mid-navigation that read as scraper bugs. - SecurityOpt=seccomp=unconfined, or Chromium exits at startup three namespaces deep (LXC -> rootless podman -> container). - ExecStartPre mkdir (gotcha 9: podman fails rather than creating a missing bind-mount source) and [Install] WantedBy=default.target (gotcha 6: silently absent after a CT reboot without it). - Chrome validates the DevTools Host header, so the access path must present localhost:9222 — a reverse proxy on a hostname would get a 403 on /json/version with everything else looking healthy. - Do NOT migrate the laptop's profile/ dir; a new device fingerprint risks a Facebook checkpoint. Log in fresh through KasmVNC. Also fills in the README members table, which had drifted: 3003/3004/3005 exist as shapes but were unlisted, and 3003 was still shown as parked woodpecker. Co-Authored-By: Claude Opus 5 --- README.md | 10 +- podman-host.lxc.yaml | 128 ++++++++++++++++++++++++ podman-host/README.md | 108 ++++++++++++++++++++ podman-host/quadlets/chromium.container | 120 ++++++++++++++++++++++ 4 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 podman-host.lxc.yaml create mode 100644 podman-host/README.md create mode 100644 podman-host/quadlets/chromium.container diff --git a/README.md b/README.md index e58417e..0caed6b 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,20 @@ flowchart LR CTID block **3000–3999** (declared in [`stack.yaml`](stack.yaml); members inherit its `defaults`). +> **`podman-host` (3006) overrides `node`.** The stack default is `desktop-01`, which is the +> on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)); 3006 is +> pinned to `hpe-01` because it hosts an always-on browser that a scheduled sweep has to reach. + | CTID | Member | Script | Channel | Status | |------|--------|--------|---------|--------| | 3000 | [forgejo](forgejo.lxc.yaml) | `ct/forgejo.sh` | stable | ready to deploy | | 3001 | [forgejo-runner](forgejo-runner.lxc.yaml) | `ct/forgejo-runner.sh` | **dev** (ProxmoxVED) | ready — in-dev script | | 3002 | [github-runner](github-runner.lxc.yaml) | `ct/github-runner.sh` | stable | ready (new, independent of CT 2005) | -| 3003 | woodpecker | — | — | **parked** — no community-script in either repo yet | +| 3003 | [shell](shell.lxc.yaml) | `ct/debian.sh` + `shell` provisioner | stable | deployed | +| 3004 | [cloudflared](cloudflared.lxc.yaml) | `ct/cloudflared.sh` | stable | deployed | +| 3005 | [newt](newt.lxc.yaml) | `ct/debian.sh` + `newt` provisioner | stable | deployed | +| 3006 | [podman-host](podman-host.lxc.yaml) | `ct/podman.sh` | stable | **new** ([#447](https://github.com/Chrison-Homelab/Homelab/issues/447)) — pinned to `hpe-01`, see [its README](podman-host/README.md) | +| — | woodpecker | — | — | **parked** — no community-script in either repo yet | ## Deploying diff --git a/podman-host.lxc.yaml b/podman-host.lxc.yaml new file mode 100644 index 0000000..5691062 --- /dev/null +++ b/podman-host.lxc.yaml @@ -0,0 +1,128 @@ +# DevOps podman host — the stack's rootless Podman + quadlet host (ADR-0009, #447). +# +# DELIBERATELY GENERIC, exactly like stacks/SmartHome/podman-host.lxc.yaml. This is +# *the DevOps stack's container host*, not "the browser box". The marketplace-tools +# Chromium is simply the first service to land here — chosen as the pilot because it +# is self-contained (one container, no NFS, all state in one volume) and nothing else +# depends on it, so the fiddly rootless plumbing proves itself where the blast radius +# is smallest. Further DevOps tooling joins as extra quadlets in podman-host/quadlets/ +# rather than as new one-app CTs. +# +# WHY A NEW CT AND NOT A QUADLET ON AN EXISTING HOST — #447 suggested landing this on +# the Media podman host (CT 5114). That option IS available (Homelab.Stacks.Media +# declares podman-host.lxc.yaml + podman-host/; it just looks absent locally when the +# submodule is not initialised), so this is a deliberate choice against it, not an +# absence of alternatives. +# +# The ADR-0009 rule quoted in SmartHome's host — "further SmartHome services join as +# extra quadlets rather than as new one-app Docker CTs" — is scoped to services joining +# THEIR OWN stack's host. It argues against one-app CTs inside a stack that already has +# a podman host; it does not argue for parking one stack's tooling on another stack's +# host. DevOps has no podman host, so creating one is the same act SmartHome (6004), +# Monitoring (4001) and Media (5114) each already performed — not a violation of it. +# The alternative homes were rejected on their merits: +# · Media 5114 — cross-stack. A browser holding a live Facebook session would share a +# host with Plex and both *arr fleets, coupling the lifecycle of a scraper to the +# media stack; a restart to fix a wedged Chrome would touch unrelated services. +# · Monitoring 4001 — stack.yaml says it is "deliberately NOT co-located with any +# other stack: this host watches everything else, so it needs its own lifecycle". +# It also holds Pulse's admin creds, API token and authentik client secret, and a +# browser carrying a live Facebook session does not belong in that blast radius +# in either direction. +# · SmartHome 6004 — sits on IoT VLAN 1040, firewalled off from 1010. +# So this establishes the DevOps stack's own podman host, the same role 4001 and 6004 +# play for their stacks. Net CT count is unchanged versus the alternative of a +# one-app CT, and the next piece of DevOps tooling costs zero CTs. +# +# ⚠ NODE OVERRIDE IS LOAD-BEARING. The DevOps stack default is `desktop-01`, which is +# the designated on-demand SLEEP node (#191) — a browser hosted there would be asleep +# precisely when a scheduled sweep wanted it. Pinned to hpe-01, which is an always-on +# sentinel with the measured headroom for it (see #447: 9.1Gi available, load 0.47). +# Longer term the natural home is node #4 once #334 unblocks; migrating means changing +# this one line. +# +# WHAT THE ENGINE DOES HERE (app: podman → PodmanProvisioner, superproject #284): +# ct/podman.sh creates the CT (rootful podman + a root podman.socket), then the +# provisioner converts it to the rootless quadlet model — masks that socket, adds +# the non-root `podman` user with an in-map subuid range + linger, makes +# network-online.target reachable, renders the quadlets below and drives +# `systemctl --user`. Read docs/plans/284-podman-platform.md before touching a +# quadlet: the eleven gotchas there are all load-bearing. +# +# EXPOSURE — INTERNAL ONLY, AND THIS ONE MATTERS MORE THAN USUAL. No tunnel ingress, +# no Pangolin route, no wildcard DNS. The container holds a logged-in Facebook +# session, and the Chrome DevTools protocol has NO AUTHENTICATION WHATSOEVER: anything +# that can reach its port gets full control of that session — read messages, post +# listings, change the account. The quadlet therefore publishes CDP on the CT's +# LOOPBACK only and is reached by SSHing INTO THIS CT and tunnelling. See +# podman-host/README.md for the exact invocation and why the obvious +# `ssh -L … root@hpe-01` is the wrong one. +# +# Validate against ../../Infrastructure/schema/shape.schema.json. + +apiVersion: homelab/v1 +kind: LXC + +metadata: + name: podman-host + stack: DevOps + tags: [podman, quadlet, container-host, adr-0009, browser-automation] + +spec: + app: podman # community-scripts ct/podman.sh + the `podman` provisioner + ctid: 3006 # in-block (3000–3999); 3000 forgejo · 3001/3002 runners · + # 3003 shell · 3004 cloudflared · 3005 newt + node: hpe-01 # OVERRIDES the DevOps default of desktop-01 — see the node + # note above; desktop-01 sleeps (#191), this must not. + cores: 2 # Chrome is the only real consumer; 2 is enough for a scraper + # driving one or two tabs, and leaves hpe-01's 4 cores alone. + memory: 4096 # MB — Chrome alone sits at 1–1.5 GB with a few tabs, plus the + # KasmVNC server, plus podman itself. 2048 was the first draft + # and is too tight once /dev/shm is actually used (see the + # quadlet's ShmSize): shm is tmpfs and counts against this. + disk: 12 # GB — the linuxserver image is ~2 GB of layers, the rest is + # the Chrome profile volume (cache grows; it is disposable). + + # DHCP reservation (#416) so the SSH target for the tunnel is stable. The DNS record + # rides the reservation, so name and lease move together and nothing points at a + # bare address again. + network: + reservation: + fixedIp: 10.10.204.36 + name: devops podman-host (CT 3006) + + features: + nesting: true # rootless podman needs nesting for its own nested userns + keyctl: true # containers keyring — settable only via the pct/SSH create path + fuse: true # fuse-overlayfs fallback storage driver + # NOTE: ct/podman.sh silently DROPS var_fuse; the provisioner + # reconciles features from this shape after create (#284). + + # node is set above; network/os/source/timezone otherwise inherited from stack.yaml + # (VLAN 1010, debian 13, unprivileged, onboot). + + config: + # Rootless user + subuid window. The default 10000:50000 fits inside an + # unprivileged LXC's own 65536-uid map — the provisioner verifies this against + # /proc/self/uid_map and fails loudly rather than letting podman break at first run. + user: podman + quadlets: podman-host/quadlets + + # Rootless API socket for local inspection (`podman ps` over the user socket). + userSocket: true + + # ⚠ COCKPIT DELIBERATELY OFF, unlike SmartHome's host. Cockpit would put a + # privileged web console on the same CT as a browser holding live account + # credentials, on a VLAN the workstation can already reach. The management value + # is small here — one container — and `pct exec` plus the user socket cover it. + # cockpit: false (omitted; the default) + + # podman secrets seeded from the gitignored secrets.env, add-only (rotation is an + # explicit operator action, never a silent converge side effect). + # + # These gate the KasmVNC web UI, which is the one interactive surface: it exists so + # the Facebook login can be done by hand once. Without auth, anyone on VLAN 1010 + # could open that UI and drive the logged-in session. Set both in secrets.env. + secrets: + chromium_kasm_user: CHROMIUM_KASM_USER + chromium_kasm_password: CHROMIUM_KASM_PASSWORD diff --git a/podman-host/README.md b/podman-host/README.md new file mode 100644 index 0000000..b8d4256 --- /dev/null +++ b/podman-host/README.md @@ -0,0 +1,108 @@ +# DevOps podman host (CT 3006) + +The DevOps stack's rootless Podman + quadlet host, per [ADR-0009](../../../docs/adr/ADR-0009-container-runtime.md) +and [#447](https://github.com/Chrison-Homelab/Homelab/issues/447). Pinned to **hpe-01** — +the stack default `desktop-01` is the on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)), +which would be asleep exactly when a scheduled sweep wanted the browser. + +| Quadlet | What it is | Ports | +|---|---|---| +| `chromium` | Persistent Chromium with a CDP endpoint, for `~/marketplace-tools` | CDP `127.0.0.1:9222` · KasmVNC `:3010` / `:3011` | + +Further DevOps tooling joins as **extra quadlets in `quadlets/`**, not as new one-app CTs. +There is deliberately no `.network` unit: a single container needs no by-name DNS, and one +gets added the moment a second container has to resolve the first (see the Monitoring +host's `monitoring.network` for why it becomes mandatory then). + +## Deploy + +```bash +set -a && . ./secrets.env && set +a # CHROMIUM_KASM_USER + CHROMIUM_KASM_PASSWORD +./build.sh Preview --stack DevOps # dry-run +./build.sh Deploy --stack DevOps # live apply +``` + +Config is **not** baked into the guest — the quadlets are rendered onto the host and folded +into the managed marker, so editing one re-converges and restarts the unit that consumes it. + +## One-off: log Chromium into Facebook + +The KasmVNC UI exists for exactly this. It is the only interactive surface. + +1. Open `http://devops-podman-host.homelab.chrison.internal:3010` (or `https://…:3011`, + self-signed) and authenticate with `CHROMIUM_KASM_USER` / `CHROMIUM_KASM_PASSWORD`. +2. Log into Facebook by hand, and complete any 2FA challenge. +3. That's it — the session persists in the `/config` volume across restarts. + +> ⚠ **Do not copy `~/marketplace-tools/profile/` into the volume.** Moving a session onto a +> new device fingerprint is very likely to trip Facebook's new-device check, and can cost a +> checkpoint or a temporary lockout on the account. Logging in fresh takes two minutes. + +Trade Me needs no login for scraping. (Note that this also means the container cannot add +things to a Trade Me *watchlist* — that has bitten us before, when clicks were reported as +successful against a profile that was only ever logged into Facebook.) + +## Using it from the workstation + +`~/marketplace-tools/lib.js` honours **`CDP_URL`** (default `http://127.0.0.1:9222`), so +every script — `scan.js`, `watch.js`, `detail.js`, `search.js`, `trademe.js` — picks this up +with no further change. + +```bash +ssh -N -L 9222:127.0.0.1:9222 root@devops-podman-host.homelab.chrison.internal & +CDP_URL=http://127.0.0.1:9222 node ~/marketplace-tools/scan.js +``` + +### ⚠ The obvious tunnel is the wrong one + +```bash +# WRONG — forwards to hpe-01's OWN loopback, where nothing is listening. +ssh -N -L 9222:127.0.0.1:9222 root@hpe-01.homelab.chrison.internal +``` + +CDP is published on **CT 3006's** loopback, not the node's. The tunnel must terminate +*inside the CT*, so SSH to the CT itself. If the CT is not directly reachable, jump: + +```bash +ssh -N -J root@hpe-01.homelab.chrison.internal \ + -L 9222:127.0.0.1:9222 root@10.10.204.36 & +``` + +Rewriting the quadlet to publish on the CT's LAN address so that the node-level tunnel +"just works" would expose an unauthenticated CDP port to everything on VLAN 1010. Don't. + +## Why loopback-only is not paranoia + +The DevTools protocol has **no authentication of any kind** — no token, no password, no +ACL. Anything that can open a TCP connection to 9222 gets full control of a browser holding +a live Facebook session: reading DMs, posting and editing listings, changing account +settings. Port 9222 is equivalent to the account password, and should be treated that way. + +The KasmVNC UI is the deliberate exception — it must be reachable to be useful — which is +why `CUSTOM_USER`/`PASSWORD` are non-optional. That basic auth is the only thing between +VLAN 1010 and the session. + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| `connectOverCDP` hangs or refuses | Tunnel terminates on the node, not the CT — see above. | +| **403 on `/json/version`**, everything else healthy | Chrome rejects DevTools requests whose `Host` header is not `localhost`/a bare IP (DNS-rebinding defence). The access path must be an SSH tunnel presenting `localhost:9222`, never a reverse proxy on a hostname. | +| Tabs crash, blank screenshots, "Target closed" mid-navigation | `/dev/shm` too small. Reads like a scraper bug; it isn't. `ShmSize=1g` in the quadlet. | +| Container exits immediately at startup | Chrome's sandbox vs. the nested userns — `SecurityOpt=seccomp=unconfined` is required. Last resort `CHROME_CLI=--no-sandbox`, which disables the renderer sandbox. | +| Unit absent after a CT reboot | Missing `[Install] WantedBy=default.target` (gotcha 6). Note also that quadlets do not start for ~92s after boot because `network-online.target` is never reached — gotcha 7, handled by the provisioner. | +| Trade Me returns `ERR_BLOCKED_BY_RESPONSE` | Rate limiting, not a container fault. The scripts back off 30/60/90s; pace multi-page runs with `DELAY=6000`. | +| Selectors break, pages look mobile | Viewport too narrow — both sites are responsive and switch DOM. `CUSTOM_WIDTH/HEIGHT` are set to 1920×1080 for this reason. | + +## What this unblocks + +A sweep that runs **on the homelab** rather than the laptop, which is the actual point: + +```bash +node ~/marketplace-tools/scan.js nodes >> /var/log/marketplace-scan.log +``` + +Driven by a systemd timer on the node instead of a human remembering. The "check back +before the auction closes" case has been impossible until now: a cloud agent cannot reach +the laptop, and Trade Me is a client-rendered SPA, so plain HTTP fetches return no live bid +data. diff --git a/podman-host/quadlets/chromium.container b/podman-host/quadlets/chromium.container new file mode 100644 index 0000000..c57c9b0 --- /dev/null +++ b/podman-host/quadlets/chromium.container @@ -0,0 +1,120 @@ +# Chromium for marketplace-tools (#447) — a persistent, always-on browser with a CDP +# endpoint, so the Facebook Marketplace / Trade Me scraping in ~/marketplace-tools stops +# depending on a headed Chrome window on the MacBook (which dies whenever the window is +# closed, and can never run unattended). +# +# HEADED CHROMIUM UNDER A VIRTUAL DISPLAY, NOT TRUE HEADLESS — this is a deliberate +# choice, not laziness. Facebook and Trade Me both fingerprint for headless, and Trade Me +# is ALREADY rate-limiting us (ERR_BLOCKED_BY_RESPONSE, handled with 30/60/90s backoff in +# trademe.js and detail.js). linuxserver/chromium runs a real Chromium under KasmVNC, +# which also gives us the one thing a headless container cannot: a way to do the +# interactive Facebook login by hand, once. +# +# ⚠ SECURITY — READ BEFORE CHANGING ANY PublishPort LINE. +# The Chrome DevTools protocol has NO AUTHENTICATION OF ANY KIND. There is no token, no +# password, no ACL. Anything that can open a TCP connection to the CDP port gets full +# control of this browser, and this browser holds a logged-in Facebook session: reading +# DMs, posting and editing listings, changing account settings. Treat port 9222 as +# equivalent to the account password. +# · CDP is published on 127.0.0.1 ONLY — the CT's loopback, reachable by nothing on +# the network. It is used via an SSH tunnel INTO THIS CT (see ../README.md). +# · The KasmVNC UI is published on all interfaces because it must be reachable +# interactively, and is therefore gated by CUSTOM_USER/PASSWORD below. That basic +# auth is the only thing standing between VLAN 1010 and the session. +# · No Pangolin route, no wildcard DNS, no tunnel ingress. Ever. + +[Unit] +Description=Chromium (marketplace-tools CDP browser) +After=network-online.target +Wants=network-online.target + +[Container] +Image=lscr.io/linuxserver/chromium:latest +ContainerName=chromium +AutoUpdate=registry + +# ── Ports ───────────────────────────────────────────────────────────────────────────── +# CDP, LOOPBACK-ONLY. The bind address here is the whole security control; dropping the +# `127.0.0.1:` prefix would silently expose the session to the entire VLAN. +PublishPort=127.0.0.1:9222:9222 +# KasmVNC web UI — the interactive surface for the one-time Facebook login. HTTP on 3000, +# HTTPS (self-signed) on 3001. Gated by CUSTOM_USER/PASSWORD. +PublishPort=3010:3000 +PublishPort=3011:3001 + +# ── Chrome's shared memory ──────────────────────────────────────────────────────────── +# Chrome hammers /dev/shm and the container default (64 MB) does NOT produce a clean +# error — it produces tab crashes, blank screenshots and "Target closed" mid-navigation, +# which read like scraper bugs. Same reason authentik's quadlets set 512m. 1g is ample +# for a couple of tabs. This is tmpfs, so it counts against the CT's 4096 MB. +ShmSize=1g + +# ── Chrome's sandbox vs. nested userns ──────────────────────────────────────────────── +# Chrome's own sandbox needs syscalls that podman's default seccomp profile blocks, and +# we are already three namespaces deep (LXC → rootless podman → container). Without this +# Chromium exits immediately at startup. This is what linuxserver document for the image. +# If it STILL refuses to start, the next step is CHROME_CLI=--no-sandbox — but that +# disables Chrome's renderer sandbox entirely, so try it only to confirm the diagnosis +# and prefer keeping the sandbox on. +SecurityOpt=seccomp=unconfined + +# ── Chrome flags ────────────────────────────────────────────────────────────────────── +# ⚠ THE ADDRESS PAIR LOOKS CONTRADICTORY AND IS NOT. Chrome binds remote debugging to +# the loopback of ITS OWN network namespace by default, which podman cannot forward to. +# So Chrome listens on 0.0.0.0 *inside the container namespace*, and the loopback +# restriction is applied one layer out by PublishPort above. Both halves are required: +# drop the flag and the tunnel connects to nothing; drop the PublishPort bind address +# and 0.0.0.0 becomes literal. +# +# ⚠ AND CHROME VALIDATES THE HOST HEADER. It rejects DevTools requests whose Host is not +# localhost or a bare IP — a defence against DNS rebinding. This is why the access path +# MUST be an SSH tunnel presenting `localhost:9222` and not, say, a reverse proxy on a +# hostname: Playwright's connectOverCDP fetches /json/version first, and that fetch is +# what would get refused. A 403 on /json/version with everything else looking healthy is +# this, not a networking fault. +# ⚠ QUOTED, AND IT MUST BE — same trap as pulse.container's OIDC_SCOPES. systemd's +# `Environment=` splits an UNQUOTED value on whitespace, so writing this bare would set +# CHROME_CLI=--remote-debugging-port=9222 and then discard +# `--remote-debugging-address=0.0.0.0` as a malformed second assignment. The result is the +# nastiest possible outcome: Chrome starts, CDP listens on the container's own loopback, +# podman forwards to nothing, and the tunnel fails with a connection error that looks like +# a networking problem rather than a missing flag. +Environment="CHROME_CLI=--remote-debugging-port=9222 --remote-debugging-address=0.0.0.0" + +# ── KasmVNC auth ────────────────────────────────────────────────────────────────────── +# Seeded from secrets.env via the shape's `secrets:` map — see ../../podman-host.lxc.yaml. +Secret=chromium_kasm_user,type=env,target=CUSTOM_USER +Secret=chromium_kasm_password,type=env,target=PASSWORD + +Environment=PUID=1000 +Environment=PGID=1000 +Environment=TZ=Pacific/Auckland +# Window size the scrapers were written against — Marketplace and Trade Me are both +# responsive, and a narrow viewport switches them to mobile layouts with different DOM, +# which breaks the selectors rather than merely looking odd. +Environment=CUSTOM_WIDTH=1920 +Environment=CUSTOM_HEIGHT=1080 + +# ── State ───────────────────────────────────────────────────────────────────────────── +# The Chrome profile lives under /config, so this volume is what makes the Facebook +# session survive a restart — the equivalent of marketplace-tools/profile/ today. +# +# ⚠ DO NOT COPY THE LAPTOP'S profile/ DIRECTORY IN. Moving a session to a new device +# fingerprint is very likely to trip Facebook's new-device check and can cost a +# checkpoint/lockout on the account. Log in fresh through the KasmVNC UI instead — it is +# a two-minute job, once. +Volume=/home/podman/devops/data/chromium:/config + +[Service] +# Gotcha 9 in docs/plans/284-podman-platform.md: podman does NOT create a missing +# bind-mount source, it fails the unit. Create it before the container starts. +ExecStartPre=/usr/bin/mkdir -p /home/podman/devops/data/chromium +Restart=always +RestartSec=10 +# Chrome under KasmVNC is slow to first paint on a cold image pull. +TimeoutStartSec=300 + +[Install] +# Gotcha 6: without this the unit does not come back after a CT reboot, and the failure +# is silent — the container is simply absent until someone starts it by hand. +WantedBy=default.target From 7261a0ea4f8952bf2b0aa40704e94cf5586affea Mon Sep 17 00:00:00 2001 From: Chrison Simtian Date: Thu, 27 Aug 2026 20:53:22 +1200 Subject: [PATCH 2/2] Fix the chromium quadlet so CDP is actually reachable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independent faults, both found only by deploying: 1. `SecurityOpt=` is not a quadlet key. An unsupported key makes the generator reject the WHOLE file and emit no unit, so `systemctl --user start chromium.service` reported "Unit chromium.service not found" — which reads as a rendering or linger problem, not a syntax error. Replaced with `PodmanArgs=--security-opt seccomp=unconfined`. My pre-flight "every key already appears in this repo's quadlets" check was vacuous: the glob included the file under test, so every key matched itself. SecurityOpt appears nowhere else in the repo. 2. Modern Chrome ignores --remote-debugging-address. Verified on Chrome 151 in this image: the flag reaches the process cmdline and Chrome binds 127.0.0.1:9222 regardless. With a bridge network, PublishPort=127.0.0.1:9222:9222 therefore forwarded to the container's external interface where nothing listened. The symptom was maximally misleading — CT shows a healthy pasta listener on 127.0.0.1:9222, container up, correct flags, and curl INSIDE the container returns /json/version perfectly, yet the tunnel got an empty reply. Switched to Network=host so Chrome's own localhost bind lands on the CT's loopback. This is a stronger guarantee than the publish was: CDP is loopback- only because Chrome refuses to do otherwise, not because of an argument someone could delete. Consequence: KasmVNC is on the CT's own 3000/3001, not the mapped 3010/3011. Also corrects the documented access path, which could not have worked. `ssh -L` is impossible here on three counts: Chrome will not bind off localhost; this CT has no sshd (rootless-podman hosts are driven over pct exec, and authorizedKeys is a shell-provisioner feature); and tunnelling to the node reaches the node's loopback. The workstation now uses marketplace-tools/cdp-tunnel.js, which splices stdio through `ssh pct exec 3006 -- nc 127.0.0.1 9222` — nothing new listens in the CT, the LAN never sees 9222, and no credential is added. Verified end to end: chromium.service active, container on host networking, /json/version answering through the tunnel from the workstation, and `watch.js` scraping a live Trade Me listing with no local browser. Co-Authored-By: Claude Opus 5 --- podman-host/README.md | 53 ++++++++++-------- podman-host/quadlets/chromium.container | 74 ++++++++++++++++--------- 2 files changed, 78 insertions(+), 49 deletions(-) diff --git a/podman-host/README.md b/podman-host/README.md index b8d4256..fe9fe06 100644 --- a/podman-host/README.md +++ b/podman-host/README.md @@ -7,7 +7,7 @@ which would be asleep exactly when a scheduled sweep wanted the browser. | Quadlet | What it is | Ports | |---|---|---| -| `chromium` | Persistent Chromium with a CDP endpoint, for `~/marketplace-tools` | CDP `127.0.0.1:9222` · KasmVNC `:3010` / `:3011` | +| `chromium` | Persistent Chromium with a CDP endpoint, for `~/marketplace-tools` | CDP `127.0.0.1:9222` (CT loopback) · KasmVNC `:3000` / `:3001` | Further DevOps tooling joins as **extra quadlets in `quadlets/`**, not as new one-app CTs. There is deliberately no `.network` unit: a single container needs no by-name DNS, and one @@ -29,8 +29,10 @@ into the managed marker, so editing one re-converges and restarts the unit that The KasmVNC UI exists for exactly this. It is the only interactive surface. -1. Open `http://devops-podman-host.homelab.chrison.internal:3010` (or `https://…:3011`, - self-signed) and authenticate with `CHROMIUM_KASM_USER` / `CHROMIUM_KASM_PASSWORD`. +1. Open **`http://:3000`** (or `https://:3001`, self-signed) and authenticate + with `CHROMIUM_KASM_USER` / `CHROMIUM_KASM_PASSWORD` from `secrets.env`. + The shape reserves `10.10.204.36`, but a freshly-created CT keeps its original lease + until it renews — check the current address with `pct exec 3006 -- hostname -I`. 2. Log into Facebook by hand, and complete any 2FA challenge. 3. That's it — the session persists in the `/config` volume across restarts. @@ -44,32 +46,35 @@ successful against a profile that was only ever logged into Facebook.) ## Using it from the workstation -`~/marketplace-tools/lib.js` honours **`CDP_URL`** (default `http://127.0.0.1:9222`), so -every script — `scan.js`, `watch.js`, `detail.js`, `search.js`, `trademe.js` — picks this up -with no further change. +`~/marketplace-tools/lib.js` honours **`CDP_URL`** (default `http://127.0.0.1:9222`), so every +script — `scan.js`, `watch.js`, `detail.js`, `search.js`, `trademe.js` — picks this up with no +code change. ```bash -ssh -N -L 9222:127.0.0.1:9222 root@devops-podman-host.homelab.chrison.internal & +node ~/marketplace-tools/cdp-tunnel.js & # 127.0.0.1:9222 -> CT 3006 CDP_URL=http://127.0.0.1:9222 node ~/marketplace-tools/scan.js ``` -### ⚠ The obvious tunnel is the wrong one +### ⚠ `ssh -L` does not work here, and cannot -```bash -# WRONG — forwards to hpe-01's OWN loopback, where nothing is listening. -ssh -N -L 9222:127.0.0.1:9222 root@hpe-01.homelab.chrison.internal -``` +Three facts rule it out, and each one on its own is enough: -CDP is published on **CT 3006's** loopback, not the node's. The tunnel must terminate -*inside the CT*, so SSH to the CT itself. If the CT is not directly reachable, jump: +1. **Chrome refuses to bind DevTools off localhost.** Modern Chrome (verified on 151) ignores + `--remote-debugging-address` — the flag appears on the cmdline and Chrome binds + `127.0.0.1:9222` anyway. So CDP lives on the *container's* loopback and cannot be published, + proxied or bound to the LAN even deliberately. This is why the quadlet uses `Network=host`: + it makes Chrome's own localhost bind land on the CT's loopback. +2. **This CT has no sshd.** It is a rootless-podman host driven entirely over `pct exec`, so + `ssh -L 9222:127.0.0.1:9222 root@` has nothing to connect to. `authorizedKeys` is a + `shell`-provisioner feature, not part of the `podman` app. +3. **`ssh -L … root@hpe-01` forwards to the NODE's loopback**, where nothing is listening. -```bash -ssh -N -J root@hpe-01.homelab.chrison.internal \ - -L 9222:127.0.0.1:9222 root@10.10.204.36 & -``` +So `cdp-tunnel.js` listens on the workstation's loopback and, per connection, splices stdio +through `ssh pct exec 3006 -- nc 127.0.0.1 9222`. Nothing new listens in the CT, the LAN +never sees 9222, and no credential is added anywhere. Knobs: `NODE_HOST`, `CTID`, `PORT`. -Rewriting the quadlet to publish on the CT's LAN address so that the node-level tunnel -"just works" would expose an unauthenticated CDP port to everything on VLAN 1010. Don't. +> If `cdp-tunnel.js` reports `EADDRINUSE`, a local Chrome from `launch.js` still holds 9222. +> Close it, or use `PORT=9333` and point `CDP_URL` at the same port. ## Why loopback-only is not paranoia @@ -86,10 +91,12 @@ VLAN 1010 and the session. | Symptom | Cause | |---|---| -| `connectOverCDP` hangs or refuses | Tunnel terminates on the node, not the CT — see above. | -| **403 on `/json/version`**, everything else healthy | Chrome rejects DevTools requests whose `Host` header is not `localhost`/a bare IP (DNS-rebinding defence). The access path must be an SSH tunnel presenting `localhost:9222`, never a reverse proxy on a hostname. | +| `connectOverCDP` hangs or refuses | `cdp-tunnel.js` not running, or you tried `ssh -L` — see above for why that cannot work. | +| **403 on `/json/version`**, everything else healthy | Chrome rejects DevTools requests whose `Host` header is not `localhost`/a bare IP (DNS-rebinding defence). `cdp-tunnel.js` presents `localhost:9222`, which is fine — but a reverse proxy on a hostname would be refused. | | Tabs crash, blank screenshots, "Target closed" mid-navigation | `/dev/shm` too small. Reads like a scraper bug; it isn't. `ShmSize=1g` in the quadlet. | -| Container exits immediately at startup | Chrome's sandbox vs. the nested userns — `SecurityOpt=seccomp=unconfined` is required. Last resort `CHROME_CLI=--no-sandbox`, which disables the renderer sandbox. | +| Container exits immediately at startup | Chrome's sandbox vs. the nested userns — `PodmanArgs=--security-opt seccomp=unconfined` is required. Last resort `CHROME_CLI=--no-sandbox`, which disables the renderer sandbox. | +| **`Unit chromium.service not found`** | Quadlet rejected the file over ONE unsupported key and generated nothing. `SecurityOpt=` is not a quadlet key (use `PodmanArgs=--security-opt …`). Diagnose in one second with `/usr/libexec/podman/quadlet -dryrun -user` as the `podman` user. | +| CDP silent but container healthy, Chrome flags correct, curl works *inside* the container | Chrome bound its own loopback and the publish forwarded to an interface nothing listens on. Needs `Network=host`. | | Unit absent after a CT reboot | Missing `[Install] WantedBy=default.target` (gotcha 6). Note also that quadlets do not start for ~92s after boot because `network-online.target` is never reached — gotcha 7, handled by the provisioner. | | Trade Me returns `ERR_BLOCKED_BY_RESPONSE` | Rate limiting, not a container fault. The scripts back off 30/60/90s; pace multi-page runs with `DELAY=6000`. | | Selectors break, pages look mobile | Viewport too narrow — both sites are responsive and switch DOM. `CUSTOM_WIDTH/HEIGHT` are set to 1920×1080 for this reason. | diff --git a/podman-host/quadlets/chromium.container b/podman-host/quadlets/chromium.container index c57c9b0..41d00dc 100644 --- a/podman-host/quadlets/chromium.container +++ b/podman-host/quadlets/chromium.container @@ -10,17 +10,18 @@ # which also gives us the one thing a headless container cannot: a way to do the # interactive Facebook login by hand, once. # -# ⚠ SECURITY — READ BEFORE CHANGING ANY PublishPort LINE. +# ⚠ SECURITY — READ BEFORE CHANGING THE NETWORKING. # The Chrome DevTools protocol has NO AUTHENTICATION OF ANY KIND. There is no token, no # password, no ACL. Anything that can open a TCP connection to the CDP port gets full # control of this browser, and this browser holds a logged-in Facebook session: reading # DMs, posting and editing listings, changing account settings. Treat port 9222 as # equivalent to the account password. -# · CDP is published on 127.0.0.1 ONLY — the CT's loopback, reachable by nothing on -# the network. It is used via an SSH tunnel INTO THIS CT (see ../README.md). -# · The KasmVNC UI is published on all interfaces because it must be reachable -# interactively, and is therefore gated by CUSTOM_USER/PASSWORD below. That basic -# auth is the only thing standing between VLAN 1010 and the session. +# · CDP is on the CT's LOOPBACK only, because Chrome itself refuses to bind DevTools +# anywhere else (see the Network=host block). Reached by an SSH tunnel INTO THIS CT +# — see ../README.md, and note the obvious node-level tunnel is the wrong one. +# · The KasmVNC UI is on the CT's 3000/3001 and IS reachable on VLAN 1010, because it +# has to be for the one-time Facebook login. It is gated by CUSTOM_USER/PASSWORD, +# and that basic auth is the only thing between the VLAN and the session. # · No Pangolin route, no wildcard DNS, no tunnel ingress. Ever. [Unit] @@ -33,14 +34,32 @@ Image=lscr.io/linuxserver/chromium:latest ContainerName=chromium AutoUpdate=registry -# ── Ports ───────────────────────────────────────────────────────────────────────────── -# CDP, LOOPBACK-ONLY. The bind address here is the whole security control; dropping the -# `127.0.0.1:` prefix would silently expose the session to the entire VLAN. -PublishPort=127.0.0.1:9222:9222 -# KasmVNC web UI — the interactive surface for the one-time Facebook login. HTTP on 3000, -# HTTPS (self-signed) on 3001. Gated by CUSTOM_USER/PASSWORD. -PublishPort=3010:3000 -PublishPort=3011:3001 +# ── Networking: HOST, and this is load-bearing ──────────────────────────────────────── +# ⚠ DO NOT "FIX" THIS BACK TO PublishPort. That was the first design and it does not work, +# for a reason that is invisible from the outside: +# +# MODERN CHROME IGNORES --remote-debugging-address. Verified on Chrome 151 in this very +# image: given `--remote-debugging-port=9222 --remote-debugging-address=0.0.0.0`, the +# flags appear on the process cmdline and in CHROME_CLI, and Chrome then binds +# 127.0.0.1:9222 ANYWAY — DevTools off-localhost was locked down years ago. `ss` inside +# the container shows 127.0.0.1:9222, not 0.0.0.0:9222. +# +# With a bridge network, `PublishPort=127.0.0.1:9222:9222` therefore forwards to the +# container's EXTERNAL interface, where nothing listens. The failure is maximally +# confusing: the CT shows a healthy listener on 127.0.0.1:9222 (pasta), the container is +# up, Chrome has the right flags, curl INSIDE the container returns /json/version +# perfectly — and the tunnel still gets an empty reply. +# +# Host networking makes Chrome's own localhost bind land directly on the CT's loopback, +# which is exactly the topology we wanted. It is also a STRONGER guarantee than the +# publish ever was: CDP is now loopback-only because CHROME binds it that way and refuses +# to do otherwise, not because of an argument someone could delete. +# +# Consequence: published ports do not apply under host networking, so KasmVNC is on the +# CT's OWN 3000/3001 (not the 3010/3011 an earlier draft mapped). It is still reachable on +# VLAN 1010 and still gated by CUSTOM_USER/PASSWORD — that basic auth remains the only +# thing between the VLAN and the session. +Network=host # ── Chrome's shared memory ──────────────────────────────────────────────────────────── # Chrome hammers /dev/shm and the container default (64 MB) does NOT produce a clean @@ -56,15 +75,20 @@ ShmSize=1g # If it STILL refuses to start, the next step is CHROME_CLI=--no-sandbox — but that # disables Chrome's renderer sandbox entirely, so try it only to confirm the diagnosis # and prefer keeping the sandbox on. -SecurityOpt=seccomp=unconfined +# ⚠ THIS MUST GO THROUGH PodmanArgs. `SecurityOpt=` is NOT a quadlet key — quadlet has +# SeccompProfile= (which takes a PATH, so it cannot express "unconfined"), and an +# unsupported key makes the generator reject THE WHOLE FILE, producing no unit at all. +# The failure mode is silent and misleading: `systemctl --user start chromium.service` +# reports "Unit chromium.service not found", which reads as a rendering or linger +# problem rather than a syntax error. Verify with: +# /usr/libexec/podman/quadlet -dryrun -user +PodmanArgs=--security-opt seccomp=unconfined # ── Chrome flags ────────────────────────────────────────────────────────────────────── -# ⚠ THE ADDRESS PAIR LOOKS CONTRADICTORY AND IS NOT. Chrome binds remote debugging to -# the loopback of ITS OWN network namespace by default, which podman cannot forward to. -# So Chrome listens on 0.0.0.0 *inside the container namespace*, and the loopback -# restriction is applied one layer out by PublishPort above. Both halves are required: -# drop the flag and the tunnel connects to nothing; drop the PublishPort bind address -# and 0.0.0.0 becomes literal. +# --remote-debugging-address=0.0.0.0 IS KEPT DELIBERATELY, even though Chrome 151 ignores +# it (see the Network=host block above for the evidence). It costs nothing, it documents +# the intent, and if a future Chrome ever honours it again the setup still works. What we +# rely on is Chrome's localhost bind landing on the CT loopback via host networking. # # ⚠ AND CHROME VALIDATES THE HOST HEADER. It rejects DevTools requests whose Host is not # localhost or a bare IP — a defence against DNS rebinding. This is why the access path @@ -72,13 +96,11 @@ SecurityOpt=seccomp=unconfined # hostname: Playwright's connectOverCDP fetches /json/version first, and that fetch is # what would get refused. A 403 on /json/version with everything else looking healthy is # this, not a networking fault. +# # ⚠ QUOTED, AND IT MUST BE — same trap as pulse.container's OIDC_SCOPES. systemd's # `Environment=` splits an UNQUOTED value on whitespace, so writing this bare would set -# CHROME_CLI=--remote-debugging-port=9222 and then discard -# `--remote-debugging-address=0.0.0.0` as a malformed second assignment. The result is the -# nastiest possible outcome: Chrome starts, CDP listens on the container's own loopback, -# podman forwards to nothing, and the tunnel fails with a connection error that looks like -# a networking problem rather than a missing flag. +# CHROME_CLI=--remote-debugging-port=9222 and discard the rest as a malformed second +# assignment. Verified in the generated ExecStart: it must appear as ONE quoted value. Environment="CHROME_CLI=--remote-debugging-port=9222 --remote-debugging-address=0.0.0.0" # ── KasmVNC auth ──────────────────────────────────────────────────────────────────────