Host the marketplace-tools browser on the homelab - #23
Conversation
…ace-tools Closes #447 (superproject). Establishes the DevOps stack's rootless podman + quadlet host (CT 3006) and lands a persistent Chromium with a CDP endpoint as its first quadlet, so the marketplace-tools scraping stops depending on a headed Chrome window on the MacBook — which dies whenever the window is closed and can never run unattended. Placement: #447 suggested a quadlet on the Media host (5114). That is available, so this is a deliberate choice against it. ADR-0009's "join as extra quadlets" rule is scoped to services joining their OWN stack's host; it argues against one-app CTs where a stack already has a podman host, not for parking one stack's tooling on another's. DevOps had no podman host, so creating one is the same act SmartHome (6004), Monitoring (4001) and Media (5114) each performed. Media was rejected as cross-stack coupling (a wedged Chrome restart would touch Plex and the *arr fleets); Monitoring because it deliberately keeps its own lifecycle and holds Pulse's admin creds and authentik client secret; SmartHome for being on IoT VLAN 1040. Node is pinned to hpe-01, overriding the DevOps default of desktop-01 — that is the on-demand sleep node (#191) and would be asleep exactly when a scheduled sweep needed the browser. Security posture is the reason for most of the quadlet's shape. The DevTools protocol has no authentication of any kind, and this browser holds a live Facebook session, so CDP is published on the CT's loopback only and reached by an SSH tunnel terminating inside the CT. The README spells out why the obvious `ssh -L … root@hpe-01` is the wrong tunnel, and why publishing on the CT's LAN address to make it "just work" must not be done. Cockpit is deliberately off here, unlike SmartHome's host. The KasmVNC UI is the one interactive surface — needed for the one-time Facebook login — so it is gated by CUSTOM_USER/PASSWORD from secrets.env. Gotchas encoded from docs/plans/284-podman-platform.md and hard-won elsewhere: - CHROME_CLI is QUOTED. systemd's Environment= splits unquoted values on whitespace, which would silently drop --remote-debugging-address and leave CDP bound to the container's own loopback where podman cannot forward to it — failing as a connection error that looks like a network fault. Same trap as pulse.container's OIDC_SCOPES. - ShmSize=1g. Chrome's default 64MB /dev/shm does not error cleanly; it produces tab crashes and "Target closed" mid-navigation that read as scraper bugs. - SecurityOpt=seccomp=unconfined, or Chromium exits at startup three namespaces deep (LXC -> rootless podman -> container). - ExecStartPre mkdir (gotcha 9: podman fails rather than creating a missing bind-mount source) and [Install] WantedBy=default.target (gotcha 6: silently absent after a CT reboot without it). - Chrome validates the DevTools Host header, so the access path must present localhost:9222 — a reverse proxy on a hostname would get a 403 on /json/version with everything else looking healthy. - Do NOT migrate the laptop's profile/ dir; a new device fingerprint risks a Facebook checkpoint. Log in fresh through KasmVNC. Also fills in the README members table, which had drifted: 3003/3004/3005 exist as shapes but were unlisted, and 3003 was still shown as parked woodpecker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent faults, both found only by deploying: 1. `SecurityOpt=` is not a quadlet key. An unsupported key makes the generator reject the WHOLE file and emit no unit, so `systemctl --user start chromium.service` reported "Unit chromium.service not found" — which reads as a rendering or linger problem, not a syntax error. Replaced with `PodmanArgs=--security-opt seccomp=unconfined`. My pre-flight "every key already appears in this repo's quadlets" check was vacuous: the glob included the file under test, so every key matched itself. SecurityOpt appears nowhere else in the repo. 2. Modern Chrome ignores --remote-debugging-address. Verified on Chrome 151 in this image: the flag reaches the process cmdline and Chrome binds 127.0.0.1:9222 regardless. With a bridge network, PublishPort=127.0.0.1:9222:9222 therefore forwarded to the container's external interface where nothing listened. The symptom was maximally misleading — CT shows a healthy pasta listener on 127.0.0.1:9222, container up, correct flags, and curl INSIDE the container returns /json/version perfectly, yet the tunnel got an empty reply. Switched to Network=host so Chrome's own localhost bind lands on the CT's loopback. This is a stronger guarantee than the publish was: CDP is loopback- only because Chrome refuses to do otherwise, not because of an argument someone could delete. Consequence: KasmVNC is on the CT's own 3000/3001, not the mapped 3010/3011. Also corrects the documented access path, which could not have worked. `ssh -L` is impossible here on three counts: Chrome will not bind off localhost; this CT has no sshd (rootless-podman hosts are driven over pct exec, and authorizedKeys is a shell-provisioner feature); and tunnelling to the node reaches the node's loopback. The workstation now uses marketplace-tools/cdp-tunnel.js, which splices stdio through `ssh <node> pct exec 3006 -- nc 127.0.0.1 9222` — nothing new listens in the CT, the LAN never sees 9222, and no credential is added. Verified end to end: chromium.service active, container on host networking, /json/version answering through the tunnel from the workstation, and `watch.js` scraping a live Trade Me listing with no local browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deployed and verified end to endCT 3006 is live and 1.
|
Closes Chrison-Homelab/Homelab#447.
Establishes the DevOps stack's rootless podman + quadlet host (CT 3006) and lands a persistent Chromium with a CDP endpoint as its first quadlet. The marketplace-tools scraping currently depends on a headed Chrome window on the MacBook, which dies whenever the window is closed and can never run unattended.
~/marketplace-tools/lib.jsalready honoursCDP_URL, so every script picks this up with no code change.Files
podman-host.lxc.yamlhpe-01, 2 cores / 4 GB / 12 GBpodman-host/quadlets/chromium.containerpodman-host/README.mdREADME.mdNo
.networkunit: a single container needs no by-name DNS. One gets added the moment a second container has to resolve the first.Placement — a deliberate choice, not a default
#447 suggested a quadlet on the Media host (5114). That option is available —
Homelab.Stacks.Mediadeclarespodman-host.lxc.yaml; it just looks absent locally when the submodule isn't initialised. So this is a choice against it.ADR-0009's "further services join as extra quadlets rather than as new one-app Docker CTs" is scoped to services joining their own stack's host. It argues against one-app CTs where a stack already has a podman host — not for parking one stack's tooling on another's. DevOps had no podman host, so creating one is the same act SmartHome (6004), Monitoring (4001) and Media (5114) each performed.
stack.yamlsays it is "deliberately NOT co-located with any other stack", and it holds Pulse's admin creds, API token and authentik client secret. A browser carrying a live Facebook session doesn't belong in that blast radius in either direction.Node override is load-bearing: the DevOps default is
desktop-01, the on-demand sleep node (#191) — asleep exactly when a scheduled sweep wants the browser.Security posture drives most of the quadlet's shape
The DevTools protocol has no authentication of any kind — no token, no password, no ACL. Anything that can open a TCP connection to 9222 gets full control of a browser holding a live Facebook session: reading DMs, posting and editing listings, changing account settings. Port 9222 is equivalent to the account password.
127.0.0.1only — the CT's loopback.ssh -L … root@hpe-01is the wrong tunnel (it forwards to the node's loopback, where nothing listens), and why publishing on the CT's LAN address to make that "just work" must not be done.CUSTOM_USER/PASSWORDfromsecrets.env.Gotchas encoded
From
docs/plans/284-podman-platform.mdand elsewhere:CHROME_CLIis quoted. systemd'sEnvironment=splits unquoted values on whitespace, which would silently drop--remote-debugging-addressand leave CDP on the container's own loopback where podman can't forward to it — surfacing as a connection error that looks like a network fault. Same trap aspulse.container'sOIDC_SCOPES. I wrote this bug, then caught it on review.ShmSize=1g. Chrome's default 64 MB/dev/shmdoesn't error cleanly — it produces tab crashes and "Target closed" mid-navigation that read as scraper bugs.SecurityOpt=seccomp=unconfined, or Chromium exits at startup three namespaces deep (LXC → rootless podman → container).ExecStartPremkdir (gotcha 9) and[Install] WantedBy=default.target(gotcha 6 — silently absent after a CT reboot otherwise).Hostheader, so the access path must presentlocalhost:9222. A 403 on/json/versionwith everything else healthy is this, not networking.profile/— a new device fingerprint risks a Facebook checkpoint. Log in fresh through KasmVNC.Verification
Infrastructure/schema/shape.schema.json.Environment=, no unescaped%(gotcha 5).build.sh Previewnot yet run — a liveDeploy --stack Workshopwas in progress on the same checkout and holds.fallout/temp/build.logopen. Worth a dry-run before merge.Drift fixed in passing
The members table listed 3000–3003 only, with 3003 shown as parked
woodpecker. In fact 3003 (shell), 3004 (cloudflared) and 3005 (newt) all exist as shapes. Added, along with 3006.🤖 Generated with Claude Code