Host the marketplace-tools browser on the homelab

README.md

@@ -26,12 +26,20 @@ flowchart LR
 
 CTID block **3000–3999** (declared in [`stack.yaml`](stack.yaml); members inherit its `defaults`).
 
+> **`podman-host` (3006) overrides `node`.** The stack default is `desktop-01`, which is the
+> on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)); 3006 is
+> pinned to `hpe-01` because it hosts an always-on browser that a scheduled sweep has to reach.
+
 | CTID | Member | Script | Channel | Status |
 |------|--------|--------|---------|--------|
 | 3000 | [forgejo](forgejo.lxc.yaml) | `ct/forgejo.sh` | stable | ready to deploy |
 | 3001 | [forgejo-runner](forgejo-runner.lxc.yaml) | `ct/forgejo-runner.sh` | **dev** (ProxmoxVED) | ready — in-dev script |
 | 3002 | [github-runner](github-runner.lxc.yaml) | `ct/github-runner.sh` | stable | ready (new, independent of CT 2005) |
-| 3003 | woodpecker | — | — | **parked** — no community-script in either repo yet |
+| 3003 | [shell](shell.lxc.yaml) | `ct/debian.sh` + `shell` provisioner | stable | deployed |
+| 3004 | [cloudflared](cloudflared.lxc.yaml) | `ct/cloudflared.sh` | stable | deployed |
+| 3005 | [newt](newt.lxc.yaml) | `ct/debian.sh` + `newt` provisioner | stable | deployed |
+| 3006 | [podman-host](podman-host.lxc.yaml) | `ct/podman.sh` | stable | **new** ([#447](https://github.com/Chrison-Homelab/Homelab/issues/447)) — pinned to `hpe-01`, see [its README](podman-host/README.md) |
+| — | woodpecker | — | — | **parked** — no community-script in either repo yet |
 
 ## Deploying
 

podman-host.lxc.yaml

@@ -0,0 +1,128 @@
+# DevOps podman host — the stack's rootless Podman + quadlet host (ADR-0009, #447).
+#
+# DELIBERATELY GENERIC, exactly like stacks/SmartHome/podman-host.lxc.yaml. This is
+# *the DevOps stack's container host*, not "the browser box". The marketplace-tools
+# Chromium is simply the first service to land here — chosen as the pilot because it
+# is self-contained (one container, no NFS, all state in one volume) and nothing else
+# depends on it, so the fiddly rootless plumbing proves itself where the blast radius
+# is smallest. Further DevOps tooling joins as extra quadlets in podman-host/quadlets/
+# rather than as new one-app CTs.
+#
+# WHY A NEW CT AND NOT A QUADLET ON AN EXISTING HOST — #447 suggested landing this on
+# the Media podman host (CT 5114). That option IS available (Homelab.Stacks.Media
+# declares podman-host.lxc.yaml + podman-host/; it just looks absent locally when the
+# submodule is not initialised), so this is a deliberate choice against it, not an
+# absence of alternatives.
+#
+# The ADR-0009 rule quoted in SmartHome's host — "further SmartHome services join as
+# extra quadlets rather than as new one-app Docker CTs" — is scoped to services joining
+# THEIR OWN stack's host. It argues against one-app CTs inside a stack that already has
+# a podman host; it does not argue for parking one stack's tooling on another stack's
+# host. DevOps has no podman host, so creating one is the same act SmartHome (6004),
+# Monitoring (4001) and Media (5114) each already performed — not a violation of it.
+# The alternative homes were rejected on their merits:
+#   · Media 5114 — cross-stack. A browser holding a live Facebook session would share a
+#     host with Plex and both *arr fleets, coupling the lifecycle of a scraper to the
+#     media stack; a restart to fix a wedged Chrome would touch unrelated services.
+#   · Monitoring 4001 — stack.yaml says it is "deliberately NOT co-located with any
+#     other stack: this host watches everything else, so it needs its own lifecycle".
+#     It also holds Pulse's admin creds, API token and authentik client secret, and a
+#     browser carrying a live Facebook session does not belong in that blast radius
+#     in either direction.
+#   · SmartHome 6004 — sits on IoT VLAN 1040, firewalled off from 1010.
+# So this establishes the DevOps stack's own podman host, the same role 4001 and 6004
+# play for their stacks. Net CT count is unchanged versus the alternative of a
+# one-app CT, and the next piece of DevOps tooling costs zero CTs.
+#
+# ⚠ NODE OVERRIDE IS LOAD-BEARING. The DevOps stack default is `desktop-01`, which is
+# the designated on-demand SLEEP node (#191) — a browser hosted there would be asleep
+# precisely when a scheduled sweep wanted it. Pinned to hpe-01, which is an always-on
+# sentinel with the measured headroom for it (see #447: 9.1Gi available, load 0.47).
+# Longer term the natural home is node #4 once #334 unblocks; migrating means changing
+# this one line.
+#
+# WHAT THE ENGINE DOES HERE (app: podman → PodmanProvisioner, superproject #284):
+#   ct/podman.sh creates the CT (rootful podman + a root podman.socket), then the
+#   provisioner converts it to the rootless quadlet model — masks that socket, adds
+#   the non-root `podman` user with an in-map subuid range + linger, makes
+#   network-online.target reachable, renders the quadlets below and drives
+#   `systemctl --user`. Read docs/plans/284-podman-platform.md before touching a
+#   quadlet: the eleven gotchas there are all load-bearing.
+#
+# EXPOSURE — INTERNAL ONLY, AND THIS ONE MATTERS MORE THAN USUAL. No tunnel ingress,
+# no Pangolin route, no wildcard DNS. The container holds a logged-in Facebook
+# session, and the Chrome DevTools protocol has NO AUTHENTICATION WHATSOEVER: anything
+# that can reach its port gets full control of that session — read messages, post
+# listings, change the account. The quadlet therefore publishes CDP on the CT's
+# LOOPBACK only and is reached by SSHing INTO THIS CT and tunnelling. See
+# podman-host/README.md for the exact invocation and why the obvious
+# `ssh -L … root@hpe-01` is the wrong one.
+#
+# Validate against ../../Infrastructure/schema/shape.schema.json.
+
+apiVersion: homelab/v1
+kind: LXC
+
+metadata:
+  name: podman-host
+  stack: DevOps
+  tags: [podman, quadlet, container-host, adr-0009, browser-automation]
+
+spec:
+  app: podman            # community-scripts ct/podman.sh + the `podman` provisioner
+  ctid: 3006             # in-block (3000–3999); 3000 forgejo · 3001/3002 runners ·
+                         # 3003 shell · 3004 cloudflared · 3005 newt
+  node: hpe-01           # OVERRIDES the DevOps default of desktop-01 — see the node
+                         # note above; desktop-01 sleeps (#191), this must not.
+  cores: 2               # Chrome is the only real consumer; 2 is enough for a scraper
+                         # driving one or two tabs, and leaves hpe-01's 4 cores alone.
+  memory: 4096           # MB — Chrome alone sits at 1–1.5 GB with a few tabs, plus the
+                         # KasmVNC server, plus podman itself. 2048 was the first draft
+                         # and is too tight once /dev/shm is actually used (see the
+                         # quadlet's ShmSize): shm is tmpfs and counts against this.
+  disk: 12               # GB — the linuxserver image is ~2 GB of layers, the rest is
+                         # the Chrome profile volume (cache grows; it is disposable).
+
+  # DHCP reservation (#416) so the SSH target for the tunnel is stable. The DNS record
+  # rides the reservation, so name and lease move together and nothing points at a
+  # bare address again.
+  network:
+    reservation:
+      fixedIp: 10.10.204.36
+      name: devops podman-host (CT 3006)
+
+  features:
+    nesting: true        # rootless podman needs nesting for its own nested userns
+    keyctl: true         # containers keyring — settable only via the pct/SSH create path
+    fuse: true           # fuse-overlayfs fallback storage driver
+                         # NOTE: ct/podman.sh silently DROPS var_fuse; the provisioner
+                         # reconciles features from this shape after create (#284).
+
+  # node is set above; network/os/source/timezone otherwise inherited from stack.yaml
+  # (VLAN 1010, debian 13, unprivileged, onboot).
+
+  config:
+    # Rootless user + subuid window. The default 10000:50000 fits inside an
+    # unprivileged LXC's own 65536-uid map — the provisioner verifies this against
+    # /proc/self/uid_map and fails loudly rather than letting podman break at first run.
+    user: podman
+    quadlets: podman-host/quadlets
+
+    # Rootless API socket for local inspection (`podman ps` over the user socket).
+    userSocket: true
+
+    # ⚠ COCKPIT DELIBERATELY OFF, unlike SmartHome's host. Cockpit would put a
+    # privileged web console on the same CT as a browser holding live account
+    # credentials, on a VLAN the workstation can already reach. The management value
+    # is small here — one container — and `pct exec` plus the user socket cover it.
+    # cockpit: false   (omitted; the default)
+
+    # podman secrets seeded from the gitignored secrets.env, add-only (rotation is an
+    # explicit operator action, never a silent converge side effect).
+    #
+    # These gate the KasmVNC web UI, which is the one interactive surface: it exists so
+    # the Facebook login can be done by hand once. Without auth, anyone on VLAN 1010
+    # could open that UI and drive the logged-in session. Set both in secrets.env.
+    secrets:
+      chromium_kasm_user: CHROMIUM_KASM_USER
+      chromium_kasm_password: CHROMIUM_KASM_PASSWORD

podman-host/README.md

@@ -0,0 +1,115 @@
+# DevOps podman host (CT 3006)
+
+The DevOps stack's rootless Podman + quadlet host, per [ADR-0009](../../../docs/adr/ADR-0009-container-runtime.md)
+and [#447](https://github.com/Chrison-Homelab/Homelab/issues/447). Pinned to **hpe-01** —
+the stack default `desktop-01` is the on-demand sleep node ([#191](https://github.com/Chrison-Homelab/Homelab/issues/191)),
+which would be asleep exactly when a scheduled sweep wanted the browser.
+
+| Quadlet | What it is | Ports |
+|---|---|---|
+| `chromium` | Persistent Chromium with a CDP endpoint, for `~/marketplace-tools` | CDP `127.0.0.1:9222` (CT loopback) · KasmVNC `:3000` / `:3001` |
+
+Further DevOps tooling joins as **extra quadlets in `quadlets/`**, not as new one-app CTs.
+There is deliberately no `.network` unit: a single container needs no by-name DNS, and one
+gets added the moment a second container has to resolve the first (see the Monitoring
+host's `monitoring.network` for why it becomes mandatory then).
+
+## Deploy
+
+```bash
+set -a && . ./secrets.env && set +a    # CHROMIUM_KASM_USER + CHROMIUM_KASM_PASSWORD
+./build.sh Preview --stack DevOps      # dry-run
+./build.sh Deploy  --stack DevOps      # live apply
+```
+
+Config is **not** baked into the guest — the quadlets are rendered onto the host and folded
+into the managed marker, so editing one re-converges and restarts the unit that consumes it.
+
+## One-off: log Chromium into Facebook
+
+The KasmVNC UI exists for exactly this. It is the only interactive surface.
+
+1. Open **`http://<ct-ip>:3000`** (or `https://<ct-ip>:3001`, self-signed) and authenticate
+   with `CHROMIUM_KASM_USER` / `CHROMIUM_KASM_PASSWORD` from `secrets.env`.
+   The shape reserves `10.10.204.36`, but a freshly-created CT keeps its original lease
+   until it renews — check the current address with `pct exec 3006 -- hostname -I`.
+2. Log into Facebook by hand, and complete any 2FA challenge.
+3. That's it — the session persists in the `/config` volume across restarts.
+
+> ⚠ **Do not copy `~/marketplace-tools/profile/` into the volume.** Moving a session onto a
+> new device fingerprint is very likely to trip Facebook's new-device check, and can cost a
+> checkpoint or a temporary lockout on the account. Logging in fresh takes two minutes.
+
+Trade Me needs no login for scraping. (Note that this also means the container cannot add
+things to a Trade Me *watchlist* — that has bitten us before, when clicks were reported as
+successful against a profile that was only ever logged into Facebook.)
+
+## Using it from the workstation
+
+`~/marketplace-tools/lib.js` honours **`CDP_URL`** (default `http://127.0.0.1:9222`), so every
+script — `scan.js`, `watch.js`, `detail.js`, `search.js`, `trademe.js` — picks this up with no
+code change.
+
+```bash
+node ~/marketplace-tools/cdp-tunnel.js &                     # 127.0.0.1:9222 -> CT 3006
+CDP_URL=http://127.0.0.1:9222 node ~/marketplace-tools/scan.js
+```
+
+### ⚠ `ssh -L` does not work here, and cannot
+
+Three facts rule it out, and each one on its own is enough:
+
+1. **Chrome refuses to bind DevTools off localhost.** Modern Chrome (verified on 151) ignores
+   `--remote-debugging-address` — the flag appears on the cmdline and Chrome binds
+   `127.0.0.1:9222` anyway. So CDP lives on the *container's* loopback and cannot be published,
+   proxied or bound to the LAN even deliberately. This is why the quadlet uses `Network=host`:
+   it makes Chrome's own localhost bind land on the CT's loopback.
+2. **This CT has no sshd.** It is a rootless-podman host driven entirely over `pct exec`, so
+   `ssh -L 9222:127.0.0.1:9222 root@<ct-ip>` has nothing to connect to. `authorizedKeys` is a
+   `shell`-provisioner feature, not part of the `podman` app.
+3. **`ssh -L … root@hpe-01` forwards to the NODE's loopback**, where nothing is listening.
+
+So `cdp-tunnel.js` listens on the workstation's loopback and, per connection, splices stdio
+through `ssh <node> pct exec 3006 -- nc 127.0.0.1 9222`. Nothing new listens in the CT, the LAN
+never sees 9222, and no credential is added anywhere. Knobs: `NODE_HOST`, `CTID`, `PORT`.
+
+> If `cdp-tunnel.js` reports `EADDRINUSE`, a local Chrome from `launch.js` still holds 9222.
+> Close it, or use `PORT=9333` and point `CDP_URL` at the same port.
+
+## Why loopback-only is not paranoia
+
+The DevTools protocol has **no authentication of any kind** — no token, no password, no
+ACL. Anything that can open a TCP connection to 9222 gets full control of a browser holding
+a live Facebook session: reading DMs, posting and editing listings, changing account
+settings. Port 9222 is equivalent to the account password, and should be treated that way.
+
+The KasmVNC UI is the deliberate exception — it must be reachable to be useful — which is
+why `CUSTOM_USER`/`PASSWORD` are non-optional. That basic auth is the only thing between
+VLAN 1010 and the session.
+
+## Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `connectOverCDP` hangs or refuses | `cdp-tunnel.js` not running, or you tried `ssh -L` — see above for why that cannot work. |
+| **403 on `/json/version`**, everything else healthy | Chrome rejects DevTools requests whose `Host` header is not `localhost`/a bare IP (DNS-rebinding defence). `cdp-tunnel.js` presents `localhost:9222`, which is fine — but a reverse proxy on a hostname would be refused. |
+| Tabs crash, blank screenshots, "Target closed" mid-navigation | `/dev/shm` too small. Reads like a scraper bug; it isn't. `ShmSize=1g` in the quadlet. |
+| Container exits immediately at startup | Chrome's sandbox vs. the nested userns — `PodmanArgs=--security-opt seccomp=unconfined` is required. Last resort `CHROME_CLI=--no-sandbox`, which disables the renderer sandbox. |
+| **`Unit chromium.service not found`** | Quadlet rejected the file over ONE unsupported key and generated nothing. `SecurityOpt=` is not a quadlet key (use `PodmanArgs=--security-opt …`). Diagnose in one second with `/usr/libexec/podman/quadlet -dryrun -user` as the `podman` user. |
+| CDP silent but container healthy, Chrome flags correct, curl works *inside* the container | Chrome bound its own loopback and the publish forwarded to an interface nothing listens on. Needs `Network=host`. |
+| Unit absent after a CT reboot | Missing `[Install] WantedBy=default.target` (gotcha 6). Note also that quadlets do not start for ~92s after boot because `network-online.target` is never reached — gotcha 7, handled by the provisioner. |
+| Trade Me returns `ERR_BLOCKED_BY_RESPONSE` | Rate limiting, not a container fault. The scripts back off 30/60/90s; pace multi-page runs with `DELAY=6000`. |
+| Selectors break, pages look mobile | Viewport too narrow — both sites are responsive and switch DOM. `CUSTOM_WIDTH/HEIGHT` are set to 1920×1080 for this reason. |
+
+## What this unblocks
+
+A sweep that runs **on the homelab** rather than the laptop, which is the actual point:
+
+```bash
+node ~/marketplace-tools/scan.js nodes >> /var/log/marketplace-scan.log
+```
+
+Driven by a systemd timer on the node instead of a human remembering. The "check back
+before the auction closes" case has been impossible until now: a cloud agent cannot reach
+the laptop, and Trade Me is a client-rendered SPA, so plain HTTP fetches return no live bid
+data.