feat(nebula): advertise a nebula relay — a read-only check, and the sudo apply beside it

[ZacxDev]

What

relay.relays in a nebula config is the list of mesh IPs peers may use to relay packets TO this host — not the list of relays this host uses to reach others. That is why use_relays: true on its own changes nothing. Workbench renders relays: [], so no peer can relay to it and a peer that cannot hole-punch (CGNAT, symmetric NAT, off-LAN) simply never connects. That is the blocker behind clawgate #497.

This PR ships two scripts:

file	privilege	what it does
nix/system/check-nebula-relays.sh	none	read-only verifier; no writes, no rebuild, no sudo
nix/system/apply-nebula-relay.sh	root	patches /etc/nixos/configuration.nix, then nixos-rebuild test → verify → switch → verify

🔴 The second one is a root-privileged, system-mutating script. An earlier version of this body said the PR "ships only the check … does NOT edit that file and does NOT rebuild … no sudo". That was true at 405ecfd4 and false from e894fe10 onward — the apply script has been on the branch since. Corrected here; read the apply script as what it is.

Why the check exists separately

• Parses relay.relays using the stdlib only, so it has no PyYAML dependency (import yaml is not satisfiable from the python3 a sudo bash … actually gets here).
• Never greps the rendered YAML for the relay address. lighthouse.hosts carries the same mesh addresses, so a pattern like ^\s+- <relay>$ matches a config whose relays: is empty. Measured: it reported a match on the pre-change file and would have certified a change that had not happened. The trap is in the header and asserted in the self-test.
• Gates its own verdict on --self-test, which must read a relays: [] fixture and a populated one correctly before any PASS is printed.
• Exit codes: 0 advertised · 1 not advertised · 2 cannot determine. Every rc-2 path also prints one machine-readable REASON: <token> line, so a caller can branch on the condition instead of pattern-matching prose. A --self-test failure is rc 2, not rc 1 — 1 means "the relay is absent", a different claim. (An earlier version of this body said rc 1. Corrected.)

Round 3 — the eight findings from the audit of e894fe10..e7f2e45a

finding	fix
F-A the rollback was not a rollback	switch registers the profile and installs the bootloader before activation, so a failed activation still moved the machine — measured: generation 387 held relays: - 10.42.0.2 after the script reported a clean rollback. The order is now nixos-rebuild test → verify → nixos-rebuild switch → verify. test activates without registering a generation or touching the bootloader, so a failure there really does persist nothing. The trap tracks five states and its message distinguishes never activated · test failed · activated-not-persisted · switch attempted (profile may have moved, with readlink /nix/var/nix/profiles/system to check) · persisted.
F-B the anchor was not scoped to NEBULA_NET	reproduced: NEBULA_NET=travel reported DONE … advertised by nebula@travel.service while the four lines landed in the mesh block. The services.nebula.networks.<NET> = { block is now located first (brace depth, comments and string bodies skipped) and the anchor pair is only accepted inside it. No unique declaration, a block that never closes, or an anchor count ≠ 1 inside the range each abort.
F-C root wrote to a predictable path in world-writable /tmp	mktemp -d (0700) for the verifier capture, removed by the trap; the patch temp is a mktemp sibling of $CFG so the final mv stays atomic.
F-D a missing backup skipped the rollback silently	the else exists, cp's status is checked, and failure prints 🔴 ROLLBACK FAILED — your config is still patched at <path> with the exact four lines to delete by hand.
F-E [ -f "$CFG" ] accepted a symlink and mv destroyed it	refused, not followed — the deliberate half of the choice, because following a caller-chosen path as root is the same hazard as the /tmp one. Compared via readlink -f, not [ -L ], so a symlinked directory component is caught too; the error names the resolved target and the one-flag fix.
F-F a fixture comment claimed coverage it did not provide	fixture 5's comment said it killed a mutant that scans past the end of the relay block. Measured: replacing the terminator with if False: left --self-test green. Fixture 5 keeps its (real) shape with an honest comment, and a new fixture 5b — a later top-level am_relay: — kills the mutant with its own message. 🔴 The comment states plainly that pkgs.formats.yaml cannot emit that shape, so 5b pins the parser's structural correctness, not a reachable production config.
F-G the egress-cost warning never reached the operator	on rc 1 the verifier's full output is now printed before proceeding, so the 🔴 paragraph about relayed traffic egressing a billed host is read at the moment of choosing.
F-I the rc-2 retry was wider than its comment	narrowed: the single systemctl restart fires only on REASON: unit-process-disagree. Six other rc-2 conditions (inactive unit, unreadable -config, parser failure, …) no longer drop every mesh session for nothing.

Nits also fixed: unused indent and local spellings removed; the preflight tool list now names every binary the script execs (diff, mktemp, tr, cut, wc, cp, mv, date, readlink, grep, …); both scripts now warn that applying a relay restarts the mesh the operator may be connected over.

F-H — the scripts now have repo-gate coverage

They had none: --self-test was never invoked by scripts/gate.sh and the apply script had no automated exercise at all.

• scripts/tests/test_nebula_relay_apply.py — 30 tests, in the existing hermetic scripts/tests target, so scripts/gate.sh runs them. Nothing in it can reach a real nixos-rebuild, /nix/var/nix/profiles/system or /etc/nixos: the config is a fixture under tmp_path reached via NEBULA_CFG, and id/ip/systemctl/nixos-rebuild/nix-instantiate are shims on PATH.
• The verifier under test is the real one, and it reads the running process — so the rig spawns a real child whose argv carries -config <rendered.yml> and the nixos-rebuild shim re-renders that yml from the .nix the script just patched. Every fake process is stopped by its own PID; no pattern ever reaches pkill.
• scripts/tests/mutants-nebula-relay.sh — the mutation battery, in the tree so "mutation-verified" can be re-derived rather than believed. Same shape as the existing mutants-*.sh: CONTROL-CLEAN / CONTROL-KILL / CONTROL-DETECTOR, each mutation applied-verified by occurrence count, and each mutant naming the test that must kill it and the message it must die with — a mutant that dies to a different test's error reports WRONG-KILLER rather than counting as covered.

Coverage matrix (all red before the fix, green at HEAD)

case	rc	resulting config state
happy path	0	patched, exactly 4 lines, inside the mesh block; 1 backup
re-run when already satisfied	0	unchanged, no backup, nixos-rebuild never invoked
wrong host	1	unchanged, no rebuild
config already carries a relays list	1	unchanged, merge-by-hand message naming the line
nixos-rebuild test fails (F-A)	≠0	restored; "NOT PERSISTED"; switch never reached
test succeeds, verifier fails	≠0	restored; "ACTIVATED, NOT PERSISTED"
switch fails after a good test	≠0	restored; "THE PROFILE MAY HAVE MOVED" + readlink …; never claims nothing was persisted
backup missing at rollback (F-D)	≠0	still patched, and the message says so
CFG is a symlink (F-E)	1	symlink intact, target untouched
CFG under a symlinked directory	1	target untouched
NEBULA_NET whose block lacks the anchor (F-B)	1	unchanged — mesh block byte-identical
two anchors, the named net is the second (F-B)	0	lands in mesh, not the first anchor
unknown NEBULA_NET	1	unchanged
no predictable /tmp path live while the verifier runs (F-C)	0	observed from inside the run
verifier FAIL text reaches stdout (F-G)	0	EGRESSES THE RELAY present
deferred restart → retried once (F-I)	0	1 restart per verify pass
a different rc 2 → no restart (F-I)	≠0	0 restarts
inactive unit	≠0	rejected in the preflight, before any write
invalid Nix after the patch	1	restored, no backup taken, no temp sibling left
check --self-test (F-H)	0	—

Mutation results — 22/22, one declared survivor

20 mutants killed by their named test, with that test's own message, plus CONTROL-CLEAN / CONTROL-KILL / CONTROL-DETECTOR green (22/22 including the controls). Two entries are recorded honestly rather than rounded up:

• M-X-1-line-count-guard-off ([ "$added" = "4" ]) is a declared survivor: the awk pass already exits 3 unless it made exactly one insertion, and one insertion is always four lines, so no input reachable through the script's own guards can move $added. Kept as defence in depth, not counted as coverage. The battery asserts it survives, so if the justification ever stops holding the battery says so.
• M-FB-2-patch-pass-unscoped is killed, but by the awk n != 1 guard (two anchors ⇒ two insertions ⇒ abort), not by the assertion naming the wrong block. So the range guard inside the patch pass is defence in depth behind the scoped anchor count; the battery scores it on the message it really dies with.

Both scripts also exercised end-to-end against a copy of the live /etc/nixos/configuration.nix (not the file itself): block located at lines 702–733, anchor unique inside it, +4 lines landing in the mesh block, rebuild log == ["test", "switch"]. shellcheck -S warning is clean on both.

One thing the gate turned up that is NOT this PR's

A full scripts/gate.sh --tier pytest on the branch reports two failures. One was mine and is fixed (test_no_test_writes_a_usr_bin_env_shebang_at_runtime — the harness wrote its own #!/usr/bin/env bash; it now goes through testlib.mockbin.write_exec, and the shim bodies are POSIX sh, controlled with dash -n and a full behavioural re-run under dash, because this host's /bin/sh is bash and would hide a bashism until the sandbox).

The other — test_clawgate_task_interview_guard.py::test_a_body_file_written_by_a_heredoc_on_the_same_line_is_read — is not from this work. scripts/claude-hooks is byte-identical between this branch's tip before my commits and after (ls-tree OID 2a363038 both sides), and main has since deleted that test along with 85 lines of the guard. So it is a branch-is-behind-main artifact, and the merge resolves it.

Gating was therefore done on an integration tree (origin/main + this branch), not on the branch alone: scripts/gate.sh --tier pytest there is PASS, 30/30 targets, TOTAL collected=21508 passed=21505 skipped=3 failed=0, with scripts/tests at 12477/12477.

Not verified

• Nothing was applied for real. /etc/nixos/configuration.nix is unmodified, nixos-rebuild was never run in any form, and /nix/var/nix/profiles/system was not touched or rolled back. The pending generation from the earlier run is still in place, by the operator's choice.
• nixos-rebuild test's no-profile-no-bootloader property is taken from its documented behaviour and from the F-A measurement of switch; it was not re-measured on this host, because doing so means running a rebuild.
• The /tmp symlink attack is shown closed by observing that no predictable path is live while the verifier runs (plus a source-level check). A literal symlink-planting reproduction against the fixed script was not performed — it needs the script's $$ in advance.
• Nothing off-LAN, and no real relayed traffic, has been exercised.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW

[ZacxDev]

Audit round 1 — findings and what the fix round did

Full adversarial audit at 4ca29d7a. Verdict was needs rework (3 🔴, 5 🟡, 2 🟢). Rather than fix six findings on the apply-script design, the fix round replaced it with a read-only verifier — see 405ecfd4.

Findings F2, F3, F4, F6 and F7 are closed by deletion: the apply script no longer exists, so there is no second-run verdict, no substring idempotency guard, no timestamped backup, no host assumption and no half-patched-config failure mode.

1. F1 (public IP literal in a PUBLIC repo, gate red) — fixed: the file carrying it is deleted; the replacement has no public-IP literal, and the fixtures use RFC 5737 documentation addresses, which the scan explicitly treats as non-reportable. Negative-controlled: test_no_public_ips is green on the branch and red at the old file:line once that file is staged again.
2. F2 (verdict cannot distinguish "already correct" from "broken"; second run FAILs a correct host) — closed by deletion: no apply script, no RENDERED_BEFORE/RENDERED_AFTER pair. The verifier is stateless and idempotent by construction; running it twice gives the same answer.
3. F3 ("relays = [" whole-file substring silently half-applies where apply-travel-prep.sh already ran) — closed by deletion: nothing patches the config now, and the verifier asserts the DESIRED END STATE (is the expected relay in relay.relays) rather than the presence of a token.
4. F4 (same-second re-run overwrites the only pristine backup, so the printed rollback restores the patched file) — closed by deletion: no backup is taken because no write is made.
5. F5 (PyYAML unavailable on the PATH sudo actually gets, so the script could not run as documented) — fixed: the replacement parses relay.relays with the stdlib only. Run and verified under the default-PATH python3, which cannot import yaml.
6. F6 (no host assertion, and hostname does not discriminate — both hosts are "nixos") — closed by deletion: the verifier only reads, so running it on the wrong host misreports nothing and changes nothing.
7. F7 (no failure atomicity around nixos-rebuild; config left patched with no rollback instruction) — closed by deletion: there is no rebuild and no patch.
8. F9 (the comment's justification for the second relay was factually wrong — 10.42.0.1 is publicly reachable via the home uplink, not a LAN-only fallback) — fixed: the second relay is dropped entirely; the snippet the verifier prints lists the production lighthouse only. The incorrect claim is corrected in the PR body.
9. F10 (relays_ok asserted only one of the two relays; is-active ran after PASS was printed) — closed: only one relay is expected now, and there is no unit-state check printed after the verdict.
10. F8 (unpriced Hetzner egress for relayed traffic) — NOT addressed in code, and deliberately: it is a property of using a relay at all, not of this script. Recorded here so it is open rather than absent.

Not fixed, recorded as open: F8 (relayed workbench traffic bills Hetzner egress). This is inherent to advertising the production lighthouse as a relay and is a decision for the operator, not a defect in the check.

The verifier's own evidence — seven exit-code cases and two parser mutants — is in the PR body and the commit message. The relay is still not applied; that needs the operator's password.

[ZacxDev]

Audit round 2 — findings and what the fix round did

Delta audit of 4ca29d7a..405ecfd4. Verdict was merge after fixing 🟡-1 and 🟡-2; all three 🟡 and three of the four 🟢 are now fixed in e7f2e45a. Every finding was reproduced before being fixed — none was taken on the auditor's word.

1. 🟡-1 (the file claimed a PASS meant the RUNNING service had loaded the value, while the code read only `systemctl cat`) — fixed. Confirmed first that `systemctl cat` exits 0 for an inactive unit, so the claim was false, not merely weak. The config now comes from /proc/<MainPID>/cmdline, the unit must be active, and a unit-file/process disagreement is its own reported outcome rather than a PASS.
2. 🟡-2a (a one-line flow mapping parsed to nothing and was reported as "relays is EMPTY") — fixed. Reproduced: the block-capture loop returned [] on that shape. A flow-mapping branch was added and is covered by a self-test fixture.
3. 🟡-2b (a python traceback inside the command substitution did not abort under set -e, so a crashed parser read as a definitive answer) — fixed. Reproduced with a non-UTF-8 config. The parser now exits 2 on unreadable/undecodable input and on an absent relay key, and the caller checks that status.
4. 🟡-3 (the -config extraction matched only `-config X`, so a drop-in using another spelling silently fell back to the BASE unit's config) — fixed. Measured old vs new on the same input: old returned /OLD.yml for `--config` and `-config=`, new returns /NEW.yml for all four spellings Go's flag package accepts. One extractor now feeds from both the process cmdline and the unit's last ExecStart.
5. 🟢-1 (the header claimed the generator never emits flow style) — fixed. The claim was false in the one case that matters: `relays: []`, the state the script exists to detect, IS flow style. Both branches are now marked load-bearing.
6. 🟢-2 (--self-test exit code collided with "relay absent") — fixed: self-test failure exits 2, relay-absent exits 1.
7. 🟢-3 (self-test fixtures too narrow; mutants M1 and M3 survived) — fixed. Fixtures went from 3 to 8. M1 and M3 now killed, plus a new mutant for the flow-mapping branch; positive control green.
8. 🟢-4 / F8 (relayed traffic egresses a billed host, previously declared open) — now stated in the remediation text the script prints, so the operator sees it at the moment they choose a relay.
9. M5 (terminator `break` -> `in_block = False`) is NOT claimed as killed. Measured: it differs only on a config with a DUPLICATE top-level `relay:` key, which a JSON-derived generator cannot emit. It is an equivalent mutant; a fixture for an impossible input would be manufacturing coverage, not gaining it.
10. apply-nebula-relay.sh adjusted: because the verifier now returns 2 when the running process has not picked up a new config, the post-switch check retries ONCE after restarting the unit before rolling back.

Field report from the first real run. The operator ran apply-nebula-relay.sh under sudo while this was in review. The patch applied correctly, the run failed at or after nixos-rebuild switch, and the rollback worked — /etc/nixos/configuration.nix came back byte-identical to its backup, which was preserved.

🔴 It also exposed a design defect not in either audit round: this machine has a pending 26.05 → 26.11 release jump, so nixos-rebuild switch rebuilt the world (2h16m, largely wine-wow-11.0) and attempted a full release activation. A script advertised as "add four lines to the nebula config" triggered an OS upgrade. It needs a preflight comparing the pending closure against the running system, refusing or warning when they differ by more than the delta. Not fixed here — recorded so it is open rather than absent.

Separately and pre-existing: nothing has activated on that host since it booted 30 days ago (/run/current-system is 26.05 while the profile points at a 26.11 generation), which is why nothing done to configuration.nix has been taking effect.