Add a launchd launcher for supervised runs and document what kills them

docs/runbook.md

@@ -4,6 +4,96 @@ This is the canonical procedure for paid no-tools benchmark runs. Treat
 `results/local/` as scratch space; git history and release snapshots are the
 archive, not superseded local files.
 
+## 0. Launch long runs under launchd
+
+A paid `policybench run` takes hours. Start it with `scripts/launch_run.sh`,
+which installs a per-run launchd user agent, rather than with `nohup … &` from
+a terminal or an agent session. The job then belongs to launchd, not to the
+shell, terminal, or Claude Code session that started it: it keeps running when
+that process group is torn down, it comes back after a reboot (`RunAtLoad`),
+and if it is killed before finishing, launchd relaunches it and the supervisor
+resumes from `run_state.json` and the per-scenario CSVs (`KeepAlive` on an
+unfinished exit). A run that stops on purpose (budget stop, rounds exhausted,
+every scenario complete) unloads itself.
+
+```bash
+export OPENROUTER_API_KEY=...            # provider credentials are copied into the job
+scripts/launch_run.sh start --name glm53 --model glm-5.3 \
+  --scenario-manifest paper/snapshot/20260501/us_scenarios.csv \
+  --run-dir results/local/newmodels/glm53/run \
+  --budget-usd 40 --max-workers 5
+
+scripts/launch_run.sh status glm53       # launchd state, heartbeat summary, last log lines
+scripts/launch_run.sh logs glm53         # supervisor.log and launchd.log from the run dir
+scripts/launch_run.sh stop glm53         # SIGTERM, then SIGKILL, of the job's process group
+scripts/launch_run.sh list
+```
+
+Run it from the checkout whose code you want to benchmark; that checkout
+becomes `PYTHONPATH` and the supervisor executable defaults to its `.venv`
+(or the main clone's `.venv` when run from a worktree). Add `--dry-run` to
+print the plist instead of installing it. Variables named `*_API_KEY` or
+`*_API_TOKEN`, or prefixed `ANTHROPIC_`, `OPENAI_`, `OPENROUTER_`, `GEMINI_`,
+`GOOGLE_`, `XAI_`, `DEEPSEEK_`, `LITELLM_`, or `POLICYENGINE_`, are forwarded
+into the job because launchd does not inherit a shell's environment; endpoint
+overrides (`*_BASE_URL`, `*_API_BASE`) are not, since a Claude Code session
+exports `ANTHROPIC_BASE_URL` for its own proxy. Pass anything else with
+`--env NAME` or `--env NAME=VALUE`, or keep secrets in a mode-600 file and pass
+`--env-file`. The plist itself is written mode 600 under
+`~/Library/LaunchAgents/org.policyengine.policybench.<name>.plist`.
+
+The supervisor's stdout and stderr append to `<run dir>/supervisor.log`
+across relaunches; launchd's own output goes to `<run dir>/launchd.log`.
+`.launchd_restarts` counts consecutive unfinished exits (the wrapper gives up
+after `--max-restarts`, default 5, and writes `.launchd_gave_up`);
+`.launchd_done` marks a finished run.
+
+### What actually kills a run
+
+Investigated on 2026-09-04, when two board runs launched with
+`nohup caffeinate -i policybench run … & disown` from a Claude Code session
+died with `stopped_reason: null` and an empty log:
+
+- A **reboot** (`sysctl kern.boottime`; the Mac restarted at 10:14) killed the
+  GLM-5.3 run. `pmset -g log` showed no sleep, which is what made the deaths
+  look like session restarts. A launchd job with `RunAtLoad` resumes after a
+  reboot; a `nohup` job does not.
+- A **broad pattern kill** from another session,
+  `for p in $(pgrep -f "codex-api|gpt-6-astra"); do kill -9 $p; done`, took
+  out the GPT-6 Astra supervisor and its workers three minutes after launch:
+  the model id is in every worker's command line and is shared with unrelated
+  tooling that runs the same model. Never `pkill -f` a model id. Match the
+  run directory instead (`pkill -f -- "--run-dir $RUN_DIR"`), or use
+  `scripts/launch_run.sh stop`. launchd relaunches a job killed this way.
+- The run's own `pkill -f "policybench run --model glm-5.3"` during a
+  deliberate relaunch.
+
+What does **not** kill a `nohup … & disown` child: the Bash tool call
+returning, the session process exiting, or the session process receiving
+SIGTERM or SIGKILL. All three were tested against the Claude Code binary; the
+harness only signals the process group of a command that is still running,
+timed out, or was aborted, and background tasks (`run_in_background`) of an
+exiting agent. Children of a finished foreground command are not tracked.
+Note that `caffeinate -i cmd` execs `cmd` in the original pid and forks a
+helper, so `pkill -f` on the command line matches the helper too.
+
+### Regression check
+
+`scripts/check_run_survival.sh` (macOS only, no model calls) launches
+`/bin/sleep` through the launcher next to a plain `nohup` control from a
+throwaway process group, kills that whole group with SIGTERM and SIGKILL,
+checks that the control died and the launchd job survived, SIGKILLs the job
+and checks that launchd relaunched it, then stops it through the launcher and
+checks that nothing is left. Run it after touching the launcher or the
+wrapper. `tests/test_launch_run.py` covers the plist rendering, credential
+forwarding, and the wrapper's exit-code policy without launchd, so it runs in
+CI.
+
+To confirm survival across a Claude Code session restart specifically, start
+a run with the launcher from a session, restart or pause that session, and run
+`scripts/launch_run.sh status <name>` from a new one: the launchd pid and the
+`supervisor.log` mtime keep advancing.
+
 ## 1. Pick a Run Directory
 
 Use a dated, descriptive run directory and keep US and UK artifacts under it.

scripts/check_run_survival.sh

@@ -0,0 +1,108 @@
+#!/bin/bash
+# Live regression check (macOS only): a run started by scripts/launch_run.sh
+# outlives the process group that launched it, is relaunched by launchd when
+# killed before finishing, and stops cleanly. This is the reproducible stand-in
+# for "restart the Claude Code session and see whether the supervisor is still
+# there": a session teardown, a terminal closing, and a stray `kill -9` are all
+# signals aimed at a process group or a pid, and that is exactly what this
+# script sends.
+#
+#   scripts/check_run_survival.sh            # prints PASS or FAIL, exit 0/1
+#
+# It launches `/bin/sleep 3600` through the launcher (no model calls, no spend),
+# next to a plain `nohup sleep & disown` control launched from the same
+# throwaway shell, then:
+#   1. SIGTERMs and SIGKILLs that shell's whole process group. The control dies;
+#      the launchd job must survive (it lives in launchd's session, not ours).
+#   2. SIGKILLs the job's sleep. launchd must relaunch it (KeepAlive on an
+#      unfinished exit; --throttle-seconds 5 keeps the wait short).
+#   3. Stops the job through the launcher and checks that nothing is left.
+set -u
+
+SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
+LAUNCHER="$SCRIPT_DIR/launch_run.sh"
+NAME="survival-check-$$"
+LABEL="org.policyengine.policybench.$NAME"
+DOMAIN="gui/$(id -u)"
+WORK=$(mktemp -d "${TMPDIR:-/tmp}/pb-survival.XXXXXX")
+RUN_DIR="$WORK/run"
+fail=0
+
+command -v launchctl >/dev/null 2>&1 || { echo "SKIP: launchctl not found (macOS only)"; exit 0; }
+
+say() { printf '%s\n' "$*"; }
+check() {  # $1 = description, $2 = 0/1 (1 = ok)
+  if [ "$2" -eq 1 ]; then say "  ok   $1"; else say "  FAIL $1"; fail=1; fi
+}
+job_pid() {
+  launchctl print "$DOMAIN/$LABEL" 2>/dev/null | sed -n 's/^[[:space:]]*pid = \([0-9]*\).*/\1/p' | head -1
+}
+sleep_pid_under() {  # $1 = wrapper pid -> pid of the /bin/sleep it runs
+  pgrep -P "$1" -x sleep 2>/dev/null | head -1
+}
+cleanup() {
+  "$LAUNCHER" stop "$NAME" >/dev/null 2>&1
+  pkill -f "^pb-survival-control-$$" 2>/dev/null
+  rm -rf "$WORK"
+}
+trap cleanup EXIT
+
+say "1. launching a dummy run and a nohup control from a throwaway process group"
+# A python-spawned bash in its own session stands in for the shell a Claude Code
+# Bash tool call (or a terminal tab) would use.
+launcher_shell_pgid=$(python3 - "$LAUNCHER" "$NAME" "$RUN_DIR" "$$" <<'PY'
+import subprocess, sys
+launcher, name, run_dir, tag = sys.argv[1:]
+script = f"""
+"{launcher}" start --name "{name}" --run-dir "{run_dir}" --throttle-seconds 5 --no-caffeinate -- /bin/sleep 3600 >"{run_dir}.start.log" 2>&1
+nohup bash -c 'exec -a pb-survival-control-{tag} /bin/sleep 3601' >/dev/null 2>&1 & disown
+sleep 3
+"""
+p = subprocess.Popen(["bash", "-c", script], start_new_session=True)
+print(p.pid)  # session leader: pid == pgid
+p.wait()
+PY
+)
+sleep 1
+control_pid=$(pgrep -f "^pb-survival-control-$$" | head -1)
+wrapper_pid=$(job_pid)
+check "launchd job $LABEL is running (pid ${wrapper_pid:-none})" "$([ -n "$wrapper_pid" ] && echo 1 || echo 0)"
+check "nohup control sleep is running (pid ${control_pid:-none})" "$([ -n "$control_pid" ] && echo 1 || echo 0)"
+if [ -n "$wrapper_pid" ]; then
+  wrapper_pgid=$(ps -o pgid= -p "$wrapper_pid" | tr -d ' ')
+  wrapper_ppid=$(ps -o ppid= -p "$wrapper_pid" | tr -d ' ')
+  check "job runs under launchd (ppid $wrapper_ppid) in its own process group ($wrapper_pgid, launcher shell was $launcher_shell_pgid)" \
+    "$([ "$wrapper_ppid" = "1" ] && [ "$wrapper_pgid" != "$launcher_shell_pgid" ] && echo 1 || echo 0)"
+fi
+
+say "2. killing the launcher shell's whole process group (SIGTERM, then SIGKILL)"
+kill -TERM -- "-$launcher_shell_pgid" 2>/dev/null
+sleep 1
+kill -KILL -- "-$launcher_shell_pgid" 2>/dev/null
+sleep 2
+check "nohup control died with its process group" "$([ -n "$control_pid" ] && ! kill -0 "$control_pid" 2>/dev/null && echo 1 || echo 0)"
+check "launchd job survived the group kill" "$([ -n "$wrapper_pid" ] && kill -0 "$wrapper_pid" 2>/dev/null && echo 1 || echo 0)"
+
+say "3. SIGKILLing the job's sleep; launchd should relaunch it within ~5s"
+sleep_pid=$(sleep_pid_under "$wrapper_pid")
+[ -n "$sleep_pid" ] && kill -KILL "$sleep_pid"
+new_pid=""
+for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
+  sleep 1
+  new_pid=$(job_pid)
+  [ -n "$new_pid" ] && [ "$new_pid" != "$wrapper_pid" ] && break
+done
+check "launchd relaunched the job (new wrapper pid ${new_pid:-none}, old $wrapper_pid)" \
+  "$([ -n "$new_pid" ] && [ "$new_pid" != "$wrapper_pid" ] && echo 1 || echo 0)"
+check "wrapper recorded the unfinished exit in $RUN_DIR/.launchd_restarts" \
+  "$([ "$(cat "$RUN_DIR/.launchd_restarts" 2>/dev/null)" = "1" ] && echo 1 || echo 0)"
+
+say "4. stopping through the launcher"
+"$LAUNCHER" stop "$NAME" >/dev/null 2>&1
+sleep 2
+check "job unloaded" "$(launchctl print "$DOMAIN/$LABEL" >/dev/null 2>&1 && echo 0 || echo 1)"
+check "no sleep left from the job" "$([ -z "$(pgrep -f "^/bin/sleep 3600$")" ] && echo 1 || echo 0)"
+check "plist removed" "$([ ! -f "$HOME/Library/LaunchAgents/$LABEL.plist" ] && echo 1 || echo 0)"
+
+if [ "$fail" -eq 0 ]; then say "PASS: launchd-launched runs survive the launching process group and are relaunched when killed"; else say "FAIL: see the lines above; logs in $RUN_DIR (kept)"; trap - EXIT; "$LAUNCHER" stop "$NAME" >/dev/null 2>&1; fi
+exit "$fail"