Add a launchd launcher for supervised runs and document what kills them

[MaxGhenis]

What

• scripts/launch_run.sh start --name N --model M [--run-dir … --budget-usd … --max-workers …] installs a per-run launchd user agent (org.policyengine.policybench.N) that runs caffeinate -i policybench run … through scripts/policybench_launchd_wrapper.sh. status, logs, stop, list; --dry-run prints the plist.
• The job belongs to launchd, not to the shell, terminal, or Claude Code session that started it. It survives that process group being torn down, resumes after a reboot (RunAtLoad), and is relaunched after an unfinished exit (KeepAlive with SuccessfulExit=false), resuming from run_state.json and the scenario CSVs. A run that finishes or records a stopped_reason exits 0 and unloads itself; the wrapper gives up after --max-restarts consecutive unfinished exits and leaves a .launchd_gave_up marker.
• Provider credentials (*_API_KEY, *_API_TOKEN, OPENAI_*, OPENROUTER_*, …) are forwarded into the job since launchd does not inherit a shell's environment; endpoint overrides (*_BASE_URL, *_API_BASE) are not, because a Claude Code session exports ANTHROPIC_BASE_URL for its own proxy. The plist is written mode 600; --env-file (must be mode 600) keeps secrets out of it.
• docs/runbook.md gains §0 with the procedure, the findings below, and the rule to never pkill -f a model id.

Why (what actually killed the 2026-09-04 runs)

Two board runs launched with nohup caffeinate -i policybench run … & disown from a Claude Code session died with stopped_reason: null and an empty log. It was not session restarts:

Run	Death	Cause
glm-5.3 run2 (07:47)	~10:14	Mac reboot at 10:14:34 (sysctl kern.boottime; pmset -g log only shows sleep)
gpt-6-astra (17:13)	17:16:14	another session's lane cleanup: for p in $(pgrep -f "codex-api|gpt-6-astra"); do kill -9 $p; done matched the supervisor and workers by model id
glm-5.3 run6 (18:18)	18:18:45	the launching session's own pkill -f "policybench run --model glm-5.3" before a relaunch

Tested against the harness binary (2.1.258): children of a finished nohup … & disown command survive the tool call returning (47 min in a live desktop session) and survive the session process's graceful exit, SIGTERM, and SIGKILL. The harness signals only the process group of a running, timed-out, or aborted command and run_in_background tasks of an exiting agent. caffeinate -i cmd execs cmd in the original pid and forks a helper, so pkill -f on the command line matches the helper too.

Verification

• scripts/check_run_survival.sh (macOS, no model calls): launches /bin/sleep through the launcher next to a nohup control from a throwaway process group, kills the group with SIGTERM and SIGKILL (control dies, job survives), SIGKILLs the job (launchd relaunches it, .launchd_restarts = 1), stops it (nothing left). PASS locally.
• tests/test_launch_run.py (11 tests, no launchd): plist rendering, credential forwarding and the base-URL exclusion, command override, env-file permission check, and the wrapper's exit-code policy (complete → 0, recorded stop with rc 1 → 0, unfinished death → 75, give-up after max restarts, no-heartbeat follows rc, env file sourced).
• ruff check ., ruff format --check ., pytest -m "not slow": 769 passed, 5 skipped.
• Dry run from a live session forwarded OPENAI_API_KEY and excluded ANTHROPIC_BASE_URL; the generated plist passes plutil -lint.

No results under results/local/ were touched.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Sep 5, 2026 3:30am UTC