fix(sessionservice): close the spawn/remap race; surface container failures

[dkrattiger]

What

Two related reliability fixes for task-container spawns, both diagnosed live under load (~26 concurrent tasks):

1. The spawn/remap race (tasks failing to start)

docker run --detach returns at PID-1 start — not after the entrypoint's usermod/chown remap (the old comment in local_runner.py claimed otherwise). Under load, the tmux pane's docker exec --user panopticon frequently ran first, resolved the user to the pre-remap uid, and died:

PermissionError: [Errno 13] Permission denied: '/home/panopticon/.claude/commands'

The dead pane took the session with it, heal respawned, the respawn lost the same race, and the churn ran until the crash-loop cap — tasks perpetually "failing to start".

Fix: the entrypoint touches /run/panopticon-ready after the remap, just before exec gosu; the pane command is now a bounded host-shell wait for that marker (150 × 0.2 s, then proceeds so pre-marker images still launch) before exec'ing in. Because the guard is the pane command itself, spawner, heal, dashboard R, and manual respawn-pane are all covered.

2. Container failures were invisible

An OOM-killed container (the burst trigger for the churn above — docker inspect shows OOMKilled=true with a deceptively clean exit 0 and empty logs) read as a bare down, and the heal crash-loop cap only logged to the runner session.

Fix:

• LocalRunner.exit_reason — a docker inspect probe of a stopped container (OOMKilled checked before the exit code).
• Spawner.reconcile reports failed + that reason (e.g. container OOM-killed (exit 0)) instead of clearing to an unexplained down, including for a task that went down after being live.
• The heal cap reports failed — keeps losing its tmux session (5 respawns) — press R to respawn, once, instead of an error log every pass.

The dashboard's detail pane already renders container: failed — <detail>; it just never had these details to show.

Testing

• 718 tests pass (strict mypy included).
• New unit tests pin the docker inspect probe, OOM-over-exit-code precedence, reconcile surfacing (in-flight, down, and the no-reason/no-churn case), the report-once cap, and the new pane command (wait loop + bounded fallback + exec).
• Real-container integration test (test_spawn_and_stop_real_container_and_session): with the rebuilt image the pane execs as soon as the marker lands (~11 s total, vs. burning the 30 s fallback against a pre-marker image).
• Deployed live: base image rebuilt, runner restarted; previously-stuck tasks spawn cleanly.

Notes

• Needs make build on hosts after merge (entrypoint change ships in the base image; composed images rebuild automatically via the changed FROM layer).
• Task 184c5f5f was filed for this fix but couldn't get off the ground (it kept being killed by the bug it was fixing). Its remaining un-duplicated scope — --continue fallback, sticky crash-loop cap, bootstrap-exception lifecycle reporting — can be re-scoped or dropped.

🤖 Generated with Claude Code