Skip to content

fix(spawner): reap terminal task containers instead of waiting for a nonexistent exit#324

Open
dkrattiger wants to merge 1 commit into
mainfrom
panopticon/reap-terminal-containers
Open

fix(spawner): reap terminal task containers instead of waiting for a nonexistent exit#324
dkrattiger wants to merge 1 commit into
mainfrom
panopticon/reap-terminal-containers

Conversation

@dkrattiger

Copy link
Copy Markdown
Contributor

Problem

When a task reaches a terminal state (COMPLETE/DROPPED), its Docker container is never removed, so containers leak. We found 18 orphans on one host, causing significant CPU/memory pressure and typing lag in live task panes.

Root cause chain:

  • Spawner.cleanup only removed the per-task workspace, and only once self._runner_for(task).is_running(...) already read False — it waited for the container to "exit naturally". It never removed the container.
  • But the container never exits on its own: the liveness loop while running(): in container/entrypoint.py uses _until_signalled(), which flips only on SIGTERM/SIGINT and never checks task state.
  • Nothing called the runner's stop() (LocalRunner.stopdocker rm --force + tmux kill-session) on a terminal transition.

So cleanup waited forever for an exit that never came; the container ran indefinitely.

Fix

Spawner.cleanup now, when a task is terminal and its container is still running, actively stops+removes it via the runner's stop() before releasing the claim and cleaning the workspace. It runs for every task on every host pass, so:

  • terminal + runningstop() (docker rm --force + tmux kill-session, idempotent), then release claim, then remove workspace.
  • terminal + already gonestop() skipped (safe no-op), workspace still cleaned — unchanged.
  • non-terminal → untouched (early return): container never touched, workspace never deleted.

The cleanup docstring is updated — the "wait for it to exit naturally" assumption was wrong.

Plan: see the plan.md task artifact.

Follow-up (out of scope here)

Container-side self-exit on terminal state ("option 1") is a complementary improvement: have running() in container/entrypoint.py also return False once the task is terminal, and/or run the container with docker run --rm so it self-removes. It fails differently from the host-side reap (defense-in-depth), but carries real trade-offs — it needs the container to poll task state (new REST traffic in the one LLM-bearing package), docker run --rm would remove crashed containers too (hiding docker logs the crash-loop/heal machinery relies on), and immediate self-exit removes the post-completion grace window for attaching to a finished task's pane. Worth its own design discussion rather than bundling here. The host-side reap in this PR robustly closes the leak on its own — including for wedged containers a self-exit path couldn't reach.

…nonexistent exit

A terminal task's container never self-exits — the container liveness loop
only stops on SIGTERM/SIGINT, never on task state — so `Spawner.cleanup`'s
"wait for it to exit naturally" gate never fired and containers leaked
(18 orphans on one host, causing CPU/memory pressure and typing lag).

Now when a task is terminal and its container is still running, cleanup
actively stops it (`stop` → `docker rm --force` + `tmux kill-session`,
idempotent) before releasing the claim and cleaning the workspace. Runs
for every task each host pass, so a COMPLETE/DROPPED task is reaped on the
next pass; a terminal task whose container is already gone skips the stop
(safe no-op) and non-terminal tasks are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dkrattiger
dkrattiger force-pushed the panopticon/reap-terminal-containers branch from 5895c8f to fa0179f Compare July 18, 2026 03:46
@dkrattiger
dkrattiger marked this pull request as ready for review July 18, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant