fix(spawner): reap terminal task containers instead of waiting for a nonexistent exit#324
Open
dkrattiger wants to merge 1 commit into
Open
fix(spawner): reap terminal task containers instead of waiting for a nonexistent exit#324dkrattiger wants to merge 1 commit into
dkrattiger wants to merge 1 commit into
Conversation
…nonexistent exit A terminal task's container never self-exits — the container liveness loop only stops on SIGTERM/SIGINT, never on task state — so `Spawner.cleanup`'s "wait for it to exit naturally" gate never fired and containers leaked (18 orphans on one host, causing CPU/memory pressure and typing lag). Now when a task is terminal and its container is still running, cleanup actively stops it (`stop` → `docker rm --force` + `tmux kill-session`, idempotent) before releasing the claim and cleaning the workspace. Runs for every task each host pass, so a COMPLETE/DROPPED task is reaped on the next pass; a terminal task whose container is already gone skips the stop (safe no-op) and non-terminal tasks are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dkrattiger
force-pushed
the
panopticon/reap-terminal-containers
branch
from
July 18, 2026 03:46
5895c8f to
fa0179f
Compare
dkrattiger
marked this pull request as ready for review
July 18, 2026 03:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a task reaches a terminal state (COMPLETE/DROPPED), its Docker container is never removed, so containers leak. We found 18 orphans on one host, causing significant CPU/memory pressure and typing lag in live task panes.
Root cause chain:
Spawner.cleanuponly removed the per-task workspace, and only onceself._runner_for(task).is_running(...)already readFalse— it waited for the container to "exit naturally". It never removed the container.while running():incontainer/entrypoint.pyuses_until_signalled(), which flips only on SIGTERM/SIGINT and never checks task state.stop()(LocalRunner.stop→docker rm --force+tmux kill-session) on a terminal transition.So
cleanupwaited forever for an exit that never came; the container ran indefinitely.Fix
Spawner.cleanupnow, when a task is terminal and its container is still running, actively stops+removes it via the runner'sstop()before releasing the claim and cleaning the workspace. It runs for every task on every host pass, so:stop()(docker rm --force+tmux kill-session, idempotent), then release claim, then remove workspace.stop()skipped (safe no-op), workspace still cleaned — unchanged.The
cleanupdocstring is updated — the "wait for it to exit naturally" assumption was wrong.Plan: see the
plan.mdtask artifact.Follow-up (out of scope here)
Container-side self-exit on terminal state ("option 1") is a complementary improvement: have
running()incontainer/entrypoint.pyalso returnFalseonce the task is terminal, and/or run the container withdocker run --rmso it self-removes. It fails differently from the host-side reap (defense-in-depth), but carries real trade-offs — it needs the container to poll task state (new REST traffic in the one LLM-bearing package),docker run --rmwould remove crashed containers too (hidingdocker logsthe crash-loop/heal machinery relies on), and immediate self-exit removes the post-completion grace window for attaching to a finished task's pane. Worth its own design discussion rather than bundling here. The host-side reap in this PR robustly closes the leak on its own — including for wedged containers a self-exit path couldn't reach.