What happened
A hosted runner (one 4 vCPU / 16 GB machine) refused new sessions with "no free capacity" while only three agents were actually working. Fourteen other sessions had finished hours earlier; their sandboxes were still up, each holding a slot. runnerd starts with -slots (default 16), and every session whose sandbox is up occupies one until it is stopped or deleted, whether or not its child process has exited or anyone is attached. Stopping the idle sessions by hand freed the runner immediately.
Why a fixed count is the wrong admission control
It answers "can this runner take one more session?" badly in both directions: it refuses when the machine is idle, and it would admit a sixteenth active agent onto a machine that saturates at about three. What actually runs out, in order: CPU (~3 active coding agents per 4 vCPU), memory (1–2 GB per active agent), disk (workspaces, the only thing idle sessions consume in quantity), then kernel/daemon limits (netns, fds, container count) far behind.
Proposal to discuss and finalise
The lifecycle vocabulary already has the states: running, suspended_warm, suspended_cold, queued.
- Admission on resources, not a count. Admit while there is memory headroom for one more active sandbox and disk for its workspace; otherwise the session is
queued, not refused.
- Idle sessions suspend themselves. Child exited, or no attachment and no activity for a configurable period →
suspended_warm (container stopped, files kept, resumes on attach in seconds). The definition of "idle" should be the same one the activity signal uses.
- Disk bounds the idle set. When disk is short, the oldest warm sessions go
suspended_cold (snapshotted off the runner). Cold resume is slower; that is the honest trade.
- One hard cap stays as a safety valve, set high, for kernel and daemon limits.
rainier status says what is held: active, warm, cold, and headroom, rather than "no free capacity".
Near-term relief (agreed, small)
- Raise the default the hosted runner startup passes to something large, so the count is a safety valve rather than the effective limit.
- Idle auto-stop in
runnerd: a sandbox whose child has exited and that has had no attachment for N minutes (configurable) is stopped, files kept, exactly what rainier stop does today.
Open questions
- Who owns the policy: the runner (local resource view) or the cell's placement loop (fleet view)? Probably admission locally, cold-migration centrally.
- Idle thresholds for warm and cold, and whether they are per environment.
- How queued sessions are surfaced to the CLI and browser, and whether they time out.
- Interaction with the controller lease: a suspended session has no controller; resume should not auto-claim.
Acceptance for the design
A written contract for the four states and their transitions, the admission rule, the status report, and tests that drive a runner to each limit and observe the right transition rather than a refusal.
What happened
A hosted runner (one 4 vCPU / 16 GB machine) refused new sessions with "no free capacity" while only three agents were actually working. Fourteen other sessions had finished hours earlier; their sandboxes were still up, each holding a slot.
runnerdstarts with-slots(default 16), and every session whose sandbox is up occupies one until it is stopped or deleted, whether or not its child process has exited or anyone is attached. Stopping the idle sessions by hand freed the runner immediately.Why a fixed count is the wrong admission control
It answers "can this runner take one more session?" badly in both directions: it refuses when the machine is idle, and it would admit a sixteenth active agent onto a machine that saturates at about three. What actually runs out, in order: CPU (~3 active coding agents per 4 vCPU), memory (1–2 GB per active agent), disk (workspaces, the only thing idle sessions consume in quantity), then kernel/daemon limits (netns, fds, container count) far behind.
Proposal to discuss and finalise
The lifecycle vocabulary already has the states:
running,suspended_warm,suspended_cold,queued.queued, not refused.suspended_warm(container stopped, files kept, resumes on attach in seconds). The definition of "idle" should be the same one the activity signal uses.suspended_cold(snapshotted off the runner). Cold resume is slower; that is the honest trade.rainier statussays what is held: active, warm, cold, and headroom, rather than "no free capacity".Near-term relief (agreed, small)
runnerd: a sandbox whose child has exited and that has had no attachment for N minutes (configurable) is stopped, files kept, exactly whatrainier stopdoes today.Open questions
Acceptance for the design
A written contract for the four states and their transitions, the admission rule, the status report, and tests that drive a runner to each limit and observe the right transition rather than a refusal.