bug(driver-podman): per-sandbox secrets and volumes are orphaned when a container disappears without DeleteSandbox

[feloy]

Agent Diagnostic

Reproduced live against main at 8cf2673c (2026-07-17, the commit behind release v0.0.86), with the gateway, CLI, Podman driver, and openshell/supervisor:dev image all rebuilt from that commit so no branch code was involved.

• Traced the create path (PodmanComputeDriver::create_sandbox, crates/openshell-driver-podman/src/driver.rs) and the store-side removal paths (apply_deleted_locked, prune_missing_sandbox, crates/openshell-server/src/compute/mod.rs).
• Ran a standalone Podman-backed gateway (mise run gateway), created a sandbox, removed its container out of band, and observed what survived.
• Confirmed no reconciliation exists for either resource family: git grep list_secrets / reconcile_orphaned in crates/openshell-driver-podman/ returns nothing on main.
• Searched open and closed issues for orphaned secret, secret reconciliation, podman secret cleanup, openshell-token, secret leak: no matching issue. The search was full-text, so an issue using different vocabulary could have been missed.
• Corroborating evidence that this happens in normal use, not just under a synthetic trigger: before starting, this machine had four openshell-token-* secrets, ~10 days old, with zero containers present. They were removed to get a clean baseline for the reproduction below.
• Surfaced during review of feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy #2245, which adds a third per-sandbox secret with the same lifecycle. That PR's family is deliberately out of scope here — see the Scope note.

Description

The Podman driver creates two per-sandbox resources before it creates the container, and removes them in exactly two places: the create path's own error cleanup, and delete_sandbox. Neither runs if the container disappears without a DeleteSandbox call — whether because the gateway died mid-create, or the container was removed out of band.

Affected resources on main:

• Workspace volume — created at driver.rs:509, removed only in create-error cleanup (driver.rs:515,528,543,557,564,580) and delete_sandbox (driver.rs:672).
• openshell-token-* secret — created at driver.rs:101, removed only in those same paths and delete_sandbox (driver.rs:681).

When the container goes missing, the server removes its store record through apply_deleted_locked (compute/mod.rs:1381) without ever asking the driver to clean up. Two paths reach it:

• The watch path — the Podman watcher emits a deleted event, apply_deleted fires immediately, no grace period.
• The prune sweep — prune_missing_sandbox (compute/mod.rs:1476-1517) after ORPHAN_GRACE_PERIOD (300s).

apply_deleted_locked calls cleanup_sandbox_owned_records, which deletes only store objects (SSH sessions, sandbox settings). Nothing in either path touches Podman. The record naming the sandbox disappears while its volume and secret persist, leaving a bearer token and a workspace volume with nothing left in the system to attribute them to.

For the token secret this is a leaked credential (CWE-459, missing release of resource after effective lifetime). For the volume it is unbounded disk growth.

Expected: per-sandbox resources whose container no longer exists are reclaimed — either by the driver reconciling against live container IDs, or by the removal path asking the driver to clean up.

Scope note

PR #2245 adds a third family, openshell-proxy-auth-*, with the same lifecycle and a higher-value credential. It is not in scope here because it is not on main yet, but a fix should cover it too rather than be redone later. See #2245 (comment) for the scope discussion.

Reproduction Steps

Verified end to end on main at 8cf2673c.

1. Start a standalone Podman-backed gateway: mise run gateway
2. Create a sandbox: openshell sandbox create --name orphan-repro-main
3. Note the per-sandbox resources: podman secret ls, podman volume ls
4. Remove the container out of band: podman rm -f openshell-sandbox-orphan-repro-main
5. Wait ~10 seconds, then: openshell sandbox list (empty), podman secret ls and podman volume ls (both still present)

Observed timeline, sandbox ID 1eadfcbe-ad00-4ef9-9c95-1f92be936441:

Time (UTC)	Event
16:32:37	CreateSandbox — secret, volume, container created
16:35:38	podman rm -f (out-of-band removal)
16:35:48	Watcher: "Container already removed when inspecting after event, emitting deleted event"
16:35:48	Store record deleted via apply_deleted_locked
16:37:41	Secret and volume still present
16:39:44	Still present, past the 300s grace and ~4 reconcile sweeps; zero cleanup log lines

The same end state is reachable by killing the gateway between secret creation and container creation (driver.rs:509-580), which additionally strands the volume created moments earlier.

Environment

• OS: macOS (Darwin 25.5.0), Apple Silicon; the affected code is platform-independent
• Podman: 5.8.1, rootless, podman machine (libkrun)
• OpenShell: main at 8cf2673c, matching release v0.0.86 — gateway, CLI, driver, and supervisor image all rebuilt from this commit
• Latest release checked: yes — v0.0.86 is latest and contains the affected code
• Possible duplicates checked: yes — none found (see diagnostic for terms and caveat)

Logs

2026-07-19T16:35:48.490839Z  INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=stop
2026-07-19T16:35:48.491922Z  INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=die

$ openshell sandbox list
No sandboxes found.

$ podman secret ls --format '{{.Name}}'
openshell-token-1eadfcbe-ad00-4ef9-9c95-1f92be936441

$ podman volume ls --format '{{.Name}}'
openshell-sandbox-1eadfcbe-ad00-4ef9-9c95-1f92be936441-workspace

[elezar]

One question I would have here is whether this indicates a general driver-agnostic gap in the existing gateway <-> driver API?

[feloy]

One question I would have here is whether this indicates a general driver-agnostic gap in the existing gateway <-> driver API?

Yes — this is a driver-agnostic gap in the gateway↔driver API, not just a Podman-specific bug.

The ComputeDriver gRPC service has no cleanup or reconciliation hook. The only way to release driver-side resources is DeleteSandbox, and the server never calls it when a sandbox disappears out-of-band. apply_deleted_locked and prune_missing_sandbox only clean the store — they never reach back into the driver.

Every driver is affected:

Driver	Orphaned resources	Impact
Podman	Named volume + bearer-token secret	Disk growth + credential leak
VM	State dir (GiB), GPU reservation, subnet, TAP	Disk + hardware resource leak
Docker	Host-side token file	Small file but credential material
Kubernetes	Sandbox CR left behind (reverse orphan)	Low — K8s GC handles downstream pods

A durable fix should live at the API boundary with two complementary pieces:

1. Reactive cleanup — add a CleanupSandboxResources RPC (or make DeleteSandbox idempotent/best-effort) that apply_deleted_locked calls into the driver before dropping the store record.

2. Startup/crash-recovery reconciliation — add a ReconcileResources or ListOwnedResources RPC so the server's periodic sweep can ask the driver "what resource families do you still have?" and prune anything that no longer maps to a known sandbox.

Option 1 alone doesn't cover gateway crashes mid-create (the store record may never have existed), so both are needed. The VM driver's existing cleanup_stale_tap_interfaces at startup is a driver-local version of option 2 — the need has been felt before but solved ad-hoc rather than at the trait level.