fix: K8s driver watch stream crashes with FD leak when non-gateway Sandbox resources exist

[rhuss]

Problem Statement

The Kubernetes driver's sandbox watch stream crashes with a file descriptor leak when non-gateway-managed Sandbox resources exist in the namespace. This happens when the Agent Sandbox operator's SandboxWarmPool controller creates Sandbox objects that the OpenShell gateway did not create and does not expect.

The gateway becomes completely unresponsive within minutes and enters a crash loop (13 restarts observed in our test environment).

Root Cause

Two related bugs in the Kubernetes driver:

1. No label selector on the watch stream.

watch_sandboxes() in crates/openshell-driver-kubernetes/src/driver.rs:717 watches ALL Sandbox resources in the namespace:

let mut sandbox_stream =
    watcher::watcher(agent_sandbox_api.api, watcher::Config::default()).boxed();

The watcher::Config::default() has no label selector, so it receives events for every Sandbox in the namespace, including ones created by the SandboxWarmPool controller.

2. Unknown objects escalate to stream failure.

sandbox_id_from_object() (line 867) extracts the sandbox ID from the openshell.ai/sandbox-id label or a sandbox- name prefix. Warm pool sandboxes have neither (they are named openshell-warm-pool-* and carry only agents.x-k8s.io/* labels). The function returns Err("sandbox id not found on object").

In the watch loop (line 741), this error is sent to the channel as a stream error. The compute layer (crates/openshell-server/src/compute/mod.rs:1002) treats any stream error as a watch failure, breaks the loop, sleeps 2 seconds, and reconnects. The reconnection opens a new HTTP/2 connection to the K8s API server. The initial object list on reconnect includes the same warm pool objects, triggering the same error immediately.

This creates a 2-second retry loop. Each iteration opens a new watch connection. The previous connections are not properly closed, accumulating file descriptors. After several hundred iterations (~20-30 minutes), the process exhausts its container file descriptor limit and starts failing with Too many open files (os error 24) on all network operations.

How to Reproduce

1. Deploy OpenShell on an OpenShift/Kubernetes cluster with the Agent Sandbox operator installed (extension CRDs enabled)
2. Deploy OpenShell via the Helm chart or the OpenShift deploy wrapper
3. Create a SandboxTemplate and SandboxWarmPool in the openshell namespace:

apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
  name: test-warm
  namespace: openshell
spec:
  podTemplate:
    spec:
      containers:
      - name: agent
        image: registry.k8s.io/pause:3.10
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: test-warm-pool
  namespace: openshell
spec:
  sandboxTemplateRef:
    name: test-warm
  replicas: 1

4. Observe the gateway logs:

kubectl logs -f openshell-0 -n openshell -c openshell-gateway

5. Within seconds, the logs show the error loop:

WARN openshell_server::compute: Compute driver watch stream errored error=code: 'Internal error', message: "sandbox id not found on object"

After 20-30 minutes, FD exhaustion begins:

ERROR axum::serve::listener: accept error: Too many open files (os error 24)
ERROR kube_client::client::builder: failed with error client error (Connect)

Environment

• OpenShift 4.22.3 (ROSA HCP, us-east-2)
• Red Hat build of Agent Sandbox v0.9.0 (Tech Preview)
• OpenShell gateway latest (Helm chart 0.0.73)
• 3x m5.2xlarge worker nodes

Impact

• Gateway becomes completely unresponsive (cannot accept new connections)
• All sandbox operations fail (create, list, delete)
• Pod enters crash loop (restarts do not help because the warm pool sandboxes persist)
• Any deployment that uses both OpenShell and the Agent Sandbox warm pool extension CRDs is affected

Suggested Fix

1. Add a label selector to the watch stream so it only receives Sandbox objects created by the gateway:

let config = watcher::Config::default()
    .labels(&format!("{}={}", LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE));

The gateway already sets openshell.ai/managed-by=openshell-gateway on every Sandbox it creates (line 857), so this filter is safe.

2. Skip unknown objects instead of propagating errors. Even with the label filter, a defensive change in the watch loop should silently skip objects that fail sandbox_from_object():

Ok(Some(Event::Applied(obj))) => {
    match sandbox_from_object(&namespace, obj) {
        Ok(sandbox) => { /* existing handler */ }
        Err(err) => {
            debug!(%err, name = ?obj.metadata.name, "Skipping non-gateway Sandbox");
            // Do NOT send error to channel
        }
    }
}

The same change should apply to the Event::Deleted and Event::Restarted branches.

Context

This was discovered during a warm pool feasibility study. The gateway itself does not use warm pools yet, but the Agent Sandbox operator's warm pool extension CRDs create Sandbox objects in the same namespace that the gateway watches. This is the expected co-existence pattern for the upcoming warm pool integration (#2157).

[drew]

📋 triage-agent

Triage Assessment

Classification: bug-confirmed

Summary

The Kubernetes driver watches every Sandbox resource, converts an unmanaged object into sandbox id not found on object, and propagates that as a fatal compute-watch error. The two-second reconnect path can retain detached producer tasks and Kubernetes watch streams after their consumer is dropped, making the reported file-descriptor exhaustion credible.

Investigation

watch_sandboxes uses watcher::Config::default() without a selector (crates/openshell-driver-kubernetes/src/driver.rs:710-718). Unknown objects fail ID extraction at lines 867-880, and Applied, Deleted, and Restarted conversion failures are sent downstream at lines 727-787. The compute loop treats any such item as fatal and reconnects after two seconds (crates/openshell-server/src/compute/mod.rs:974-1021).

Each call spawns a detached producer that owns both Kubernetes watchers but has no tx.closed() cancellation branch. After it successfully sends the conversion error, dropping the receiving stream does not stop that task until another channel send occurs. Repeated initial-list failures can therefore retain sockets and FDs.

list_sandboxes has the adjacent unfiltered, fail-the-whole-list behavior at driver.rs:477-503, so reconciliation is also affected. OpenShell-created resources actually use openshell.ai/managed-by=openshell, not openshell-gateway; the issue's constants-based selector remains correct. Warm-pool/adoption issues #2157, #1879, #1447, and #2187 do not duplicate this resilience defect.

Recommendation

Filter both list and watch operations to OpenShell-managed resources, defensively skip unmanaged objects, and cancel producer tasks when the downstream receiver closes. Cover Applied, Deleted, and Restarted foreign objects and add Kubernetes coexistence coverage with an unmanaged or warm-pool-created Sandbox.