Skip to content

server.codeium.com kills streaming connections after ~60s of no bytes (idle timeout); long generations die mid-stream #349

Description

@qian-123456

Summary

server.codeium.com (the endpoint serving swe-2 / devin-agent chat streams) forcibly closes TCP connections that carry no bytes for ~60 seconds. Any model generation with a silent window longer than 60s — long thinking pauses, or large buffered tool-call arguments — is killed mid-stream. Clients see The socket connection was closed unexpectedly and cannot recover, because retrying replays the same long generation.

Reproduction

Raw TLS connection to server.codeium.com:443, no request sent:

handshake ok at 0.6s, holding idle...
recv returned at 60.6s: b''   # server-initiated close
  • Reproduced via direct route and via an HTTP/SOCKS proxy — identical 60.6s, so it's the server/LB, not the path.
  • Control: api.github.com closes idle TLS at ~5.4s; the 60s value matches a typical managed load-balancer idle timeout (e.g. AWS ALB default).

In production traffic (via the omp coding agent, devin provider, model swe-2): 15 stream failures logged, durations [60, 60, 61, 61, 61, 62, 62, 62, 62, 63, 64, 66, 76, 118, 530]s — clustering exactly at the 60s idle window. Successful streams in the same sessions ran up to 1198s, confirming the limit is silence, not total duration.

Impact

Agentic coding clients routinely trigger >60s silent windows: the model thinks for a long stretch, then emits a large tool call (e.g. dispatching several subagents with inlined specs) whose arguments are generated server-side before any bytes stream out. Every such turn dies deterministically, and client retries fail identically — the feature is effectively unusable for large single generations.

Request

Emit periodic heartbeat/keepalive frames on chat streams (e.g. an SSE comment : keepalive or a no-op protobuf message every ~15–30s) so intermediate timeouts never see a fully idle connection. Alternatively, raise the idle timeout for streaming endpoints well above the worst-case generation latency.

Happy to provide packet captures or additional timing data if helpful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions