Skip to content

fix(client): back off event stream retries to a server that keeps failing - #60523

Open
Hona wants to merge 1 commit into
devfrom
event-stream-backoff
Open

Hona wants to merge 1 commit into
devfrom
event-stream-backoff

Conversation

@Hona

@Hona Hona commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

When a server in the list stops answering, the app retries its event stream every 3 s indefinitely: a 2 s connect timeout plus a 1 s pause. A debug export from a desktop with an offline Tailscale server showed:

  • 1,016 [global-sdk] event stream disconnected lines in 56 min.
  • Each attempt left a CORS preflight queued in Chromium on that origin's six sockets. Aborting the fetch does not cancel it: at peak 139 were queued, each for a median of 271 s until ERR_CONNECTION_TIMED_OUT.
  • Every line read [object Object], so finding which server failed took the netlog.

Fix

  • createClientConnection doubles the pause after each attempt that fails before the stream connects: 1, 2, 4, 8, 16, then 30 s. A stream that stays connected for 1 s resets the pause, as before.
  • resync during the pause ends it. Coming back online, or returning to a window whose stream went quiet, retries at once.
  • The disconnect and request-queue console lines are JSON strings, so a debug export shows the server URL and the error.
flowchart LR
  A["attempt fails before server.connected"] --> B{"resync?<br/>online, page visible"}
  B -- yes --> C["retry now"]
  B -- no --> D["pause min(1 s × 2^(n-1), 30 s)"] --> C
  C --> A
  C -- "connected ≥ 1 s" --> E["attempt = 0"]
Loading
pause = new AbortController()
await wait(
  Math.min(reconnectDelay * 2 ** (attempt - 1), maxReconnectDelay),
  AbortSignal.any([controller.signal, pause.signal]),
)

Unreachable server, 40 min

dev (netlog) This PR (modelled)
Attempts 767 79
Attempts per hour 1,150 119
Peak queued preflights 139 13
Longest wait after it comes back 3 s 32 s, or none after a resync

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none">
  <defs>
    <pattern id="dense" width="6" height="16" patternUnits="userSpaceOnUse">
      <rect width="1" height="16" fill="currentColor" />
    </pattern>
  </defs>
  <rect x="8" y="12" width="226" height="16" fill="url(#dense)" opacity=".35" />
  <g fill="currentColor" opacity=".8">
    <rect x="8" y="44" width="1" height="16" />
    <rect x="14" y="44" width="1" height="16" />
    <rect x="22" y="44" width="1" height="16" />
    <rect x="34" y="44" width="1" height="16" />
    <rect x="54" y="44" width="1" height="16" />
    <rect x="90" y="44" width="1" height="16" />
    <rect x="154" y="44" width="1" height="16" />
    <rect x="218" y="44" width="1" height="16" />
  </g>
  <line x1="8" y1="36" x2="232" y2="36" stroke="currentColor" opacity=".12" />
  <rect x="8" y="8" width="2" height="56" fill="currentColor" opacity=".25">
    <animate attributeName="x" values="8;232" dur="6s" repeatCount="indefinite" />
  </rect>
</svg>

Added by the opencode bot (AI-generated).

Fixes #48014

Related: #53498, #52271, #57807

Copilot AI balanced review requested due to automatic review settings October 30, 2026 07:23
@Hona
Hona requested a review from Brendonovich as a code owner October 30, 2026 07:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no blocking problems. I pointed createClientConnection at an unreachable server for 20 s on both checkouts. The base retried every ~1 s (20 attempts). This PR backed off 1/2/4/8 s (5 attempts), and a resync during the pause retried immediately. solid-connection.test.ts passes, and the new test fails on the base. tsc --noEmit is clean for packages/client and packages/app.

The public API is unchanged: @opencode/client/solid keeps the same exports and options. Only its retry timing changes, and resync() now also does something while the stream is down.

One thing worth a look (inline): the TUI uses the same connection but never calls resync, so it now gets the longer wait with no early retry.

Fixes #48014
Related: #53498, #52271, #57807

continue
}
await wait(reconnectDelay, controller.signal)
pause = new AbortController()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TUI (packages/tui/src/context/client.tsx) uses this loop too, and it has no pageLifecycle or online/visibility events to trigger resync. With an explicit --server and no managed service, a server that is down for a while (for example during an upgrade restart) is now picked up up to ~30 s after it comes back, instead of within ~1 s. Is that intended for the TUI, or should the backoff (or a lower cap) be enabled only where something can call resync?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global event stream reconnects on a fixed 250ms timer with no backoff

2 participants