fix(redis): size cold-connection waits to survive a dead handshake (#7764)

* fix(redis): size cold-connection waits to survive a dead handshake

A dead Redis handshake costs two command deadlines, not one. ioredis sends
CLIENT SETNAME/SETINFO on connect and dispatches the INFO ready check only
once those settle — on a socket that never answers, they settle by timing
out. Only then does the ready check start its own deadline, fail, and tear the
socket down for retryStrategy to reconnect. Every readiness wait in the
codebase was sized to a single deadline, so it expired while the first attempt
was still being diagnosed and a configured retry could never be the thing that
rescued it.

Introduces `coldConnectionBudgetMs`, which derives a wait from the command
deadline and the reconnect delays a caller's own retryStrategy would return,
and states the 2x in one place. Both waits now derive from it:

- The execution-signal subscriber's readiness budget was the same constant as
  its commandTimeout (5s and 5s), so ioredis's reconnect was decorative on the
  cold path that failed in production. It now has a 2s command deadline — this
  client only issues SUBSCRIBE/UNSUBSCRIBE, always after ready — and a budget
  with room for two dead attempts and a healthy one.
- The shared client's warm-up budget was 10s against a 5s deadline, exactly
  the moment ioredis would first tear a stalled socket down, so on the case
  warming exists for it gave up just before recovery could land.

Warms the execution-signal subscriber alongside the shared client at process
start, in parallel, never throwing: it is the second connection a run opens
and was paying its handshake inside its own readiness budget.

The shared redis-config mock mirrors the pure budget helper, since consumers
now evaluate it at module load and would otherwise fail to import under the
global mock.

* fix(redis): keep the subscriber's live command deadline at 5s

ioredis has one `commandTimeout` for the handshake commands and every live
command on the socket, so tightening it to diagnose dead handshakes faster also
cut the initial SUBSCRIBE from 5s to 2s — on the one path where a rejection
fails the whole run. A ready-but-slow server delays SUBSCRIBE too.

Restore the 5s tolerance that subscribe has always had and derive the
readiness budget from it: one dead handshake recovers in ~10.5s, so an 11.5s
budget still lets ioredis's reconnect rescue the case that failed. The local
test mock now delegates to the shared, drift-guarded mirror of the budget
arithmetic so the derived number under test is the one production derives.

* improvement(redis): warm the signal subscriber on intent, not at worker start

Only the tasks that execute a workflow ever subscribe to cancellation signals,
and they are a minority of runs: over a week, 71% of Trigger.dev runs never
subscribe, and a single document-processing task is 61% on its own. Warming
the subscriber in the global init hook opened a TLS connection on every one
of those runs that nothing would use.

Warm it instead at `executeWorkflowCore` — the one path every execution
shares — fire-and-forget as the first statement, so the handshake overlaps the
custom-block read and preprocessing ahead of the cancellation subscribe rather
than being paid inside that subscribe's readiness budget. No task-id list to
keep in sync: coverage follows from the funnel. The subscribe and the warm-up
share the hub's memoized readiness wait, so their budgets never compound
within a run.

The global init keeps warming only the shared client, which nearly every task
uses; the Next server keeps warming both at boot, one connection per
long-lived process.

* chore(instrumentation): import the warm-ups through the path alias

The dynamic imports in the Redis warm-up block used relative paths; the file
already resolves `@/` and the repository rule is absolute imports.

* fix(redis): give every readiness waiter its own deadline and correct the budget model

Three corrections from an independent review of the branch.

The shared readiness wait carried one timer, so a subscribe that joined an
in-flight warm-up inherited only the remainder of that warm-up's budget — down
to nothing — where before this branch it was guaranteed a full wait of its own.
Waiters now share one ready/end listener pair but each runs its own deadline,
and the signal is torn down when its last waiter gives up.

The budget model claimed a dead handshake always costs two command deadlines.
That is true only without a password: with one, ioredis treats a timed-out
AUTH as fatal and tears the socket down after a single deadline, and a connect
that never completes is bounded by connectTimeout, which the formula had no
term for. `coldConnectionBudgetMs` now charges the largest of those so it holds
for either URL shape, and moves to @sim/utils/retry: a pure helper evaluated at
module load must not live in a module the test setup replaces wholesale, which
had forced the shared mock to carry a verbatim mirror of the arithmetic. The
subscriber derives its budget in its constructor from the exact options it is
built with; the shared client's reconnect delay is a pure function used by
both retryStrategy and the warm-up budget.

The Next server no longer opens the signal subscriber at boot: the execution
entry point already warms it on intent, and an eagerly opened subscriber that
cannot reach Redis would reconnect on a five-second cadence in every idle
replica for the life of the process.

* fix(redis): leave the readiness wait exactly once per waiter

A waiter that timed out was still subscribed to the shared readiness signal,
so when that signal later settled for another waiter its cleanup ran a second
time. The waiter count drifted negative, the last-waiter teardown could never
match again, and the settled signal stayed memoized — a later subscribe during
a reconnect would have observed a readiness that had already passed.

Each waiter now leaves exactly once, and a settling signal clears itself from
the memo so the next waiter observes the connection afresh regardless of the
count.

* test(redis): assert the fresh subscribe by channel, not by call count

A reconnect also re-subscribes surviving channels, so the new subscription is
identified by its channel rather than by SUBSCRIBE having been issued once.

* test(redis): prove waiter accounting by the teardown a timed-out waiter must perform

The settle path clears the memo on its own, so a drifted waiter count only
shows when a later lone waiter times out and fails to tear the signal down.
Exercise that path directly; the test now fails without the per-waiter guard.

* test(execution-core): type the warm-up test's fixtures instead of casting to any

The partial fixtures are widened through unknown to the real ExecutionSnapshot
and LoggingSession types, as the repository's TypeScript rule prescribes.

* fix(redis): connect the signal subscriber on intent and complete the budget model

Findings from an independent review of the finished branch.

Constructing the execution-signal hub is what connects — ioredis dials in its
constructor — so the warm-up that observed that handshake and returned a
boolean its only caller discarded is gone. The execution entry point now
constructs the hub, synchronously and never throwing, and the subscribe that
follows waits on its own budget. With no observer to accommodate, the waiter
machinery collapses to a race between the shared readiness signal and a
per-waiter deadline, with a per-signal count so the last waiter detaches.

The budget's dead-attempt term was still short for an unanswered handshake:
after the ready check times out, ioredis half-closes the socket and destroys
it only after `disconnectTimeout` when a wedged peer never answers with a FIN.
Measured against such a peer: an unauthenticated connection reconnects at
12.5s, past the 11.5s the budget allowed. The term is now
`max(connectTimeout, 2 * commandTimeout + disconnectTimeout)`, with the
disconnect deadline stated in the shared connection defaults so the budget
derives from it. The model is specific to ioredis 5's handshake sequence, so it
lives beside the pinned client in `lib/core/config/redis-budget.ts` — a module
the global test mock does not replace — rather than in a generic package.

Corrects two doc claims: no caller warms the subscriber at server boot, and a
late joiner never had a full wait of its own before this branch either — the
memoized promise shared its single timer.

Author: waleedlatif1

apps/sim/instrumentation-node.ts

@@ -407,6 +407,6 @@ export async function register() {
   // Not awaited: the connection is warmed in the background so the first request
   // that needs Redis does not pay the handshake inside its own command deadline,
   // but boot never waits on Redis to serve requests that do not touch it.
-  const { warmRedisConnection } = await import('./lib/core/config/redis')
+  const { warmRedisConnection } = await import('@/lib/core/config/redis')
   void warmRedisConnection()
 }

apps/sim/lib/core/config/redis-budget.test.ts

@@ -0,0 +1,44 @@
+/**
+ * @vitest-environment node
+ */
+import { describe, expect, it } from 'vitest'
+import { coldConnectionBudgetMs } from '@/lib/core/config/redis-budget'
+
+describe('coldConnectionBudgetMs', () => {
+  it('charges an unanswered handshake at two command deadlines plus the half-close wait', () => {
+    // Unauthenticated: SETNAME/SETINFO time out, then INFO times out, then the
+    // half-closed socket waits for a FIN a wedged peer never sends.
+    expect(
+      coldConnectionBudgetMs({
+        connectTimeoutMs: 1_000,
+        commandTimeoutMs: 5_000,
+        disconnectTimeoutMs: 2_000,
+        reconnectDelayMs: 500,
+      })
+    ).toBe(2 * 5_000 + 2_000 + 500 + 1_000)
+  })
+
+  it('charges the connect deadline when a connection that never completes is the longer case', () => {
+    // Lowering the command deadline must not shrink the budget below what a
+    // connect that never completes costs before retryStrategy can fire.
+    expect(
+      coldConnectionBudgetMs({
+        connectTimeoutMs: 10_000,
+        commandTimeoutMs: 3_000,
+        disconnectTimeoutMs: 2_000,
+        reconnectDelayMs: 500,
+      })
+    ).toBe(10_000 + 500 + 1_000)
+  })
+
+  it('always leaves room for the healthy attempt that follows the reconnect', () => {
+    expect(
+      coldConnectionBudgetMs({
+        connectTimeoutMs: 0,
+        commandTimeoutMs: 0,
+        disconnectTimeoutMs: 0,
+        reconnectDelayMs: 0,
+      })
+    ).toBe(1_000)
+  })
+})

apps/sim/lib/core/config/redis-budget.ts

@@ -0,0 +1,45 @@
+/** Generous room for the healthy handshake that follows a recovered stall; a real one takes tens of milliseconds. */
+const HEALTHY_HANDSHAKE_ALLOWANCE_MS = 1_000
+
+export interface ColdConnectionBudgetOptions {
+  /** The client's TCP connect deadline: bounds a connection that never completes at all. */
+  connectTimeoutMs: number
+  /** The client's per-command deadline: bounds a handshake command the server never answers. */
+  commandTimeoutMs: number
+  /** How long ioredis waits after half-closing a dead socket for the peer's FIN before destroying it. */
+  disconnectTimeoutMs: number
+  /** What the client's `retryStrategy` returns for its first reconnect. */
+  reconnectDelayMs: number
+}
+
+/**
+ * How long a wait for a cold ioredis connection must allow before giving up,
+ * if it is to survive one dead attempt and still see a healthy one land.
+ *
+ * A dead attempt is diagnosed by whichever deadline governs the phase it
+ * stalls in. A connect that never completes costs `connectTimeoutMs`, and the
+ * socket is destroyed outright. A connection that opens but whose handshake
+ * is never answered costs command deadlines — one with a password, because a
+ * timed-out `AUTH` is fatal; two without, because `CLIENT SETNAME`/`SETINFO`
+ * must settle, by timing out, before the `INFO` ready check starts its own —
+ * and then `disconnectTimeoutMs` more, because ioredis half-closes the socket
+ * and a peer that is wedged never answers with a FIN. The budget takes the
+ * larger phase so it holds for either URL shape without parsing it. Only then
+ * does `retryStrategy` run and a fresh attempt begin.
+ *
+ * A wait sized to a single deadline expires while the first attempt is still
+ * being diagnosed, so a configured retry can never be the thing that saves it.
+ *
+ * The guarantee is one dead attempt from a fresh or previously-ready client.
+ * ioredis feeds `retryStrategy` its running attempt count, so a wait that
+ * begins while the client is already deep in a reconnect loop faces larger
+ * delays this does not model. The handshake sequence is ioredis 5's; keep this
+ * beside the pinned client, not in a generic package.
+ */
+export function coldConnectionBudgetMs(options: ColdConnectionBudgetOptions): number {
+  const deadAttemptMs = Math.max(
+    options.connectTimeoutMs,
+    2 * options.commandTimeoutMs + options.disconnectTimeoutMs
+  )
+  return deadAttemptMs + options.reconnectDelayMs + HEALTHY_HANDSHAKE_ALLOWANCE_MS
+}

apps/sim/lib/core/config/redis.test.ts

@@ -47,14 +47,19 @@ vi.mock('ioredis', () => ({
 
 import {
   acquireLock,
+  CONNECT_TIMEOUT_MS,
   closeRedisConnection,
+  DISCONNECT_TIMEOUT_MS,
   describeRedisConnection,
   extendLock,
   getRedisClient,
   onRedisReconnect,
   resetForTesting,
+  SHARED_COMMAND_TIMEOUT_MS,
+  sharedReconnectDelayMs,
   warmRedisConnection,
 } from '@/lib/core/config/redis'
+import { coldConnectionBudgetMs } from '@/lib/core/config/redis-budget'
 
 describe('redis config', () => {
   beforeEach(() => {
@@ -471,7 +476,50 @@ describe('redis config', () => {
     })
   })
 
+  describe('sharedReconnectDelayMs', () => {
+    it('grows exponentially from the base and caps, with upward-only jitter', () => {
+      expect(sharedReconnectDelayMs(1, 0)).toBe(1_000)
+      expect(sharedReconnectDelayMs(2, 0)).toBe(2_000)
+      expect(sharedReconnectDelayMs(5, 0)).toBe(10_000)
+      expect(sharedReconnectDelayMs(6, 0)).toBe(10_000)
+      expect(sharedReconnectDelayMs(1, 1)).toBe(1_300)
+    })
+  })
+
   describe('warmRedisConnection', () => {
+    it('outlasts one dead handshake so the reconnect can be what warms it', async () => {
+      mockRedisInstance.status = 'connecting'
+      const warm = warmRedisConnection()
+
+      // The dead attempt's own diagnosis and half-close, then the longest first
+      // reconnect delay: the moment a healthy second attempt can begin.
+      await vi.advanceTimersByTimeAsync(
+        Math.max(CONNECT_TIMEOUT_MS, 2 * SHARED_COMMAND_TIMEOUT_MS + DISCONNECT_TIMEOUT_MS) +
+          sharedReconnectDelayMs(1, 1)
+      )
+      const client = getRedisClient()
+      Object.assign(client ?? {}, { status: 'ready' })
+      client?.emit('ready')
+
+      await expect(warm).resolves.toBe(true)
+    })
+
+    it('still gives up once the budget is spent', async () => {
+      mockRedisInstance.status = 'connecting'
+      const warm = warmRedisConnection()
+
+      await vi.advanceTimersByTimeAsync(
+        coldConnectionBudgetMs({
+          connectTimeoutMs: CONNECT_TIMEOUT_MS,
+          commandTimeoutMs: SHARED_COMMAND_TIMEOUT_MS,
+          disconnectTimeoutMs: DISCONNECT_TIMEOUT_MS,
+          reconnectDelayMs: sharedReconnectDelayMs(1, 1),
+        })
+      )
+
+      await expect(warm).resolves.toBe(false)
+    })
+
     it('resolves immediately when the connection is already usable', async () => {
       mockRedisInstance.status = 'ready'
 

apps/sim/lib/core/config/redis.ts

@@ -2,9 +2,10 @@ import { isIP } from 'node:net'
 import { createLogger } from '@sim/logger'
 import { toError } from '@sim/utils/errors'
 import { randomFloat } from '@sim/utils/random'
-import Redis, { type RedisOptions } from 'ioredis'
+import Redis from 'ioredis'
 import { env } from '@/lib/core/config/env'
 import { getConfiguredCacheProvider } from '@/lib/core/config/env-capabilities.server'
+import { coldConnectionBudgetMs } from '@/lib/core/config/redis-budget'
 
 const logger = createLogger('Redis')
 
@@ -42,13 +43,24 @@ function resolveRedisTlsOptions(url: string | undefined): { servername: string }
  * and TLS SNI when REDIS_URL targets an IP. Every Redis client we open should
  * spread this; callers add their own retry / timeout policy on top.
  */
-export function getRedisConnectionDefaults(
-  url: string | undefined
-): Pick<RedisOptions, 'keepAlive' | 'connectTimeout' | 'enableOfflineQueue' | 'tls'> {
+export interface RedisConnectionDefaults {
+  keepAlive: number
+  connectTimeout: number
+  disconnectTimeout: number
+  enableOfflineQueue: boolean
+  tls?: { servername: string }
+}
+
+export const CONNECT_TIMEOUT_MS = 10_000
+/** ioredis's own default, stated so readiness budgets can be derived from it. */
+export const DISCONNECT_TIMEOUT_MS = 2_000
+
+export function getRedisConnectionDefaults(url: string | undefined): RedisConnectionDefaults {
   const tls = resolveRedisTlsOptions(url)
   return {
     keepAlive: 1000,
-    connectTimeout: 10000,
+    connectTimeout: CONNECT_TIMEOUT_MS,
+    disconnectTimeout: DISCONNECT_TIMEOUT_MS,
     enableOfflineQueue: true,
     ...(tls ? { tls } : {}),
   }
@@ -203,12 +215,34 @@ export function describeRedisConnection(
 
 const PING_INTERVAL_MS = 15_000
 const MAX_PING_FAILURES = 2
+export const SHARED_COMMAND_TIMEOUT_MS = 5_000
+const RECONNECT_BASE_MS = 1_000
+const RECONNECT_MAX_BASE_MS = 10_000
+const RECONNECT_JITTER_RATIO = 0.3
+
+/**
+ * The shared client's reconnect delay for attempt `times`, with `jitter` in
+ * `[0, 1]` scaling the upward-only jitter band. Pure so the same formula can
+ * be evaluated for a budget — `sharedReconnectDelayMs(1, 1)` is the longest
+ * possible first reconnect — as well as from `retryStrategy`, which adds the
+ * bookkeeping around it.
+ */
+export function sharedReconnectDelayMs(times: number, jitter: number): number {
+  const base = Math.min(RECONNECT_BASE_MS * 2 ** (times - 1), RECONNECT_MAX_BASE_MS)
+  return Math.round(base + jitter * base * RECONNECT_JITTER_RATIO)
+}
+
 /**
- * Warm-up budget. Sized to outlast a slow handshake rather than a fast one,
- * because giving up early just returns the handshake to the first command's
- * deadline, which is the thing this exists to avoid.
+ * Warm-up budget: one dead attempt, its longest possible first reconnect
+ * delay, then a healthy attempt. Giving up sooner returns the handshake to the
+ * first command's deadline, which is the thing warming exists to avoid.
  */
-const REDIS_WARMUP_TIMEOUT_MS = 10_000
+const REDIS_WARMUP_TIMEOUT_MS = coldConnectionBudgetMs({
+  connectTimeoutMs: CONNECT_TIMEOUT_MS,
+  commandTimeoutMs: SHARED_COMMAND_TIMEOUT_MS,
+  disconnectTimeoutMs: DISCONNECT_TIMEOUT_MS,
+  reconnectDelayMs: sharedReconnectDelayMs(1, 1),
+})
 
 export function getConfiguredRedisUrl(): string | null {
   if (getConfiguredCacheProvider() === 'database') return null
@@ -296,17 +330,15 @@ export function getRedisClient(): Redis | null {
 
     state.client = new Redis(redisUrl, {
       ...defaults,
-      commandTimeout: 5000,
+      commandTimeout: SHARED_COMMAND_TIMEOUT_MS,
       maxRetriesPerRequest: 5,
 
       retryStrategy: (times) => {
         if (times > 10) {
           logger.error(`Redis reconnection attempt ${times}`, { nextRetryMs: 30000 })
           return 30000
         }
-        const base = Math.min(1000 * 2 ** (times - 1), 10000)
-        const jitter = randomFloat() * base * 0.3
-        const delay = Math.round(base + jitter)
+        const delay = sharedReconnectDelayMs(times, randomFloat())
         state.reconnects++
         logger.warn('Redis reconnecting', { attempt: times, nextRetryMs: delay })
         return delay