Add Connect tunnel retry jitter and request-correlated diagnostics

[jshph]

User-facing story

My local BB stayed healthy on 127.0.0.1:38886, but my public BB at jpham-server.getbb.app went completely offline. The established tunnel closed with WebSocket code 1006 at 18:23:27 UTC. The automatic reconnect one second later received one HTTP 500, and then the Connect plugin made no more attempts. Public routes including /install.sh and /install/bb-app.tgz returned 503 with x-bb-tunnel-offline: 1. About 40 minutes later I reloaded the plugin; it immediately reconnected and every route returned 200 again. From my end, one transient edge failure turned a healthy BB into a persistent outage that required manual intervention.

The direct retry defect behind that experience is already fixed on main by #1784: an installed ws unexpected-response listener owns handshake cleanup, so Connect must drain and terminate the rejected socket and schedule a retry instead of waiting for a close event that may never arrive.

This follow-up is based on the production incident and adds the pieces needed to prevent a reconnect wave and identify the next upstream failure precisely.

What changes

• Apply 20% jitter to the existing capped exponential reconnect backoff, including HTTP 429 and 5xx rejection retries.
• Add a per-process connection attempt ID to client connection, rejection, transport-error, and close logs.
• Record the Worker request ID or Cloudflare Ray ID in lastError when the rejected response provides one.
• Wrap only /__tunnel label resolution, credential hashing, Durable Object binding, and Durable Object dispatch with a request-correlated error boundary.
• Return x-bb-request-id on caught 500 responses and forward it into the Durable Object request.
• Log structured failure stage, label, owner kind/ID, error name/message, and Cloudflare remote, retryable, and overloaded flags.
• Never log request headers or bearer credentials. HTTP 401/403 keep their intentional credential-rejection behavior and do not retry.

Incident assessment

Confirmed:

• The client-side no-retry behavior deterministically explains why a single HTTP 500 persisted until plugin reload.
• /__tunnel has no intentional 500 response. Its expected rejections are 401, 403, 404, or 426, so the observed 500 was an uncaught Worker/D1/Durable Object exception or a Cloudflare-generated internal failure.
• Stale tunnel ownership does not reject a replacement: the Durable Object catches errors while closing tagged predecessors and then accepts the new socket.

Most likely, but not provable without the original Ray ID or Workers observability event: a transient Durable Objects failure. The timing overlaps Cloudflare incidents for Durable Objects and downstream services and elevated Durable Object errors; manual reload succeeded just after the latter moved to monitoring. A fresh D1 lookup failure remains another plausible path.

This PR deliberately does not add a speculative server-side retry. Cloudflare recommends distinguishing retryable and overloaded Durable Object errors and creating a fresh stub before retrying; the new structured boundary supplies the evidence needed to design that separately from the confirmed client recovery behavior. That infrastructure follow-up is tracked in #2065.

Verification

• bb-plugin-connect: 93 tests pass, including 429/500 recovery without close, deterministic jitter/deadline state, late-close deduplication, successful replacement, and 401/403 stop behavior.
• @bb/connect: 106 tests pass, including request-correlated D1 and Durable Object failures and credential-redaction assertions.
• Turbo typechecks pass for both packages.
• git diff --check passes.

No production deployment or credential change is included.

AGENT GENERATED: by GPT-5