Repository navigation
perf(coding-agent): preconnect to the model while the user types after an idle period - #65
Conversation
…ter an idle period A provider closes an idle keep-alive connection, so the first request after the user has read an answer for a while pays a TCP and TLS handshake before it is sent. In interactive mode, typing a prompt after 10 s without model activity now calls Bun's fetch.preconnect for the current model's HTTPS origin (openai-completions, openai-responses, anthropic-messages): DNS, TCP and TLS only, no request and no credentials. A live pooled connection makes it a no-op. At most once per idle period; not in print or RPC mode; nothing on Node. Against an HTTPS fake model whose new connections cost 150 ms: Enter to the request reaching the server, median of 9 prompts typed after 12 s idle, 179.6 ms -> 13.1 ms. One connection per prompt in both; no request or handshake data beyond TLS before Enter.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61f26f625b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e has fetch.preconnect Tests that call InteractiveMode methods on a partial instance have no warmer; like the idle collector, it is optional and set in the constructor.
…affic resets the idle timer Auth can replace a model's base URL (GitHub Copilot's OAuth credential names its endpoint), and resolving auth may refresh a token over the network. The warmer now preconnects the base URL the model's latest request in this process was sent to (ModelRuntime.lastRequestBaseUrl, recorded after auth resolution), and nothing before a model's first request. Only assistant message events and compaction count as activity; bookkeeping events such as a thinking-level change no longer postpone the preconnect.
Problem
After the user has read an answer for a while, the provider has usually closed the idle keep-alive connection. The next prompt then opens a new connection before anything is sent: a TCP and a TLS handshake, two or three round trips (50-300 ms to a distant endpoint), on the path to the first token.
Fix
In interactive mode, when the user starts typing a prompt after 10 s without model traffic, Pi calls Bun's
fetch.preconnect()for the HTTPS origin the current model's requests go to (openai-completions,openai-responses,anthropic-messages: the APIs whose SDK clients send to the model's base URL with the global fetch). The connection is ready in Bun's keep-alive pool by the time the user presses Enter.Which origin: auth can replace the model's base URL (GitHub Copilot's OAuth credential names its endpoint), and resolving auth may itself refresh a token over the network. So
ModelRuntime.prepareRequestrecords the base URL each model's request was sent to after auth resolution (lastRequestBaseUrl), and the warmer preconnects that one; before a model's first request in the process, nothing is preconnected.What
fetch.preconnectdoes (Bun source,src/runtime/webcore/fetch.rs,src/http/AsyncHTTP.rs,src/http/lib.rs): DNS, TCP and the TLS handshake, then the socket is released to the keep-alive pool (on_preconnect->release_socket). No HTTP request, no headers, no credentials. If the pool already holds a live connection to the origin, it takes and returns that one and opens nothing. An origin the environment routes through a proxy is skipped. POSIX only; Node has nofetch.preconnect, so nothing happens there.Limits:
/commandor a!shellcommand, a plain-HTTP endpoint (local, no TLS to save), other APIs, or while a run or compaction is active.connection-warmer.ts, wired ininteractive-mode.ts); print and RPC modes are unchanged.Numbers (this Mac, built with the 0.9.5 runtime)
HTTPS fake model (
dump_model.pybehind TLS with a local CA viaNODE_EXTRA_CA_CERTS) that waits 150 ms before each new connection's handshake and closes idle connections after 3 s. Prompt typed 12 s after the previous answer (first character, 0.8 s, the rest, Enter). Time from Enter to the request arriving at the server, 3 rounds x 4 prompts, interleaved (current head):Every prompt opened exactly one connection in both builds; with this PR the TLS handshake of prompts 2-4 completed before Enter, and no request arrived before Enter. With a live pooled connection, a preconnect opened no connection (checked with a 5 s server idle timeout).
Tests
packages/coding-agent/test/connection-warmer.test.ts: once per idle period, which events count as model traffic, the texts, models and unknown base URLs it skips, a missing or throwingfetch.preconnect, and that it calls the runtime'sfetch.preconnect.test/suite/model-runtime-request-base-url.test.ts: with an OAuth credential whose auth sets its own base URL,lastRequestBaseUrlis undefined before a request and the credential's URL (not the catalog's) after one.npm run checkclean;./test.shfails only in packages/env (needs the Rust daemon) and in process-spawning tests that time out under load and pass alone.