Repository navigation
perf(ai): build request bodies from the previous request's bytes - #76
Conversation
Every request resends the whole conversation; the SDK serializes all of it again and fetch copies the string again to encode it. Conversations grow at the end, so each message is now compared with a snapshot of the message at the same index in the conversation's previous request (structure only, strings shared) and reuses that request's UTF-8 bytes when equal; changed or new messages are serialized again. The body goes to fetch as a Blob of the parts. The seam is the SDK client's buildBody (openai, @anthropic-ai/sdk), so the body object is the one the SDK would serialize. Applied in openai-completions, openai-responses, azure-openai-responses and anthropic-messages, only when the client sends with the runtime's own fetch (a custom or wrapping fetch still gets the SDK's string). Anything JSON.stringify treats specially falls back to re-serializing or to the SDK. Bodies byte-identical to 0.9.5 over 1,017 requests (1.8 GB) of completions, anthropic and responses sessions, with cache markers moving, compaction and unique file reads. long_session.py, 10 interleaved sessions, per prompt at 20 / 40 prompts: instructions -18.9% / -31.7%, cycles -10.5% / -18.1%, CPU -13.0% / -21.0%.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75c3920c47
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
JSON.stringify reads an array by index and ignores an overridden Symbol.iterator. The check for items with their own toJSON used for..of, so an onPayload array whose iterator skips such an item bypassed the fallback, and a side-effecting iterator ran during serialization. It now reads by index like the rest of the path.
Problem
Every model request resends the whole conversation. In 0.9.5 each long string's JSON is reused (
json-body.ts), but every request still walks the whole conversation, copies it into a new 9 MB string (at 40 prompts ofbench/long_session.py), and fetch copies that string again to encode it as UTF-8.Microbenchmark, 400 messages / 8.85 MB, POST to a local server, user CPU per request including fetch: plain
JSON.stringify5.6 ms, 0.9.5's cache 2.7-3.3 ms.Fix
Conversations grow at the end.
packages/ai/src/utils/request-body.ts:options.sessionId; the last 4 kept, dropped after 32 requests unused), each message's UTF-8 bytes and a snapshot of its structure (plain objects with keys in order, arrays, primitives; strings shared, not copied).===on strings,Object.ison numbers, for-in key order, prototype and toJSON checks). Equal: reuse the bytes. Otherwise:TextEncoder.encode(JSON.stringify(message)). Snapshots, never the previous objects, so an in-place change (Anthropic'scache_controlmoving to the latest block, compaction, an extension'sonPayload) is seen.JSON.stringify({ ...body, [key]: PLACEHOLDER }), split at the placeholder. JSON.stringify of an array is[+ the items' JSON joined by,+], so the parts concatenate to exactly the SDK's text.Blobof the parts: one copy when the Blob joins them, and Bun's fetch sends the Blob as is. (Uint8Array: fetch copies it again, 1.55 ms; a string rope: fetch flattens and encodes it, 1.7 ms; Blob 0.95 ms, same microbenchmark.)Seam: the SDK client's
buildBody(openai and @anthropic-ai/sdk), wrapped per client. It receives the exact object the SDK would serialize (after Anthropic dropsbetasetc.), so no per-SDK body shapes; the JSON headers come from the SDK itself (a probe with an empty body). Used in openai-completions, openai-responses, azure-openai-responses, anthropic-messages.Falls back to the SDK's serialization (0.9.5 behaviour) when:
options.fetchor a wrapping global fetch may readinit.bodyas a string (Pi's own tests do). On Node the global fetch is JavaScript, so Node is unchanged;toJSON(it would get its index as key), the body or array is not plain, or anything throws.Values JSON.stringify treats specially inside a message (Date, class instances, raw JSON, functions) only make that message re-serialize every time.
Verification
cacheControlFormat: "anthropic", 15 promptspackages/ai/test/request-body.test.ts: awkward values (escapes, lone surrogates, undefined, functions, NaN, -0, Date, class instances, boxed strings, raw JSON,__proto__keys, null prototypes, holes, integer keys); in-place edits, moved markers, key order, compaction; a 300-step random-edit fuzz; interleaved conversations; the fallbacks; json-body's toJSON; and through the real SDKs (openai-completions, openai-responses, anthropic) over 8 growing requests with compaction: the Blob's text equalsJSON.stringify(payload)and the content type isapplication/json; a custom fetch still gets the string. A mutation that makes every object compare equal fails 6 of them.npm run checkclean../test.sh: packages/ai all pass; the only failures are packages/env (needs the Rust daemon), and process-spawning tests that time out under the machine's load and pass when run alone (acp-mode, mcp-mode, codemode worker-lifetime).Numbers (this Mac, load average 40-120: instructions are the steady figure)
bench/long_session.py --prompts 40 --every 20, 10 interleaved sessions per build, medians per prompt:bench/benchmark.py --runs 15: headless 312M -> 306M instructions (-2%), interactive (5 prompts) 790M -> 767M (-3%); wall and CPU within noise.Memory: per kept conversation, the last request's bytes as UTF-8 plus small snapshot objects.