Merge main into feat/plugins (session transcript, redacted bodies) - #253
Merged
Merged
Conversation
GET /sessions/{id}/transcript turns a session's stored request and
response bodies into the conversation the desktop app shows: per turn,
the messages new in that request, the answer decoded from the stored
response, and what could not be shown (gaps). Control-plane protocol 32.
Clients resend the whole history every turn, so a request's new messages
are what is left after the previous readable request's messages,
compared by fingerprint with cache_control, signatures, key order and
reasoning ignored (some clients drop earlier reasoning from the
history). The first leftover assistant message is the previous turn's
answer and is dropped, unless that answer could not be read in full.
No prefix match, or an unreadable request before it, restarts with the
whole history.
Requests are read from the client's own JSON instead of tw-dialect's IR:
the IR drops exactly what a transcript needs (where a system message
sits, tool-only messages, server-tool blocks, files). Answers reuse
tw-dialect's per-format stream parsers, and the blocks those skip are
recognized on the same frame. Every string is masked with mask_body on
the way out; images carry only their type and size.
300 turns at about 1.2 MB a request build in about 0.7 s (release).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…of bodies by default (#252) * Store bodies with secrets replaced; keep 4 MiB of each answer; 5 GiB of bodies by default Request bodies were stored as the client sent them and masked only when read back, so every credential, resident ID and card number a client sent sat on disk as it was. Bodies are now redacted before they are written, off the forwarding path (BodyRecord::for_disk, run on a blocking thread where twcore hands bodies to the store): - Every value the redaction rules recognize is taken out, whatever the mode, `off` included. Under `enforce` a request is stored in the client's format with the placeholders the upstream received; everything else is masked the way the security log shows it (internal addresses included, which the log prints as they are). Then the whole text goes through tw_secret::mask_body, the same masking the read side applies, for what the rules miss. Read-side masking stays. - Placeholders were numbered per hop in the order of that hop's body, so a converted or re-serialized hop could number the same secret differently from the stored request. guard::look now numbers the client body once and every hop continues that ledger: one value, one placeholder across hops, the stored request and the stored answer. - Ledger::avoiding / Scheme::find_in (tw-guard, additive): numbering skips placeholders already written in the text, and guard::hits drops rule hits that sit on one of ours (the connection-string rule took `app:<<TW_SECRET_2>>@` for a password). Replay and fixture export use it, so a value found again in a stored request never takes a number the stored text already uses and an echo is not restored to the wrong value. - mask_body is idempotent (a 5-character head before `…` and a 4-character tail after it are left alone) and walks bytes instead of collecting a Vec<char> (16 MB for a 4 MB body). Sizes: - The response tap keeps tw_api::BODY_MAX (4 MiB, the store's MAX_ONE is now that constant) plus a 64 KiB margin so a secret across the cut is recognized whole before the store cuts at 4 MiB. The tap grows with the answer and never past its cap. Requests over the window are copied rather than sliced, so a 256 MB request is not kept alive in the queue. - What waits to be written is capped in bytes (QUEUED_MAX, 32 MiB) instead of by count alone: 64 bodies of 4 MiB would be 256 MiB. Each record holds its share until the store accepts it, and the hand-off to the store has a single slot. - Blobs::put_with_len records the original length when it cuts a body itself; a request over 4 MiB used to be stored cut with nothing saying so, and replay sent the half JSON. - retention.body_max_bytes defaults to 5 GiB. tw-store also ran its own hourly gc with hardcoded 7 days, 90 days and 2 GiB next to the one in twcore that reads `retention`, so any setting above the defaults was cut back every hour; that loop and its constants are gone. Consumers: GET /request/{id} shows what was stored (masking it again changes nothing); history search no longer finds text that only occurs inside a secret; replay sends the stored request, with placeholders or masked values instead of secrets; session fingerprints and DeepSeek Harness session_log_bytes still come from the in-memory original. The control-plane protocol and the request store schema are unchanged; old bodies age out within body_days. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(gateway): keep no answer when there is nowhere to store it With the tap now holding up to 4 MiB per answer, an Ending without a body sink (recording did not start) still buffered every answer only to drop it at the end. It now counts the bytes and keeps nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(store): an answer to search is now up to 4 MB, not 256 KB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the CONTROL_API_VERSION note: main took 32 for the session transcript, so the plugin protocol moves to 33. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings main's #251 (session transcript) and #252 (bodies stored with secrets replaced, 4 MiB answers, 5 GiB default) into the plugins integration branch, so the plugin data plane builds on the new per-request placeholder numbering.
The only conflict was the
CONTROL_API_VERSIONnote: main took 32 for the transcript, so the plugin protocol moves to 33.This PR is for CI only. Once it is green,
feat/pluginsis fast-forwarded to the merge commit, so that main's history stays linked. It is not squash-merged.🤖 Generated with Claude Code