Skip to content

Delete written managed tool replies so the reply limit counts only live calls - #363

Merged
ifdotpy merged 1 commit into
mainfrom
fix/retire-written-tool-replies
Oct 20, 2026
Merged

ifdotpy merged 1 commit into
mainfrom
fix/retire-written-tool-replies

Conversation

@ifdotpy

@ifdotpy ifdotpy commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Problem

Live: Studio tools fail on several accounts with "Too many unresolved provider replies" (provider_callbacks/storage.rs save_reply, limit MAX_PENDING=128). Live DB: 128 prepared provider-callback rows on claude-profile-ee033126…, default and profile-1a70…, 89 on claude-profile-d7c1…. A FieldView lead cannot call orchestration tools at all.

Root cause

tools.rs saves every managed tool reply as prepared (save_reply). The only path that sends it is the account worker: account_worker/callback.rs managed_response takes reply.body and drops the identity; the worker then writes the frame with its own account-worker:callback:{scope}:{gen}:{cursor}:submitted/written receipts. Nothing calls provider_callbacks::delivery or retire_checkpointed for that reply (those run only in CallbackRuntime::deliver, which serves question cards). So every tool call left one prepared row with its body, and the 129th call on one account generation failed. The supervisor keeps the generation across backend restarts, so rows never aged out.

Python keeps no reply payload: codex_runtime.py writes the reply directly, and the durable tool result answers any replay.

Fix

  • managed_response links the reply key to its callback cursor (account-worker:callback:…:reply).
  • settle_callback_in_transaction runs only with that cursor's written proof. It now deletes the linked reply row and its body.
  • If the cursor already has written proof when the link is made (a replay), the reply is deleted at once. A replay rebuilds the same answer from the durable tool result. The worker's written receipt and checkpoint stop a second send.
  • retire_settled_tool_replies deletes prepared tool replies whose callback context is no longer the agent's live turn. That means the agent is gone or deleted, or its epoch, account, thread, turn, connection or generation changed. Such a reply can never be sent: replies never go to a replacement connection, and a replay fails the same context check first. It runs:
    • in recover_tool_requests_on_startup, which cleans the leaked rows after install,
    • in save_reply when the account is at the limit, so leaks of any origin heal without a restart.
  • Rows of a live turn, card replies, and submitted/unknown replies stay.

Leaked rows after install

The normal startup recovery deletes them. I extracted the 478 live provider-callback rows and their agent records (read-only), loaded them into a fresh store, and ran recover_tool_requests_on_startup: all 478 were deleted, 0 stayed. None belonged to a live turn. A row from a turn that is still live at install time stays. It is deleted once its write is proven, or after its turn ends (next startup, or the next time the account reaches the limit).

Tests

New tests. Each one fails without the fix (checked by disabling the delete):

  • account_worker::tests::tool_reply_retirement::written_tool_replies_leave_no_outstanding_reply_past_the_account_limit: 129 tool calls through the real worker all succeed, and 0 reply rows or links remain. Without the fix, call 129 fails with "Too many unresolved provider replies".
  • account_worker::tests::tool_reply_retirement::leaked_replies_of_a_settled_turn_do_not_block_a_new_tool_call: 128 leaked rows of an earlier turn; a new call succeeds and the leaked rows are gone.
  • provider_callbacks::recovery::settled_tool_reply_tests::startup_deletes_prepared_tool_replies_of_settled_turns_only: startup deletes settled-turn and missing-agent tool replies. It keeps the live-turn, card and unknown rows, and a repeat pass is a no-op.

Suites (cargo nextest, private target):

  • studio-runtime-services lib plus provider_callbacks, wave7_p04_review, wave7_p04_callback_runtime, claude_accounts, claude_bridge: 1228 passed, 41 failed. All 41 also fail on origin/main in the same environment (auth_hold x5, native_runtime, workspace_ready_git, image_workspace and similar environment failures). There are no new failures.
  • After the last change: account_worker::, provider_callbacks::, managed_host_tools:: and callback_runtime:: lib tests passed 347 of 352. The 5 failures are the known auth_hold x5.
  • cargo clippy -p studio-runtime-services --lib --tests gives no warnings in the changed files.
  • just test-fast was not run locally. Hosted CI runs it.

Residual

  • save_reply checks the limit after the tool has already run. At the limit, the agent sees a failure for an effect that was applied. With this fix the limit counts only live calls, so this needs about 128 calls in flight at once on one account.

…ve calls

The account worker sends every managed tool reply itself and keeps its own
submitted and written receipts per journal cursor. It never moved the
provider callback reply row out of `prepared`, so every tool call left one
row with its body, and after 128 calls on one account generation each new
tool call failed with "Too many unresolved provider replies".

- Link each managed tool reply to its callback cursor. The written proof of
  that cursor deletes the reply row and its body, as Python keeps no reply
  payload after its write. A replay rebuilds the answer from the durable
  tool result; the written receipt and checkpoint stop a second send.
- Startup recovery, and the limit check itself, delete prepared tool
  replies whose callback context is no longer the agent's live turn. Such a
  reply can never be sent, so deleting it sends nothing. Rows of a live turn,
  card replies and submitted or unknown replies stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ifdotpy
ifdotpy merged commit 8e1eeac into main Oct 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant