Repository navigation
Delete written managed tool replies so the reply limit counts only live calls - #363
Merged
Merged
Conversation
…ve calls The account worker sends every managed tool reply itself and keeps its own submitted and written receipts per journal cursor. It never moved the provider callback reply row out of `prepared`, so every tool call left one row with its body, and after 128 calls on one account generation each new tool call failed with "Too many unresolved provider replies". - Link each managed tool reply to its callback cursor. The written proof of that cursor deletes the reply row and its body, as Python keeps no reply payload after its write. A replay rebuilds the answer from the durable tool result; the written receipt and checkpoint stop a second send. - Startup recovery, and the limit check itself, delete prepared tool replies whose callback context is no longer the agent's live turn. Such a reply can never be sent, so deleting it sends nothing. Rows of a live turn, card replies and submitted or unknown replies stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Live: Studio tools fail on several accounts with "Too many unresolved provider replies" (
provider_callbacks/storage.rssave_reply, limitMAX_PENDING=128). Live DB: 128preparedprovider-callback rows onclaude-profile-ee033126…,defaultandprofile-1a70…, 89 onclaude-profile-d7c1…. A FieldView lead cannot call orchestration tools at all.Root cause
tools.rssaves every managed tool reply asprepared(save_reply). The only path that sends it is the account worker:account_worker/callback.rsmanaged_responsetakesreply.bodyand drops the identity; the worker then writes the frame with its ownaccount-worker:callback:{scope}:{gen}:{cursor}:submitted/writtenreceipts. Nothing callsprovider_callbacks::deliveryorretire_checkpointedfor that reply (those run only inCallbackRuntime::deliver, which serves question cards). So every tool call left onepreparedrow with its body, and the 129th call on one account generation failed. The supervisor keeps the generation across backend restarts, so rows never aged out.Python keeps no reply payload:
codex_runtime.pywrites the reply directly, and the durable tool result answers any replay.Fix
managed_responselinks the reply key to its callback cursor (account-worker:callback:…:reply).settle_callback_in_transactionruns only with that cursor's written proof. It now deletes the linked reply row and its body.retire_settled_tool_repliesdeletespreparedtool replies whose callback context is no longer the agent's live turn. That means the agent is gone or deleted, or its epoch, account, thread, turn, connection or generation changed. Such a reply can never be sent: replies never go to a replacement connection, and a replay fails the same context check first. It runs:recover_tool_requests_on_startup, which cleans the leaked rows after install,save_replywhen the account is at the limit, so leaks of any origin heal without a restart.submitted/unknownreplies stay.Leaked rows after install
The normal startup recovery deletes them. I extracted the 478 live provider-callback rows and their agent records (read-only), loaded them into a fresh store, and ran
recover_tool_requests_on_startup: all 478 were deleted, 0 stayed. None belonged to a live turn. A row from a turn that is still live at install time stays. It is deleted once its write is proven, or after its turn ends (next startup, or the next time the account reaches the limit).Tests
New tests. Each one fails without the fix (checked by disabling the delete):
account_worker::tests::tool_reply_retirement::written_tool_replies_leave_no_outstanding_reply_past_the_account_limit: 129 tool calls through the real worker all succeed, and 0 reply rows or links remain. Without the fix, call 129 fails with "Too many unresolved provider replies".account_worker::tests::tool_reply_retirement::leaked_replies_of_a_settled_turn_do_not_block_a_new_tool_call: 128 leaked rows of an earlier turn; a new call succeeds and the leaked rows are gone.provider_callbacks::recovery::settled_tool_reply_tests::startup_deletes_prepared_tool_replies_of_settled_turns_only: startup deletes settled-turn and missing-agent tool replies. It keeps the live-turn, card and unknown rows, and a repeat pass is a no-op.Suites (
cargo nextest, private target):studio-runtime-serviceslib plusprovider_callbacks,wave7_p04_review,wave7_p04_callback_runtime,claude_accounts,claude_bridge: 1228 passed, 41 failed. All 41 also fail onorigin/mainin the same environment (auth_holdx5,native_runtime,workspace_ready_git,image_workspaceand similar environment failures). There are no new failures.account_worker::,provider_callbacks::,managed_host_tools::andcallback_runtime::lib tests passed 347 of 352. The 5 failures are the knownauth_holdx5.cargo clippy -p studio-runtime-services --lib --testsgives no warnings in the changed files.just test-fastwas not run locally. Hosted CI runs it.Residual
save_replychecks the limit after the tool has already run. At the limit, the agent sees a failure for an effect that was applied. With this fix the limit counts only live calls, so this needs about 128 calls in flight at once on one account.