Preserve real finish_reason for tool calls truncated by length#2184
Open
mlpy0 wants to merge 1 commit into
Open
Preserve real finish_reason for tool calls truncated by length#2184mlpy0 wants to merge 1 commit into
mlpy0 wants to merge 1 commit into
Conversation
A tool call cut off by the model's stop reason (length/stop) before its closing tag was relabeled finish_reason="error", which downstream serializes as an InternalServerError error chunk — turning an out-of-budget completion into a server error. Keep the model's real finish_reason and yield the accumulated partial tool-call text instead, so clients receive standard truncation semantics (finish_reason="length") rather than an error event. Add a regression test for the truncated-tool-call path; the existing failed-parse path (complete tags that fail to parse) still yields an error as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a model stops generating mid-tool-call — e.g. it hits
max_tokensbefore emitting the tool call's closing tag —parse_tool_callsoverwrites the model's realfinish_reason(length/stop) with"error". Downstream,map_responses_to_chunksturns anyfinish_reason == "error"into anErrorChunk, which the chat-completions adapter serializes as{"error": {"type": "InternalServerError", "code": 500}}.So a routine, out-of-budget tool call is reported to clients as an internal server error rather than a truncated completion. Clients that treat that as an upstream/service failure (retrying, marking the endpoint unhealthy, failing over) then escalate a normal truncation into an apparent outage.
Fix
In the "tool call parsing interrupted" branch, keep the model's real
finish_reasonand yield the accumulated partial tool-call text. A truncated tool call now streams back as a standardfinish_reason: "length"completion (partial text as content), so clients get normal OpenAI truncation semantics instead of an error event.The separate failed-parse branch (complete tags that fail to parse into a tool call) is unchanged and still surfaces as an error.
Test
Adds
test_truncated_tool_call_preserves_finish_reasonfor an unclosed tool call cut off by the stop reason; the existingtest_failed_parse_yields_text(genuine parse failure → error) still passes.