Repository navigation
ReActV2 finish: :submit: keep the thought as the model's message and say it was not sent - #347
Merged
Merged
Conversation
…say it was not sent On the step that follows a thought-only step, the thought now stays the model's own (assistant) message and one user message follows it: "Your message above was not sent. To reply, call `submit` with <the outputs>; to keep working, call a tool." It follows every thought-only step, in later requests and on replay, so each request extends the previous one. The forced submit keeps its shape: its last thought is shown as the request's last user message, with no note after it. Measured on 21 real final steps x 2 samples across 11 model families: the thought shown back as a user message had 5 samples write text again, and some models took it for an echo and submitted nothing; with the note, none wrote text again and replies were submitted intact. Also widens the margins of the deadline tests from #346, which could fail on a loaded machine.
…ced request only when the thought moves A thought the turn's extracted outputs directly follow was answered, so on replay it gets no "Your message above was not sent" line; the extracted outputs are still not rendered. The forced submit drops the line after its last thought only when thought_as_request moves that thought to the user side, so a forced request with new inputs keeps it. Docs that said Imp adds nothing to the last request now name the unsent line as the exception.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User story
A host runs a program with
finish: :submit. After the model writes a thought and calls no tool, #346 continues the loop with an ordinary step. That step currently shows the thought back to the model as the request's last user message, under[[ ## next_thought ## ]].In a host rehearsal, some models misread that message:
They then ended with an empty
submit. That lost 4 of 74 replies that were owed.Change
The continued step's request (only under
finish: :submit):submitand the signature's outputs:answerandconfidence".The line follows every thought-only step:
So each request is still the previous one plus the newest exchange, which is what a provider's prompt cache reuses. The line states a fact about the step (only
submitis sent) and the two ways to go on, so it stays true when the history is replayed.One exception: a thought the turn's extracted outputs directly follow. That thought was answered, so it gets no line on replay. The extracted outputs are still not rendered, because those are words the model did not write.
The forced
submitis unchanged. Probe 4 measured that its shape preserves replies. The forced request's own last thought is still shown as its last user message, with no line after it. Earlier thoughts in the same history keep their line.thought_as_requestactually moves the thought to the user side.max_iters: 0) keeps the line after the held thought, before the inputs' user message.forced_submitkey in the loop's guidance.finish: :autois byte-for-byte unchanged. The line exists only underfinish: :submit.This qualifies an existing doc statement. The docs said Imp adds nothing to the last request except a host's
last_request_note. That is no longer quite true: underfinish: :submit, the earlier thought-only steps in the history keep their unsent line. The moduledoc, the option doc, react.md section 3 and decisions.md now say so.Written tool calls: the step's fields, then the line, and the request ends on the tool listing as every written-mode request does.
Evidence
Measured on 2026-10-16: 21 real final steps, 2 samples each, across 11 model families, after a plain-text step.
tool_choice: auto.The measurement is recorded in a new decisions.md row. The earlier continuation row is amended to point to it. react.md section 2 shows the line.
Empty
submit({}): what the host sawThis is already the behaviour on an ordinary step, the continued one included. A
submitwith missing outputs records the error as that call's result, and the loop goes on. I probed this with a scripted model (submit({}), then a validsubmit): the turn ended:submit, both in a first step and in the continued step.In the forced
submitit is different. That request is the turn's last, so there is no loop to go on. An invalidsubmitthere ends the turn:incomplete. Thetermination_causenames the interruption that led to the forced request (:empty_tool_callsafter two thought-only steps in a row), and the history records the error.So the host's
incompleteturns came from an emptysubmitin the forced request. They did not come from the continued step. This PR does not change that. Two options for review:submitto the extractor, as an answer withoutsubmitalready is;:incomplete.Tests
These fail on main:
test/react_v2_unsent_note_test.exs:submitshows its last thought on the user side, with no line after it; the earlier thought keeps its line.test/react_v2_finish_submit_test.exs: the continued request ends with the thought (assistant), then the line (user).The deadline tests from #346 also get wider timing margins. They could fail on a loaded machine: the deadline sometimes passed before the request meant to be slow had been sent. Each still asserts that no request is sent after the deadline.
mix checkandmix quality.checkpass.