From 18456c161c51293a9537adb312970f83f2782ce7 Mon Sep 17 00:00:00 2001 From: deepfates Date: Thu, 15 Oct 2026 21:55:18 -0700 Subject: [PATCH 1/2] ReActV2 finish: :submit: a thought-only step continues the loop before submit is forced Under finish: :submit a step that writes a thought and calls no tool is followed by an ordinary step (every tool, tool_choice auto, the thought as the request's last user message). A second thought-only step in a row, or one that says nothing, is the :empty_tool_calls interruption and forces submit; max_iters and the deadline still do. Measured on 42 real final steps from 11 models: most submitted the thought's reply unforced, Meta's models included; some went on with the announced work; 5 of 42 wrote text again. The forced submit is not sent once the deadline has passed, as the last text request already was not. An :extracted turn's history now ends with the extracted outputs, recorded as a step that called no tool. --- CHANGELOG.md | 15 +++ decisions.md | 1 + docs/diving-deeper/react.md | 48 +++++--- lib/imp/predict/react_v2.ex | 137 +++++++++++++++++++---- test/react_v2_finish_submit_test.exs | 24 ++-- test/react_v2_thought_continues_test.exs | 114 +++++++++++++++++++ 6 files changed, 294 insertions(+), 45 deletions(-) create mode 100644 test/react_v2_thought_continues_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index 66abaefc..b67dfd1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,21 @@ User-visible changes to Imp are recorded here. ### Changed +- Under ReActV2's `finish: :submit`, a step that writes a thought and calls + no tool is now followed by an ordinary step (every tool, + `tool_choice: "auto"`, the thought shown as the request's last message) + instead of the forced `submit`. A second thought-only step in a row, or a + step that says nothing, still forces `submit` (`:empty_tool_calls`), as do + `max_iters` and the deadline. A model that announced an action can take + it, and a reply in the thought is usually submitted unforced, also on + providers that cannot force a tool. Other signatures with `submit` are + unchanged. +- ReActV2's forced `submit` is not sent once the process's `Imp.Deadline` + has passed; the turn is `:incomplete` with `termination_cause: + :deadline_exceeded`, as the last text request already was. +- An `:extracted` ReActV2 turn's history now ends with the extracted + outputs, recorded as a step that called no tool (with any inputs no step + spent), so the history holds the turn's answer. - A ReActV2 forced `submit` request no longer ends on the model's own message. When the step before it wrote a thought and called nothing, the thought was the request's last message, as an assistant turn, and providers refuse that: diff --git a/decisions.md b/decisions.md index 366ed150..33fe2b67 100644 --- a/decisions.md +++ b/decisions.md @@ -8,6 +8,7 @@ not necessarily when it was made. | Date | Decision | Source and reason | Status | Retires when | | --- | --- | --- | --- | --- | +| 2026-10-16 | Under `finish: :submit`, a ReActV2 step that writes a thought and calls no tool is followed by an ordinary step (every tool, `tool_choice: "auto"`, the thought as the request's last user message), not at once by the forced `submit`; a second thought-only step in a row, or a step that says nothing, is the `:empty_tool_calls` interruption. `max_iters` and the deadline end the turn with the forced `submit` as before. Signatures with `submit` under `finish: :auto` keep DSPy's ending. | `lib/imp/predict/react_v2.ex` `continue_after_thought?/3`, `test/react_v2_thought_continues_test.exs`. Forcing at once took every other tool away from a model that had just announced its next action, and added a request whose failure lost the reply the thought held. Measured 2026-10-16 on 42 real final steps from 11 models through OpenRouter, sending that ordinary step: most samples called `submit` with the thought's reply, copied or paraphrased (Meta's models 6 of 6, unforced; Kimi, GPT and Gemini models likewise); a thought that was a closing note was submitted as an empty answer; a DeepSeek model went on to fetch what it had said it would read; 5 of 42 wrote text again. DSPy's ReActV2 forces `submit` after any step that calls no tool. | In force. | A measurement shows a different bound, or forcing at once, loses fewer answers. | | 2026-10-16 | ReActV2's forced `submit` request, when the step before it wrote a thought and called nothing, shows that thought as the request's last message, a user message under its field's marker (`[[ ## next_thought ## ]]`), not as the assistant turn the history records it as. DSPy's ReActV2 leaves the thought out of the forced request. | `lib/imp/adapter/chat.ex` `thought_as_request/4`, `test/react_v2_finish_submit_test.exs`. As a trailing assistant turn the request is refused by providers that do not take an assistant prefix: Mistral ("Cannot set an assistant prefix with function calling") and Gemini ("Requests ending with a model turn are not supported"), probed through OpenRouter 2026-10-16. Dropping it, as DSPy does, made models regenerate finished answers with reasoning off, 2 of 16 to the output limit (deepfates/imp#336). Measured 2026-10-16 on real final steps re-sent through OpenRouter: with the thought as this user message, the submitted answer kept the plain-text reply (text similarity 0.8 to 1.0 across Kimi, GPT, DeepSeek and Muse models), a model whose plain text was a closing note submitted an empty answer as it intended, Meta's models (which accept only `tool_choice: auto`) copied the reply into `submit` 3 of 3, and Gemini accepted every request; with the thought dropped, the models wrote a different answer (similarity about 0.05), one shrank to 30 characters, and the closing-note model submitted a 170-character answer. | In force. | A measurement shows the thought is better dropped, or DSPy keeps it in a form providers accept. | | 2026-10-12 | In that plain-text step (row of 2026-10-07; a program with `finish: :auto`, the default), the text output is not presented as a field: the system message lists no output fields, the default agent line names no output ("You are an Agent. Use the supplied tools to gather information and take actions."), there is no "The outputs to produce are:" list, and the finish line is "When the final answer is ready, write it as plain text without calling a tool." followed by the field's description, once. Amended 2026-10-15: the finish line was "... reply without calling a tool."; "reply" is a common tool name, so for a program holding a `reply` tool it read as an instruction about that tool, and it now uses the sentence the other tool-less text answer already used. A host's `agent_line` still replaces the agent line. | `lib/imp/adapter/chat.ex` `default_system/2`, `with_guidance/5`; `test/react_v2_plain_text_answer_test.exs`. The answer is the whole message, so a named, typed output field described a place in the message that does not exist; models on several providers wrote the name into the message as a label (`answer:`, `**answer**`, `[[ ## answer ## ]]`), after notes about the turn, and the whole message, label and notes included, was the answer. The description was listed twice. DSPy has no plain-text finish to follow (row of 2026-10-07). | In force. | ReActV2 gives that signature a `submit` tool, or DSPy defines a plain-text finish whose prompt differs. | | 2026-10-08 | A stored history turn shows the outputs it recorded and nothing for the rest: no filler, and not DSPy 3.2.1's `None`. A turn with no recorded output is its user message alone. A predictor that names a tool-calls output and is not a tool loop keeps its recorded outputs as the assistant text on native replay. The system message of every format leaves out a history input, since history arrives as turns: a field typed `history`, or one the call supplies as an `Imp.History`. | deepfates/imp#261, deepfates/imp#258. `lib/imp/adapter/chat.ex` `recorded_reply/3`, `system_signature/2`, `test/history_test.exs`. A filler or `None` puts words in the model's mouth that it never wrote, and a `[[ ## history ## ]]` section in the structure is one no message in the request contains. DSPy 3.2.1 does both (`ChatAdapter.format_assistant_message_content`, `Adapter.format`). | In force. | DSPy stops writing missing history outputs or listing the history input, and Imp follows it. | diff --git a/docs/diving-deeper/react.md b/docs/diving-deeper/react.md index 4b618145..ca7e37b2 100644 --- a/docs/diving-deeper/react.md +++ b/docs/diving-deeper/react.md @@ -52,15 +52,26 @@ A constrained output never takes that path: its allowed values are in `submit`'s schema, and text is not an answer to it. **`finish: :submit`** gives the one-text-output signature `submit` as well, -so its turn ends as DSPy's ReActV2 ends every turn: a step that writes text -and calls no tool is a thought (`:empty_tool_calls`), and the forced -`submit` follows (section 3). Use it when the model writes text it does not -mean as the answer: a sentence about what it will do next, or a report of -what it just did. The text then stays beside the calls as the step's -thought, and only `submit`'s argument is the answer. It costs one more -request whenever the model ends in prose, and it needs a provider that can -force a tool, or the extractor answers. The default, `finish: :auto`, lets -the signature decide as above. The option is saved with the program. +so its turn ends only through `submit`: a step that writes text and calls no +tool is a thought, not the answer. Use it when the model writes text it does +not mean as the answer: a sentence about what it will do next, or a report +of what it just did. The text then stays in the history as the step's +thought, and only `submit`'s argument is the answer. + +After such a step the loop goes on with an ordinary step: every tool, +`tool_choice: "auto"`, and the thought shown as the request's last message +(as in section 3). The model can then do what it said it would, or submit +what it wrote; a thought that was only a closing note can be submitted as an +empty answer. A second thought-only step in a row, or a step that says +nothing, interrupts the turn (`:empty_tool_calls`) and the forced `submit` +follows. Measured on real final steps from 11 models, most submitted the +thought's reply on that ordinary step, unforced, Meta's models included, +whose provider cannot force a tool; some went on with the work they had +announced; about one in eight wrote text again. Forcing at once, as DSPy +does, took the announced action away and added a request whose failure +lost the reply. It costs one more request whenever the model ends in prose. +The default, `finish: :auto`, lets the signature decide as above. The +option is saved with the program. The prompt says the same. When the provider holds the tools natively, the Chat adapter's system message for `Imp.react("intent -> answer", [look])` @@ -104,12 +115,15 @@ parse fails, or when a step gives neither a tool call nor an answer. Rather than return nothing, Imp makes one more request: - With `submit`, the request names `submit` as the required tool, as DSPy - does (`:forced_submit`). A provider that refuses a named `tool_choice` + does (`:forced_submit`). If the process's `Imp.Deadline` has already + passed, no request is made and the turn is `:incomplete` + (`:deadline_exceeded`). A provider that refuses a named `tool_choice` (its error names `tool_choice` and says what it supports, as Meta's "only `auto` is supported" does through OpenRouter) is asked again with `"required"` and `submit` as the only tool; if it refuses that too, or answers without `submit`, a typed extractor with no tools - reads the history and fills the outputs (`:extracted`). + reads the history and fills the outputs (`:extracted`). The history then + ends with those outputs, recorded as a step that called no tool. When the step before it wrote a thought and called nothing, that thought is the newest thing in the history, and the request would end on the @@ -284,7 +298,7 @@ answer has to follow a tool call, give the signature a second output, so the turn ends only through `submit`, or end it from the tool with `finish_on:`. Or keep the one output and pass `finish: :submit`. Text without a tool call -is then a thought, and the turn ends through `submit`: +is then a thought; the next step sees it and ends the turn through `submit`: ```elixir lm = @@ -297,7 +311,7 @@ reply = Imp.react(Imp.signature("ticket -> reply"), [on_call], lm: lm, finish: : {:ok, prediction} = Imp.call(reply, %{ticket: "We were charged twice this month."}) {Imp.get(prediction, :reply), prediction.metadata.termination_reason} -#=> {"Maya from atlas is on it.", :forced_submit} +#=> {"Maya from atlas is on it.", :submit} ``` ### One more request on a history you hold @@ -468,7 +482,7 @@ step wins. | `:submit` | the model called `submit` with valid outputs | | `:finished_by_tool` | a `finish_on` function finished the turn; `finished_by_tool` names the tool | | `:forced_submit` | interrupted; the last request, with `submit` required, answered | -| `:extracted` | interrupted; the provider could not require `submit`, and the extractor answered | +| `:extracted` | interrupted; the provider could not require `submit`, and the extractor answered; the history ends with its outputs | | `:last_text` | interrupted; one text output, and the last request's text is the answer | | `:incomplete` | interrupted, and there is no answer | @@ -479,7 +493,7 @@ When the turn was interrupted, `termination_cause` says why: | `:max_iters` | the step budget ran out | | `:parse_error` | a step's reply could not be read | | `:prediction_error` | a step's request failed | -| `:empty_tool_calls` | with `submit` (every signature but one unconstrained text output under `finish: :auto`): a step called no tool | +| `:empty_tool_calls` | with `submit` (every signature but one unconstrained text output under `finish: :auto`): a step called no tool; under `finish: :submit`, a second such step in a row, or one that said nothing | | `:context_window_exceeded` | the prompt no longer fits, even with old episodes left out | | `:deadline_exceeded` | the process's `Imp.Deadline` passed | @@ -572,7 +586,9 @@ What differs: - A signature with one text output has no `submit`; the model's text is the answer (`:answered`, and `:last_text` when interrupted). `finish: :submit` - gives it `submit`, as DSPy does. Its last request + gives it `submit`, as DSPy does, but a thought-only step is followed by an + ordinary step rather than at once by the forced `submit`, which DSPy sends + after every step that calls no tool. A plain-text turn's last request runs no tool by default; `last_request_tools:` names the ones it may run, the counterpart of DSPy executing `submit` in its forced last request. - `finish_on:` lets a tool end the turn (`:finished_by_tool`). diff --git a/lib/imp/predict/react_v2.ex b/lib/imp/predict/react_v2.ex index a6fbb7e9..e922022d 100644 --- a/lib/imp/predict/react_v2.ex +++ b/lib/imp/predict/react_v2.ex @@ -21,12 +21,17 @@ defmodule Imp.Predict.ReActV2 do signature, so a user tool cannot take it. `finish: :submit` gives the one-text-output signature `submit` too, and its - turn then ends as every other signature's does: a step that writes text and - calls no tool is a thought (`:empty_tool_calls`), and the forced `submit` - follows. It is for a program whose model writes text it does not mean as - the answer, such as a sentence about what it will do next, or a report of - what it did. The default, `finish: :auto`, leaves the choice to the - signature. + turn then ends only through `submit`: a step that writes text and calls no + tool is a thought, not the answer. It is for a program whose model writes + text it does not mean as the answer, such as a sentence about what it will + do next, or a report of what it did. Under `finish: :submit` a thought-only + step is followed by an ordinary step (every tool, `tool_choice: "auto"`, + the thought shown as the request's last message), so the model can do what + it announced or submit what it wrote; a second thought-only step in a row, + or a step that says nothing, is the `:empty_tool_calls` interruption and + the forced `submit` follows. Every other signature with `submit` keeps + DSPy's ending: its first thought-only step is that interruption. The + default, `finish: :auto`, leaves the choice to the signature. ## The prediction @@ -108,7 +113,8 @@ defmodule Imp.Predict.ReActV2 do A turn is interrupted when it reaches `max_iters`, when a step's request fails (`:prediction_error`, `:parse_error`) other than by the content filter, or when a step of a signature - with `submit` calls no tool (`:empty_tool_calls`). An `enum` output is such a + with `submit` calls no tool (`:empty_tool_calls`; under `finish: :submit`, + the second such step in a row, see above). An `enum` output is such a signature: text that is not a `submit` call is `:empty_tool_calls`, never an answer, and a turn with no valid `submit`, the forced one included, ends `:incomplete`. @@ -150,9 +156,12 @@ defmodule Imp.Predict.ReActV2 do `submit`, and that call runs. The default, `[]`, runs nothing. With `submit`, an interruption forces one more request with `tool_choice` - naming `submit` (`:forced_submit`), as DSPy does. If a provider cannot honor - that tool contract, a tools-disabled typed extractor derives the task - outputs from the original inputs and accumulated history (`:extracted`). + naming `submit` (`:forced_submit`), as DSPy does, unless the process's + `Imp.Deadline` has already passed (`:incomplete`, `:deadline_exceeded`). + If a provider cannot honor that tool contract, a tools-disabled typed + extractor derives the task outputs from the original inputs and + accumulated history (`:extracted`), and the outputs are appended to the + history as a step that called no tool. The last request says nothing about why it is being made unless `:last_request_note` is given: one line of host text put in front of it as a @@ -351,7 +360,9 @@ defmodule Imp.Predict.ReActV2 do "How the model ends its turn. `:auto` lets the signature decide: one " <> "unconstrained text output ends on plain text, every other signature through " <> "`submit`. `:submit` ends every turn through `submit`, as DSPy's ReActV2 does, " <> - "so a step that writes text and calls no tool is a thought, not the answer. " <> + "so a step that writes text and calls no tool is a thought, not the answer; " <> + "the next step continues with every tool, and a second such step in a row " <> + "forces `submit`. " <> "See \"Which signatures get `submit`\" above." ], last_request_tools: [ @@ -688,15 +699,19 @@ defmodule Imp.Predict.ReActV2 do pending, 0, max_iters, - execution + execution, + 0 ) end end - defp run(react, history, inputs, pending, turn, max_iters, execution) when turn >= max_iters, - do: interrupted(react, history, inputs, pending, :max_iters, turn, nil, execution) + # `thoughts` counts the thought-only steps since the last step that called + # a tool (see `continue_after_thought?/3`). + defp run(react, history, inputs, pending, turn, max_iters, execution, _thoughts) + when turn >= max_iters, + do: interrupted(react, history, inputs, pending, :max_iters, turn, nil, execution) - defp run(react, history, inputs, pending, turn, max_iters, execution) do + defp run(react, history, inputs, pending, turn, max_iters, execution, thoughts) do case predict(react.react, react, history, pending) do {:ok, prediction, history} -> calls = prediction |> Imp.get(:tool_calls, []) |> normalize_calls(turn) @@ -720,7 +735,20 @@ defmodule Imp.Predict.ReActV2 do {:none, cause} -> emit_reasoning(prediction, turn) {history, pending} = append_thought_only_step(history, pending, prediction, calls) - interrupted(react, history, inputs, pending, cause, turn, nil, execution) + + if continue_after_thought?(react, prediction, thoughts), + do: + run( + react, + history, + inputs, + pending, + turn + 1, + max_iters, + execution, + thoughts + 1 + ), + else: interrupted(react, history, inputs, pending, cause, turn, nil, execution) end else emit_reasoning(prediction, turn) @@ -745,7 +773,7 @@ defmodule Imp.Predict.ReActV2 do }) true -> - run(react, history, inputs, %{}, turn + 1, max_iters, execution) + run(react, history, inputs, %{}, turn + 1, max_iters, execution, 0) end end end @@ -787,6 +815,30 @@ defmodule Imp.Predict.ReActV2 do end end + # Under `finish: :submit`, a step that wrote a thought and called nothing is + # followed by an ordinary step, not at once by the forced `submit`: the + # same tools and `tool_choice: "auto"`, with the thought as the request's + # last message (`Imp.Adapter.Chat`'s `thought_as_request`). Forcing at once + # took every other tool away from a model that had just said what it would + # do next, and made a request that, if it failed, lost the reply the thought + # held. Measured on real final steps across 11 models (decisions.md, row of + # 2026-10-16), most models then called `submit` with the thought's reply, or + # went on with the work they had announced; about one in eight wrote text + # again. So one such step continues, and a second in a row is the + # `:empty_tool_calls` interruption, as is a step that said nothing (there + # is no thought to show, and the same request would be sent again). + # `max_iters` and the deadline still end the turn as before. A signature that + # has `submit` under `finish: :auto` keeps DSPy's ending, which forces at + # once. + @thought_steps_continued 1 + + defp continue_after_thought?(%{finish: :submit}, prediction, thoughts) do + thoughts < @thought_steps_continued and + Imp.get(prediction, :next_thought) not in [nil, ""] + end + + defp continue_after_thought?(_react, _prediction, _thoughts), do: false + # The one place an interrupted turn goes: the last request for a signature # with one text output, the forced submit for every other. defp interrupted(react, history, inputs, pending, cause, turn, error, execution) do @@ -805,6 +857,34 @@ defmodule Imp.Predict.ReActV2 do initial_error, execution ) do + # As for the last text request: a deadline that has already passed + # leaves no time for the forced `submit`, so none is made. + if deadline_passed?() do + incomplete_prediction(history, :deadline_exceeded, initial_error) + else + forced_submit_request( + react, + history, + inputs, + pending, + reason, + turn, + initial_error, + execution + ) + end + end + + defp forced_submit_request( + react, + history, + inputs, + pending, + reason, + turn, + initial_error, + execution + ) do {history, pending} = note_after_inputs(history, pending, react) case forced_submit_prediction(react, history, pending) do @@ -825,7 +905,7 @@ defmodule Imp.Predict.ReActV2 do ) {:extract, forced_error, history} -> - extract_final(react, inputs, history, reason, %{ + extract_final(react, inputs, pending, history, reason, %{ initial: initial_error, forced_submit: forced_error }) @@ -1055,8 +1135,10 @@ defmodule Imp.Predict.ReActV2 do submit_calls = %ToolCalls{tool_calls: Enum.filter(calls.tool_calls, &submit?/1)} if submit_calls.tool_calls == [] do - history = append_last_step(history, pending, prediction, calls, nil) - extract_final(react, inputs, history, reason, initial_error) + stepped = append_last_step(history, pending, prediction, calls, nil) + # The step spent the pending inputs if it was recorded. + pending = if stepped == history, do: pending, else: %{} + extract_final(react, inputs, pending, stepped, reason, initial_error) else case execute_calls(react, submit_calls, execution, inputs) do {:cancel, cancel_reason} -> @@ -1112,7 +1194,7 @@ defmodule Imp.Predict.ReActV2 do } end - defp extract_final(react, inputs, history, reason, initial_error) do + defp extract_final(react, inputs, pending, history, reason, initial_error) do extractor = extraction_program(react) case context_call(history, fn projected -> @@ -1126,7 +1208,18 @@ defmodule Imp.Predict.ReActV2 do case Imp.Schema.validate_fields(react.signature.outputs, final) do :ok -> - final_prediction(final, history, :extracted, %{termination_cause: reason}) + # The outputs are recorded as what they are: produced with no tool + # call, after the turn's last step, with any inputs no step spent. + # The history then holds the turn's answer, as a `submit`'s event + # does, and ends the episode. + event = + pending + |> Map.merge(final) + |> Map.merge(%{tool_calls: %ToolCalls{tool_calls: []}, tool_call_results: []}) + + final_prediction(final, append_history(history, event), :extracted, %{ + termination_cause: reason + }) {:error, errors} -> incomplete_prediction(history, reason, %{ diff --git a/test/react_v2_finish_submit_test.exs b/test/react_v2_finish_submit_test.exs index 0545d86c..f1300d6b 100644 --- a/test/react_v2_finish_submit_test.exs +++ b/test/react_v2_finish_submit_test.exs @@ -2,8 +2,9 @@ defmodule ReActV2FinishSubmitTest do use ExUnit.Case, async: true # `finish: :submit` ends a one-text-output turn through `submit`, as DSPy's - # ReActV2 ends every turn: text without a tool call is a thought, and the - # forced `submit` follows. That forced request, for every signature with + # ReActV2 ends every turn: text without a tool call is a thought, not the + # answer (`react_v2_thought_continues_test.exs` covers what follows it). The + # forced `submit` request, for every signature with # `submit`, must not end on the model's own message, which Mistral and # Gemini refuse. @@ -41,7 +42,7 @@ defmodule ReActV2FinishSubmitTest do defp submit(arguments), do: reply(%{tool_calls: [%{id: "s", name: "submit", arguments: arguments}]}) - test "with finish: :submit, text without a tool call is a thought, and submit answers" do + test "with finish: :submit, text without a tool call is a thought, and the next step submits" do program = Imp.react("intent -> answer", [look()], lm: scripted(self(), [reply(@thought), submit(%{answer: "A cat."})]), @@ -50,12 +51,21 @@ defmodule ReActV2FinishSubmitTest do assert {:ok, prediction} = Imp.call(program, %{intent: "What is in the box?"}) assert Imp.get(prediction, :answer) == "A cat." - assert prediction.metadata.termination_reason == :forced_submit - assert prediction.metadata.termination_cause == :empty_tool_calls + assert prediction.metadata.termination_reason == :submit + refute Map.has_key?(prediction.metadata, :termination_cause) - [{_step, step_opts}, {_forced, forced_opts}] = requests() + [{_step, step_opts}, {continued, continued_opts}] = requests() assert Enum.map(step_opts[:tools], & &1.function.name) == ["look", "submit"] - assert forced_opts[:tool_choice] == %{type: "tool", name: "submit"} + + # The next request is an ordinary step: every tool, nothing forced, and + # the thought as its last message. + assert Enum.map(continued_opts[:tools], & &1.function.name) == ["look", "submit"] + assert continued_opts[:tool_choice] == "auto" + + assert List.last(continued) == %{ + role: :user, + content: "[[ ## next_thought ## ]]\nNow let me look at the box." + } # The thought stays in the history as the model's step. [thought_step, _submit_step] = prediction.metadata.history.messages diff --git a/test/react_v2_thought_continues_test.exs b/test/react_v2_thought_continues_test.exs new file mode 100644 index 00000000..c964e61a --- /dev/null +++ b/test/react_v2_thought_continues_test.exs @@ -0,0 +1,114 @@ +defmodule ReActV2ThoughtContinuesTest do + use ExUnit.Case, async: true + + # Under `finish: :submit`, a step that writes a thought and calls no tool is + # followed by an ordinary step, with every tool and nothing forced, so a + # model that said what it would do next can do it. A second such step in a + # row is forced to `submit`. + + defp look, do: Imp.tool(:look, "Look at a thing", fn _arguments -> "a cat" end) + + @thought %{next_thought: "Let me look in the box first.", tool_calls: []} + @look %{tool_calls: [%{id: "c1", name: "look", arguments: %{}}]} + @submit %{tool_calls: [%{id: "s1", name: "submit", arguments: %{"reply" => "A cat."}}]} + + defp scripted(owner, replies) do + {:ok, agent} = Agent.start_link(fn -> replies end) + + Imp.LM.Static.new( + handler: fn messages, opts -> + send(owner, {:request, messages, opts}) + + Agent.get_and_update(agent, fn + [reply | rest] -> {reply, rest} + [] -> {%{tool_calls: []}, []} + end) + end + ) + end + + defp tool_choices(acc \\ []) do + receive do + {:request, _messages, opts} -> tool_choices([opts[:tool_choice] | acc]) + after + 0 -> Enum.reverse(acc) + end + end + + defp call(replies, opts \\ []) do + Imp.react( + "ticket -> reply", + [look()], + [lm: scripted(self(), replies), finish: :submit] ++ opts + ) + |> Imp.call(%{ticket: "What is in the box?"}) + end + + test "a thought, then a second in a row, is forced to submit" do + assert {:ok, prediction} = call([@thought, @thought, @submit]) + assert Imp.get(prediction, :reply) == "A cat." + assert prediction.metadata.termination_reason == :forced_submit + assert prediction.metadata.termination_cause == :empty_tool_calls + assert tool_choices() == ["auto", "auto", %{type: "tool", name: "submit"}] + end + + test "a thought, then the tool it announced, and another thought continues too" do + assert {:ok, prediction} = call([@thought, @look, @thought, @submit]) + assert Imp.get(prediction, :reply) == "A cat." + assert prediction.metadata.termination_reason == :submit + assert tool_choices() == ["auto", "auto", "auto", "auto"] + + results = + for step <- prediction.metadata.history.messages, r <- step.tool_call_results, do: r.result + + assert "a cat" in results + end + + test "the step limit still ends the turn with the forced submit" do + assert {:ok, prediction} = call([@thought, @submit], max_iters: 1) + assert prediction.metadata.termination_reason == :forced_submit + assert prediction.metadata.termination_cause == :max_iters + assert tool_choices() == ["auto", %{type: "tool", name: "submit"}] + end + + test "a deadline that has already passed makes no forced submit" do + owner = self() + + slow_look = + Imp.tool(:look, "Look at a thing", fn _arguments -> + Process.sleep(20) + "a cat" + end) + + program = + Imp.react("ticket -> reply, confidence: float", [slow_look], + lm: scripted(owner, [@look, @submit]), + max_iters: 1 + ) + + assert {:ok, prediction} = + Imp.Deadline.with_deadline(10, fn -> + Imp.call(program, %{ticket: "What is in the box?"}) + end) + + assert prediction.metadata.termination_reason == :incomplete + assert prediction.metadata.termination_cause == :deadline_exceeded + assert tool_choices() == ["auto"] + end + + test "extracted outputs are recorded in the history" do + replies = [ + @thought, + @thought, + %{next_thought: "A cat, I think.", tool_calls: []}, + %{reasoning: "The model said so.", reply: "A cat."} + ] + + assert {:ok, prediction} = call(replies) + assert prediction.metadata.termination_reason == :extracted + + last = List.last(prediction.metadata.history.messages) + assert last.reply == "A cat." + assert last.tool_calls.tool_calls == [] + end +end From ab4a7009a35cd575961cc14d6aa76b7052813fa7 Mon Sep 17 00:00:00 2001 From: deepfates Date: Thu, 15 Oct 2026 22:09:03 -0700 Subject: [PATCH 2/2] ReActV2: no request after the deadline; resume after an extracted turn ends on the user side thought_as_request looks past a last step that called nothing and wrote nothing (the extracted outputs), so resuming after an :extracted turn no longer ends on the forced step's thought as an assistant message. No continued step, required retry or extraction request is sent once the deadline has passed. A whitespace-only thought is not continued. Docs: the deadline ends the turn incomplete rather than forcing submit; with written tool calls the request ends on the tool listing; the react.md example makes the lookup it announces. --- CHANGELOG.md | 22 ++++-- decisions.md | 2 +- docs/diving-deeper/react.md | 17 +++-- lib/imp/adapter/chat.ex | 24 ++++++- lib/imp/predict/react_v2.ex | 36 ++++++++-- test/react_v2_thought_continues_test.exs | 88 ++++++++++++++++++++++++ 6 files changed, 166 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b67dfd1e..65e019b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,19 +36,27 @@ User-visible changes to Imp are recorded here. - Under ReActV2's `finish: :submit`, a step that writes a thought and calls no tool is now followed by an ordinary step (every tool, - `tool_choice: "auto"`, the thought shown as the request's last message) + `tool_choice: "auto"`, the thought shown as the request's last message when + the tools are held natively) instead of the forced `submit`. A second thought-only step in a row, or a - step that says nothing, still forces `submit` (`:empty_tool_calls`), as do - `max_iters` and the deadline. A model that announced an action can take + step that says nothing or only whitespace, still forces `submit` + (`:empty_tool_calls`), as does `max_iters`. Once the deadline has passed, + no further request is sent, continued or forced. A model that announced an action can take it, and a reply in the thought is usually submitted unforced, also on providers that cannot force a tool. Other signatures with `submit` are unchanged. -- ReActV2's forced `submit` is not sent once the process's `Imp.Deadline` - has passed; the turn is `:incomplete` with `termination_cause: - :deadline_exceeded`, as the last text request already was. +- Once the process's `Imp.Deadline` has passed, ReActV2 sends no forced + `submit`, no `"required"` retry and no extraction request (and, under + `finish: :submit`, no continued step); the turn is `:incomplete` with + `termination_cause: :deadline_exceeded`, as the last text request already + was. This applies to every signature with `submit`, `finish: :auto` + included. - An `:extracted` ReActV2 turn's history now ends with the extracted outputs, recorded as a step that called no tool (with any inputs no step - spent), so the history holds the turn's answer. + spent), so the history holds the turn's answer. This applies under + `finish: :auto` too. Such a step shows the model nothing when the history + is replayed: the request still ends on the thought before it, shown as a + user message. - A ReActV2 forced `submit` request no longer ends on the model's own message. When the step before it wrote a thought and called nothing, the thought was the request's last message, as an assistant turn, and providers refuse that: diff --git a/decisions.md b/decisions.md index 33fe2b67..c433377d 100644 --- a/decisions.md +++ b/decisions.md @@ -8,7 +8,7 @@ not necessarily when it was made. | Date | Decision | Source and reason | Status | Retires when | | --- | --- | --- | --- | --- | -| 2026-10-16 | Under `finish: :submit`, a ReActV2 step that writes a thought and calls no tool is followed by an ordinary step (every tool, `tool_choice: "auto"`, the thought as the request's last user message), not at once by the forced `submit`; a second thought-only step in a row, or a step that says nothing, is the `:empty_tool_calls` interruption. `max_iters` and the deadline end the turn with the forced `submit` as before. Signatures with `submit` under `finish: :auto` keep DSPy's ending. | `lib/imp/predict/react_v2.ex` `continue_after_thought?/3`, `test/react_v2_thought_continues_test.exs`. Forcing at once took every other tool away from a model that had just announced its next action, and added a request whose failure lost the reply the thought held. Measured 2026-10-16 on 42 real final steps from 11 models through OpenRouter, sending that ordinary step: most samples called `submit` with the thought's reply, copied or paraphrased (Meta's models 6 of 6, unforced; Kimi, GPT and Gemini models likewise); a thought that was a closing note was submitted as an empty answer; a DeepSeek model went on to fetch what it had said it would read; 5 of 42 wrote text again. DSPy's ReActV2 forces `submit` after any step that calls no tool. | In force. | A measurement shows a different bound, or forcing at once, loses fewer answers. | +| 2026-10-16 | Under `finish: :submit`, a ReActV2 step that writes a thought and calls no tool is followed by an ordinary step (every tool, `tool_choice: "auto"`, the thought as the request's last user message when the tools are held natively), not at once by the forced `submit`; a second thought-only step in a row, or a step that says nothing or only whitespace, is the `:empty_tool_calls` interruption. `max_iters` ends the turn with the forced `submit` as before; once the deadline has passed no further request is sent and the turn is `:incomplete` (`:deadline_exceeded`). With written tool calls the request ends on the tool listing, after the thought. Signatures with `submit` under `finish: :auto` keep DSPy's ending. | `lib/imp/predict/react_v2.ex` `continue_after_thought?/3`, `test/react_v2_thought_continues_test.exs`. Forcing at once took every other tool away from a model that had just announced its next action, and added a request whose failure lost the reply the thought held. Measured 2026-10-16 on 42 real final steps from 11 models through OpenRouter, sending that ordinary step: most samples called `submit` with the thought's reply, copied or paraphrased (Meta's models 6 of 6, unforced; Kimi, GPT and Gemini models likewise); a thought that was a closing note was submitted as an empty answer; a DeepSeek model went on to fetch what it had said it would read; 5 of 42 wrote text again. DSPy's ReActV2 forces `submit` after any step that calls no tool. | In force. | A measurement shows a different bound, or forcing at once, loses fewer answers. | | 2026-10-16 | ReActV2's forced `submit` request, when the step before it wrote a thought and called nothing, shows that thought as the request's last message, a user message under its field's marker (`[[ ## next_thought ## ]]`), not as the assistant turn the history records it as. DSPy's ReActV2 leaves the thought out of the forced request. | `lib/imp/adapter/chat.ex` `thought_as_request/4`, `test/react_v2_finish_submit_test.exs`. As a trailing assistant turn the request is refused by providers that do not take an assistant prefix: Mistral ("Cannot set an assistant prefix with function calling") and Gemini ("Requests ending with a model turn are not supported"), probed through OpenRouter 2026-10-16. Dropping it, as DSPy does, made models regenerate finished answers with reasoning off, 2 of 16 to the output limit (deepfates/imp#336). Measured 2026-10-16 on real final steps re-sent through OpenRouter: with the thought as this user message, the submitted answer kept the plain-text reply (text similarity 0.8 to 1.0 across Kimi, GPT, DeepSeek and Muse models), a model whose plain text was a closing note submitted an empty answer as it intended, Meta's models (which accept only `tool_choice: auto`) copied the reply into `submit` 3 of 3, and Gemini accepted every request; with the thought dropped, the models wrote a different answer (similarity about 0.05), one shrank to 30 characters, and the closing-note model submitted a 170-character answer. | In force. | A measurement shows the thought is better dropped, or DSPy keeps it in a form providers accept. | | 2026-10-12 | In that plain-text step (row of 2026-10-07; a program with `finish: :auto`, the default), the text output is not presented as a field: the system message lists no output fields, the default agent line names no output ("You are an Agent. Use the supplied tools to gather information and take actions."), there is no "The outputs to produce are:" list, and the finish line is "When the final answer is ready, write it as plain text without calling a tool." followed by the field's description, once. Amended 2026-10-15: the finish line was "... reply without calling a tool."; "reply" is a common tool name, so for a program holding a `reply` tool it read as an instruction about that tool, and it now uses the sentence the other tool-less text answer already used. A host's `agent_line` still replaces the agent line. | `lib/imp/adapter/chat.ex` `default_system/2`, `with_guidance/5`; `test/react_v2_plain_text_answer_test.exs`. The answer is the whole message, so a named, typed output field described a place in the message that does not exist; models on several providers wrote the name into the message as a label (`answer:`, `**answer**`, `[[ ## answer ## ]]`), after notes about the turn, and the whole message, label and notes included, was the answer. The description was listed twice. DSPy has no plain-text finish to follow (row of 2026-10-07). | In force. | ReActV2 gives that signature a `submit` tool, or DSPy defines a plain-text finish whose prompt differs. | | 2026-10-08 | A stored history turn shows the outputs it recorded and nothing for the rest: no filler, and not DSPy 3.2.1's `None`. A turn with no recorded output is its user message alone. A predictor that names a tool-calls output and is not a tool loop keeps its recorded outputs as the assistant text on native replay. The system message of every format leaves out a history input, since history arrives as turns: a field typed `history`, or one the call supplies as an `Imp.History`. | deepfates/imp#261, deepfates/imp#258. `lib/imp/adapter/chat.ex` `recorded_reply/3`, `system_signature/2`, `test/history_test.exs`. A filler or `None` puts words in the model's mouth that it never wrote, and a `[[ ## history ## ]]` section in the structure is one no message in the request contains. DSPy 3.2.1 does both (`ChatAdapter.format_assistant_message_content`, `Adapter.format`). | In force. | DSPy stops writing missing history outputs or listing the history input, and Imp follows it. | diff --git a/docs/diving-deeper/react.md b/docs/diving-deeper/react.md index ca7e37b2..26881145 100644 --- a/docs/diving-deeper/react.md +++ b/docs/diving-deeper/react.md @@ -60,11 +60,13 @@ thought, and only `submit`'s argument is the answer. After such a step the loop goes on with an ordinary step: every tool, `tool_choice: "auto"`, and the thought shown as the request's last message -(as in section 3). The model can then do what it said it would, or submit -what it wrote; a thought that was only a closing note can be submitted as an -empty answer. A second thought-only step in a row, or a step that says -nothing, interrupts the turn (`:empty_tool_calls`) and the forced `submit` -follows. Measured on real final steps from 11 models, most submitted the +when the tools are held natively (as in section 3; with written tool calls +the request ends on the tool listing, after the thought). The model can +then do what it said it would, or submit what it wrote; a thought that was +only a closing note can be submitted as an empty answer. A second +thought-only step in a row, or a step that says nothing, interrupts the turn +(`:empty_tool_calls`) and the forced `submit` follows. Once the process's +`Imp.Deadline` has passed, no further request is sent. Measured on real final steps from 11 models, most submitted the thought's reply on that ordinary step, unforced, Meta's models included, whose provider cannot force a tool; some went on with the work they had announced; about one in eight wrote text again. Forcing at once, as DSPy @@ -298,12 +300,15 @@ answer has to follow a tool call, give the signature a second output, so the turn ends only through `submit`, or end it from the tool with `finish_on:`. Or keep the one output and pass `finish: :submit`. Text without a tool call -is then a thought; the next step sees it and ends the turn through `submit`: +is then a thought, not the answer. Here the model announces a lookup and +forgets to make it; the next step sees what it wrote, makes the lookup, and +ends the turn through `submit`: ```elixir lm = script.([ %{next_thought: "Let me check who is on call for atlas.", tool_calls: []}, + %{tool_calls: [%{name: "on_call", arguments: %{"team" => "atlas"}}]}, %{tool_calls: [%{name: "submit", arguments: %{"reply" => "Maya from atlas is on it."}}]} ]) diff --git a/lib/imp/adapter/chat.ex b/lib/imp/adapter/chat.ex index e75e17eb..945f100f 100644 --- a/lib/imp/adapter/chat.ex +++ b/lib/imp/adapter/chat.ex @@ -1308,15 +1308,35 @@ defmodule Imp.Adapter.Chat do end end + # The last turn that is a step with something to show. A step that called + # nothing and wrote nothing renders no assistant message (ReActV2 records + # its extracted outputs that way), so the message before it is the thought + # it follows. One that carries inputs renders them as a user message, + # which already ends the request. defp last_history_turn(signature, inputs) do Enum.find_value(Enum.reverse(signature.inputs), fn field -> case fetch_field(inputs, field.name) do - %Imp.History{} = history -> history |> Imp.History.messages() |> List.last() - _other -> nil + %Imp.History{} = history -> + history + |> Imp.History.messages() + |> Enum.reverse() + |> Enum.drop_while(&silent_step?/1) + |> List.first() + + _other -> + nil end end) end + defp silent_step?(turn) do + turn = Imp.Example.new(turn) |> Imp.Example.to_map() + + native_tool_history_turn?(turn) and + normalize_history_tool_calls(fetch_field(turn, :tool_calls)) == [] and + String.trim(blank_to_empty(fetch_field(turn, :next_thought))) == "" + end + defp thought_only_step?(nil), do: false defp thought_only_step?(turn) do diff --git a/lib/imp/predict/react_v2.ex b/lib/imp/predict/react_v2.ex index e922022d..0cbc78dd 100644 --- a/lib/imp/predict/react_v2.ex +++ b/lib/imp/predict/react_v2.ex @@ -26,7 +26,8 @@ defmodule Imp.Predict.ReActV2 do text it does not mean as the answer, such as a sentence about what it will do next, or a report of what it did. Under `finish: :submit` a thought-only step is followed by an ordinary step (every tool, `tool_choice: "auto"`, - the thought shown as the request's last message), so the model can do what + the thought shown as the request's last message when the tools are held + natively), so the model can do what it announced or submit what it wrote; a second thought-only step in a row, or a step that says nothing, is the `:empty_tool_calls` interruption and the forced `submit` follows. Every other signature with `submit` keeps @@ -818,23 +819,28 @@ defmodule Imp.Predict.ReActV2 do # Under `finish: :submit`, a step that wrote a thought and called nothing is # followed by an ordinary step, not at once by the forced `submit`: the # same tools and `tool_choice: "auto"`, with the thought as the request's - # last message (`Imp.Adapter.Chat`'s `thought_as_request`). Forcing at once + # last message when the tools are held natively (`Imp.Adapter.Chat`'s + # `thought_as_request`; with written tool calls the request ends on the + # tool listing, after the thought). Forcing at once # took every other tool away from a model that had just said what it would # do next, and made a request that, if it failed, lost the reply the thought # held. Measured on real final steps across 11 models (decisions.md, row of # 2026-10-16), most models then called `submit` with the thought's reply, or # went on with the work they had announced; about one in eight wrote text # again. So one such step continues, and a second in a row is the - # `:empty_tool_calls` interruption, as is a step that said nothing (there - # is no thought to show, and the same request would be sent again). - # `max_iters` and the deadline still end the turn as before. A signature that + # `:empty_tool_calls` interruption, as is a step that said nothing or only + # whitespace (there is no thought to show, and the same request would be + # sent again). `max_iters` still ends the turn with the forced `submit`; a + # deadline that has passed sends no further request, continued or forced, + # and the turn ends `:incomplete` (`:deadline_exceeded`). A signature that # has `submit` under `finish: :auto` keeps DSPy's ending, which forces at # once. @thought_steps_continued 1 defp continue_after_thought?(%{finish: :submit}, prediction, thoughts) do thoughts < @thought_steps_continued and - Imp.get(prediction, :next_thought) not in [nil, ""] + String.trim(to_string(Imp.get(prediction, :next_thought) || "")) != "" and + not deadline_passed?() end defp continue_after_thought?(_react, _prediction, _thoughts), do: false @@ -1102,7 +1108,17 @@ defmodule Imp.Predict.ReActV2 do submit_only = %{react | tools: %{submit: submit}} fallback = forced_submit_program(submit_only, "required") - case predict(fallback, submit_only, history, pending) do + # No request is sent once the deadline has passed; the refusal is + # then the last request's failure, which names the deadline. + retry = + if deadline_passed?(), + do: :deadline_passed, + else: predict(fallback, submit_only, history, pending) + + case retry do + :deadline_passed -> + error + {:ok, prediction, history} -> {:ok, prediction, history} @@ -1195,6 +1211,12 @@ defmodule Imp.Predict.ReActV2 do end defp extract_final(react, inputs, pending, history, reason, initial_error) do + if deadline_passed?(), + do: incomplete_prediction(history, :deadline_exceeded, initial_error), + else: run_extractor(react, inputs, pending, history, reason, initial_error) + end + + defp run_extractor(react, inputs, pending, history, reason, initial_error) do extractor = extraction_program(react) case context_call(history, fn projected -> diff --git a/test/react_v2_thought_continues_test.exs b/test/react_v2_thought_continues_test.exs index c964e61a..18ca0e33 100644 --- a/test/react_v2_thought_continues_test.exs +++ b/test/react_v2_thought_continues_test.exs @@ -111,4 +111,92 @@ defmodule ReActV2ThoughtContinuesTest do assert last.reply == "A cat." assert last.tool_calls.tool_calls == [] end + + test "a thought of only whitespace is not continued: the same request would be sent again" do + assert {:ok, prediction} = call([%{next_thought: " \n ", tool_calls: []}, @submit]) + assert prediction.metadata.termination_reason == :forced_submit + assert tool_choices() == ["auto", %{type: "tool", name: "submit"}] + end + + test "resuming after an extracted turn does not end on the model's turn" do + replies = [ + @thought, + @thought, + %{next_thought: "A cat, I think.", tool_calls: []}, + %{reasoning: "The model said so.", reply: "A cat."}, + @submit + ] + + lm = scripted(self(), replies) + program = Imp.react("ticket -> reply", [look()], lm: lm, finish: :submit) + assert {:ok, extracted} = Imp.call(program, %{ticket: "What is in the box?"}) + assert extracted.metadata.termination_reason == :extracted + _first_turn = tool_choices() + + assert {:ok, _resumed} = + Imp.call(program, %{history: extracted.metadata.history, max_iters: 0}) + + assert_received {:request, messages, _opts} + assert List.last(messages).role != :assistant + end + + # The process's deadline passes during the request named first; no request + # after it is sent, and the turn ends without an answer. + @slow 20 + @refusal {:error, + %Imp.LMError{ + status: 400, + message: + "Invalid value for 'tool_choice': supported string values are none, auto, required" + }} + + for {name, signature, finish, replies} <- [ + {"the continued step", "ticket -> reply", :submit, [:slow_thought]}, + {"the required retry", "ticket -> reply, confidence: float", :auto, + [:thought, :slow_refusal]}, + {"the extractor", "ticket -> reply, confidence: float", :auto, [:thought, :slow_thought]} + ] do + @signature signature + @finish finish + @replies replies + + test "a deadline that passes before #{name} sends no request" do + owner = self() + {:ok, agent} = Agent.start_link(fn -> @replies end) + + lm = + Imp.LM.Static.new( + handler: fn _messages, opts -> + send(owner, {:request, [], opts}) + + case Agent.get_and_update(agent, fn + [next | rest] -> {next, rest} + [] -> {:unexpected, []} + end) do + :thought -> + @thought + + :slow_thought -> + Process.sleep(@slow) + @thought + + :slow_refusal -> + Process.sleep(@slow) + @refusal + end + end + ) + + program = Imp.react(@signature, [look()], lm: lm, finish: @finish) + + assert {:ok, prediction} = + Imp.Deadline.with_deadline(10, fn -> + Imp.call(program, %{ticket: "What is in the box?"}) + end) + + assert prediction.metadata.termination_reason == :incomplete + assert prediction.metadata.termination_cause == :deadline_exceeded + assert length(tool_choices()) == length(@replies) + end + end end