Skip to content

StreamEnd, Message.provider_state, Optional temperature (#182) - #183

Open
johnnichev wants to merge 2 commits into
mainfrom
feat/stream-end-provider-state
Open

johnnichev wants to merge 2 commits into
mainfrom
feat/stream-end-provider-state

Conversation

@johnnichev

Copy link
Copy Markdown
Owner

Closes #182 (items 1, 2, 3 and the loop test; item 5, a native Responses provider, is left for its own release).

What changed

StreamEnd (selectools.types, @beta): a provider may yield one as the last item of stream() / astream(). The agent consumes it in all three stream consumers (_streaming_call, _astreaming_call, the native path in Agent.astream):

  • usage goes into agent.usage, to on_llm_end / on_usage, and onto the LLM_CALL trace step. Streamed turns used to hit on_llm_end with None and never counted.
  • provider_state is copied onto the assistant Message the agent assembles from chunks.

Consumers that pattern-match on str / ToolCall ignore it. Fallback and Router re-yield it untouched. The OpenAI-compatible provider now emits it from the usage chunk it already requested on astream(), and requests it on stream() too. A usage object only counts if it has an integer prompt_tokens, so usage: null from compatible servers never produces a bogus item.

Message.provider_state: Optional[Dict[str, Any]]: opaque, JSON-safe. Carried through the tool loop (the tool-call turn is appended as-is; the final no-tool turn now copies it when the message is rebuilt), ConversationMemory._copy_message (deep copy), to_dict() / from_dict(), and the cache key (only keyed when set, so existing cache entries stay valid).

temperature: Optional[float] on AgentConfig, the Provider protocol, and every built-in provider. None means omit; the OpenAI-compatible and Anthropic request builders skip the field, Gemini already tolerated None. Default stays 0.0, so nobody's output changes. 0.0 is still sent (zero/falsy trap covered by a test).

Tests

  • tests/core/test_provider_state.py: field default, dict round trip, memory deep copy, cache key.
  • tests/agent/test_stream_end.py: usage + observers + trace on run, arun, astream; provider_state attachment; legacy providers unchanged.
  • tests/providers/test_openai_stream_end.py: mocked Chat Completions stream, sync and async, cached tokens, no-usage case, ordering after tool calls.
  • tests/providers/test_optional_temperature.py: agent boundary, cache key, OpenAI (4 methods), Anthropic, Gemini.
  • tests/agent/test_provider_state_roundtrip.py: the loop test from the issue. A fake Responses-style provider returns reasoning items in provider_state with a function call; after the tool runs, the second provider call sees the assistant message with its state intact, on run, streamed run, arun, and astream.
  • One existing test updated: test_astreaming_call_sync_fallback unpacked the helper's old two-tuple.

Full non-e2e suite: 7551 passed, 66 skipped. mypy, ruff, bandit clean.

Notes for review

  • Streamed turns on run() / arun() now record an LLM_CALL trace step; previously only astream() did. Listed under Changed in the changelog.
  • No version bump here; left for the release commit.
  • Docs: PROVIDERS.md (protocol + new section), STREAMING.md, changelog, and examples/115_stream_end_provider.py (offline).

Providers can yield a StreamEnd as the last item of stream()/astream() to
report usage (counted into agent totals, passed to on_llm_end/on_usage and
the LLM_CALL trace step) and to attach an opaque provider_state to the
assistant message the agent assembles from chunks. Message.provider_state
round-trips through to_dict/from_dict, memory copies and the cache key.
The OpenAI-compatible provider now emits it from the usage chunk it already
requested on astream() and now also requests on stream().

Refs #182
…tate

temperature is Optional[float] on AgentConfig, the Provider protocol and every
built-in provider; None omits the field (OpenAI-compatible and Anthropic
request builders), 0.0 is still sent. Default unchanged.

Adds the provider_state round-trip contract test from #182, PROVIDERS.md and
STREAMING.md sections, changelog entries and offline example 115.

Refs #182
@johnnichev
johnnichev marked this pull request as ready for review October 16, 2026 18:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Responses API gaps: usage on native streams, provider_state on Message, Optional temperature

1 participant