Skip to content

feat: public streaming + should_finalize contracts for final_turn_only - #175

Merged
johnnichev merged 3 commits into
mainfrom
feat/174-streaming-contract
Jul 14, 2026
Merged

johnnichev merged 3 commits into
mainfrom
feat/174-streaming-contract

Conversation

@johnnichev

Copy link
Copy Markdown
Owner

Closes #174.

The problem

Two behaviors that streaming integrators MUST rely on were source-only knowledge — a future refactor could flip them with no type/signature change and no test signal.

1. Streaming contract (documented + tested + strengthened)

Guarantee: final_turn_only=True + astream() never emits the structured answer as content chunks — on any path. Pinned clause-by-clause in tests/agent/test_structured_streaming_contract.py:

  • Without single_pass: loop prose streams; the synthesis call is non-streaming; StreamChunk(event="structured_synthesis_start") fires iff a synthesis call is made (reuse and should_finalize-skip paths emit no event).
  • With single_pass: content chunks are now suppressed — with the schema riding natively on every loop call, any content the model emits is the JSON envelope by construction. Tool-call chunks still stream for activity display. This closes the leak that was previously a documented caveat, which is exactly what makes the issue's "brackets on BOTH paths" ask unnecessary: there is no synthesis content to bracket, on either path.

The event/no-leak semantics are documented in AGENT.md as an explicit public contract and in the single_pass docstring.

2. should_finalize messages contract (documented + tested + backed)

Documented in the docstring and AGENT.md: messages is the run's conversation view at convergence — prior history (with memory), this turn's USER message(s), the ASSISTANT tool-call messages, and one Message(role=TOOL) per executed tool with tool_name, tool_call_id, and tool_result always populated.

To make that last guarantee true rather than aspirational: _append_tool_result now defaults tool_result to the message content, so error results and policy denials populate it too (previously only the success path did — an integrator matching on tool_result would have silently missed error turns).

Tests

13 contract tests: single_pass no-leak + tool-call chunks still visible, prose-streams/JSON-doesn't on the synthesis path, event-iff-synthesis across all three convergence paths, event chunks are content-free, predicate message-view contents (run + astream), and the tool_result invariant on success/parallel/error paths (error path was RED before the fix).

Full suite: 7,605 passed, 205 skipped (py3.12). ruff clean.

Behavior change note: single_pass + astream() stops streaming JSON as content — v1.2.0's documented caveat becomes a guarantee. single_pass is 0 days old and opt-in; this is the fix its one consumer asked for.

https://claude.ai/code/session_01L1JWChKMs9yTyK92qquhJa

An integrator adopting v1.2.0 final_turn_only in a streaming chat surface
had to read core.py to learn two load-bearing behaviors; a refactor could
have silently flipped them with zero signal. Both are now documented,
stated in docstrings, and pinned by dedicated contract tests (#174):

- Streaming guarantee: final_turn_only + astream NEVER emits the
  structured answer as content chunks, on any path. New: single_pass mode
  now SUPPRESSES content chunks (with the schema riding natively on every
  loop call, content is the JSON envelope by construction) - closing the
  leak that was previously a documented caveat. Tool-call chunks still
  stream. structured_synthesis_start fires iff a separate synthesis call
  is made (reuse and should_finalize-skip paths emit no event, pinned).

- should_finalize messages contract: the predicate receives the run's
  conversation view at convergence - prior history (with memory), this
  turn's USER message(s), ASSISTANT tool-call messages, and one
  Message(role=TOOL) per executed tool with tool_name, tool_call_id, and
  tool_result ALWAYS populated. To back the guarantee, _append_tool_result
  now defaults tool_result to the message content, so error results and
  policy denials populate it too (previously success-path only).

Closes #174

Claude-Session: https://claude.ai/code/session_01L1JWChKMs9yTyK92qquhJa
…rmalization

Self-review of the #174 PR (high-effort multi-agent review) caught the
contract overpromising; all findings addressed:

- The "structured answer NEVER streams on any path" claim was FALSE on the
  reuse path: without single_pass the loop streams model output as it
  always has, so a converged answer that validates has already streamed
  before reuse. The contract now states this precisely, and a new
  StreamChunk(event="structured_reuse") fires at the reuse decision so
  chat surfaces can convert/retract the just-streamed bubble. Pinned
  honestly in the contract tests (the reuse-leak assertion the old test
  avoided is now present).
- tool_call_id was documented as always populated - false for text-parsed
  tool calls (ToolCallParser creates id=None). Contract corrected:
  tool_name + tool_result are the guaranteed pair; tool_call_id only for
  native tool-calling providers.
- "tool_result ALWAYS populated" was false for histories persisted before
  this change (error/denial rows stored None). Message.from_dict now
  normalizes TOOL messages on load, with a test.
- Changelog restructured: single_pass chunk suppression and the
  tool_result unification moved under "### Changed" with migration notes
  (read AgentResult.parsed; match denial/error text instead of
  tool_result is None).
- _append_tool_result's now-redundant tool_result parameter removed
  (every call site passed content); policy-denial invariant test added;
  single_pass mode named once as ctx.single_pass_active instead of three
  inline reconstructions; contract test file made self-contained (no
  cross-test-module imports); version pin dropped from the AGENT.md
  section header.

Claude-Session: https://claude.ai/code/session_01L1JWChKMs9yTyK92qquhJa
@johnnichev

Copy link
Copy Markdown
Owner Author

Self-review (high-effort multi-agent review, 10 verified findings) — addressed in d0a27a3:

  1. ✅ The headline finding: my "structured answer NEVER streams on any path" guarantee was false on the reuse path — without single_pass, the loop streams model output normally, so a converged answer that validates has already streamed before reuse decides. Contract reworded precisely, and a new StreamChunk(event="structured_reuse") fires at the reuse decision so chat surfaces can convert/retract the just-streamed bubble. The reuse-leak assertion the original test conspicuously avoided is now present and pins the honest behavior.
  2. ✅ tool_call_id de-guaranteed (text-parsed tool calls have id=None); the guaranteed pair is tool_name + tool_result.
  3. ✅ Legacy sessions: Message.from_dict normalizes TOOL messages persisted before this change (tool_result=None → content), so the "always populated" claim holds for restored histories; tested.
  4. ✅ Changelog: single_pass suppression + tool_result unification moved under ### Changed with migration notes.
  5. ⚠️→✅ The tool_result is None sentinel (PLAUSIBLE finding): not used anywhere in-repo; unified semantics kept, breaking-use migration documented in the Changed entry.
  6. ✅ Policy-denial invariant test added (was untested despite being claimed).
  7. ✅ _append_tool_result's now-redundant tool_result param removed.
  8. ✅ single_pass mode named once (ctx.single_pass_active) instead of three inline reconstructions.
  9. ✅ Contract test file self-contained (no cross-test-module imports).
  10. ✅ Version pin dropped from the AGENT.md section header (changelog owns versioning).

Full suite: 7,607 passed, 205 skipped. CI green on the update pending.

…sult semantics

Max-effort self-review (39 agents, 15 verified findings incl. nits) showed
the #174 contract was only implemented on the astream() chunk surface.
All findings addressed:

- stream_handler (run/arun): the synthesis turn and single_pass loop turns
  no longer feed the token callback - the second streaming surface now
  honors the same suppression contract (mutation-verified tests).
- astream() non-streaming-provider fallback: tool-call chunks are now
  yielded there too, so single_pass suppression cannot produce a zero-
  chunk stream on providers without native async streaming
  (mutation-verified after fixing a fake whose class attribute was
  shadowed by an instance attribute).
- serve SSE: content-free event chunks are forwarded as
  {"type": "event", "event": ...} records instead of being dropped -
  the documented convert/retract flow is reachable through
  selectools.serve (test added).
- tool_result unification REVERTED in favor of the sharper v1.2-compatible
  contract: success -> tool_result carries the result text; error/policy/
  coherence denial -> None (the structural failure discriminator, now
  documented as such). from_dict round-trip fidelity restored (@stable API
  no longer rewrites persisted data); caller-seeded TOOL messages scoped
  out of the guarantee; Message docstring updated.
- Contract wording corrected everywhere: synthesis event is once-per-run
  (validation retries do not re-emit); structured_reuse qualified as
  non-single_pass in all three doc surfaces; SYSTEM context injections
  added to the messages enumeration; Since markers added (v1.3.0);
  changelog rewritten with an explicit Changed section scoped to the
  @beta single_pass surface; llms-full.txt regenerated.
- Case C pins that structured_reuse is NOT emitted in single_pass mode.

Pre-existing, unrelated: 5 starlette auth tests fail on clean main when
starlette is installed (zero CI coverage for that surface) - filed #176.

Claude-Session: https://claude.ai/code/session_01L1JWChKMs9yTyK92qquhJa
@johnnichev

Copy link
Copy Markdown
Owner Author

Second self-review at max effort (39 agents, 15 verified findings including nits) — all addressed in 30edc6c. The big one: the contract was only implemented on the astream() chunk surface; three other consumption paths violated it:

  1. ✅ stream_handler leak (run/arun): synthesis turns and single_pass loop turns no longer feed the token callback — mutation-verified tests (the first version of these tests passed trivially because the handler channel needs stream=True; caught and strengthened).
  2. ✅ Fallback path zero-chunk stream: providers without native async streaming now yield tool-call chunks in astream's acomplete fallback, so single_pass suppression can't render a silent stream — mutation-verified (also caught a test-fake bug: SharedFakeProvider.__init__ shadows subclass class-attributes).
  3. ✅ serve SSE dropped event chunks: forwarded as {"type": "event", ...} records now; test added; contract text extended to name all three surfaces.
  4. ✅ tool_result design REVERTED to the sharper contract: success ⇒ result text, error/policy/coherence denial ⇒ None — the None marker is the documented failure discriminator (v1.2-compatible, no breaking change). from_dict round-trip fidelity restored on the @stable API; caller-seeded TOOL messages scoped out; Message docstring updated. This resolves findings 4/5/6/7-adjacent cleanly instead of patching around the unification.
  5. ✅ Wording nits, all fixed: synthesis event is once-per-run (validation retries don't re-emit), structured_reuse qualified as non-single_pass consistently across the three doc surfaces, SYSTEM context injections added to the messages enumeration, Since: v1.3.0 markers, changelog restructured (Changed section scoped to the @beta single_pass surface), Case C pins no structured_reuse in single_pass, llms-full.txt regenerated.

Also filed #176: 5 pre-existing starlette auth test failures on clean main, invisible in CI (no [serve] extra in the matrix) — unrelated to this PR.

Local suite: 7,787 passed; the only failures are the pre-existing #176 set (verified failing on clean HEAD, never run in CI).

@johnnichev
johnnichev merged commit 6c0e3a9 into main Jul 14, 2026
9 checks passed
@johnnichev
johnnichev deleted the feat/174-streaming-contract branch July 14, 2026 23:42
johnnichev added a commit that referenced this pull request Jul 14, 2026
Bumps version 1.2.0 -> 1.3.0 and rolls the [Unreleased] changelog into a
dated [1.3.0] section (both changelog files).

Contents already merged via #175 (closes #174): the final_turn_only
streaming behavior and should_finalize message view are now documented,
tested public contracts enforced on astream() chunks, the stream_handler
callback, and the serve SSE stream; single_pass no longer streams the
structured answer (beta surface, Changed entry with migration note).

Claude-Session: https://claude.ai/code/session_01L1JWChKMs9yTyK92qquhJa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make final_turn_only synthesis streaming behavior + should_finalize message contract public/documented (not source-only)

1 participant