Skip to content

fix(ai-amazon-bedrock): emit streaming "finish" after metadata arrives#6272

Open
tsushanth wants to merge 1 commit into
Effect-TS:mainfrom
tsushanth:fix-bedrock-streaming-usage
Open

fix(ai-amazon-bedrock): emit streaming "finish" after metadata arrives#6272
tsushanth wants to merge 1 commit into
Effect-TS:mainfrom
tsushanth:fix-bedrock-streaming-usage

Conversation

@tsushanth

Copy link
Copy Markdown

Closes #6237.

Summary

The Bedrock Converse stream sends metadata (with the populated usage block) after messageStop. The SDK was emitting the "finish" part synchronously on messageStop, capturing the still-empty { inputTokens: undefined, outputTokens: undefined, totalTokens: undefined } defaults — so the caller's terminal stream event always saw usage.inputTokens === undefined and usage.outputTokens === undefined, matching the reporter's symptom exactly.

Fix

Buffer the finish reason on messageStop, then emit "finish" from the metadata case once usage, cacheWriteInputTokens, and trace are populated. The output payload shape (reason / usage / metadata.bedrock) is unchanged — only the timing moves.

case "messageStop": {
  finishReason = InternalUtilities.resolveFinishReason(event.messageStop.stopReason)
  break
}

case "metadata": {
  usage.inputTokens = event.metadata.usage.inputTokens
  // ... existing usage / cacheWrite / trace population ...
  if (Predicate.isNotUndefined(finishReason)) {
    parts.push({ type: "finish", reason: finishReason, usage, metadata: { bedrock: { trace, usage: { cacheWriteInputTokens } } } })
    finishReason = undefined
  }
  break
}

Notes

I didn't add a streaming test — converseStream is null as any in the existing test harness in AmazonBedrockLanguageModel.test.ts, so there's no pattern yet for mocking ConverseResponseStreamEvent. Happy to follow up with a focused stream test once the maintainers point at the preferred mocking shape.

Closes Effect-TS#6237.

The Bedrock Converse stream sends `metadata` (with the populated `usage`
block) AFTER `messageStop`, but the SDK was emitting `"finish"`
synchronously on `messageStop`, capturing the still-empty
`{ inputTokens: undefined, outputTokens: undefined, totalTokens: undefined }`
defaults. The caller's terminal stream event always saw
`usage.inputTokens === undefined` / `usage.outputTokens === undefined`.

Buffer the finish reason on `messageStop`, then emit `"finish"` from the
`metadata` case once `usage`, `cacheWriteInputTokens`, and `trace` are
populated. Output payload shape (reason / usage / metadata.bedrock) is
unchanged; only the timing moves.

I didn't add a streaming test — `converseStream` is `null as any` in the
existing test harness in `AmazonBedrockLanguageModel.test.ts`, so there's
no pattern to mock `ConverseResponseStreamEvent` against yet. Happy to
follow up with a focused stream test once the maintainers point at the
preferred mocking shape.
@tsushanth tsushanth requested a review from IMax153 as a code owner June 11, 2026 16:17
@github-project-automation github-project-automation Bot moved this to Discussion Ongoing in PR Backlog Jun 11, 2026
@changeset-bot

changeset-bot Bot commented Jun 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d6cac31

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@effect/ai-amazon-bedrock Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Discussion Ongoing

Development

Successfully merging this pull request may close these issues.

@effect/ai-amazon-bedrock: streaming token usage is always empty

1 participant