Support AssemblyAI inference context carryover#2057
Support AssemblyAI inference context carryover#2057rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 466ba88 The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| override _pushConversationItem(ev: ConversationItemAddedEvent): void { | ||
| const chatItem = ev.item; | ||
| if (chatItem instanceof ChatMessage && chatItem.role === 'assistant' && chatItem.textContent) { | ||
| let text = chatItem.textContent; | ||
| if (text.length > ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS) { | ||
| this.#logger.debug( | ||
| { fromChars: text.length, toChars: ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS }, | ||
| 'truncating agent_context carryover', | ||
| ); | ||
| text = text.slice(-ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS); | ||
| } | ||
| this.updateOptions({ modelOptions: { agent_context: text } as STTOptions<TModel> }); | ||
| } | ||
| } |
There was a problem hiding this comment.
🟡 Turned-off conversation context can still be applied to a speech model that rejects it
The assistant reply is written into the speech engine's context option (updateOptions({ modelOptions: { agent_context: text } }) at agents/src/inference/stt.ts:654) without first checking whether context carryover is actually enabled, so a model that had carryover turned off can still have the context applied.
Impact: A speech model configured without carryover (or one that doesn't support it) can still receive assistant context, which for unsupported models can cause the transcription request to be rejected or behave unexpectedly.
How the capability guard is bypassed via the fallback adapter
The base implementation agents/src/stt/stt.ts:265-275 guards on this.#capabilities.chatContext and warn-and-skips when carryover is unsupported/disabled. The agent activity only registers the forwarder when the STT's own chatContext capability is true (agents/src/voice/agent_activity.ts:624), so a directly-used inference STT with carryover disabled is safe.
However, FallbackAdapter._pushConversationItem (agents/src/stt/fallback_adapter.ts:199-204) forwards the item to every wrapped STT, explicitly relying on the comment that "unsupported ones warn-and-skip internally". Its own chatContext capability is true if any child supports it (agents/src/stt/fallback_adapter.ts:144). Because this inference override does not call super._pushConversationItem and does not check this.capabilities.chatContext, a wrapped inference STT whose carryover is disabled (unsupported model, agentContextCarryover: false, or previous_context_n_turns: 0) will still set agent_context in its modelOptions, which is then sent to the gateway even though agent_context is only supported for the U3 Pro carryover models.
| override _pushConversationItem(ev: ConversationItemAddedEvent): void { | |
| const chatItem = ev.item; | |
| if (chatItem instanceof ChatMessage && chatItem.role === 'assistant' && chatItem.textContent) { | |
| let text = chatItem.textContent; | |
| if (text.length > ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS) { | |
| this.#logger.debug( | |
| { fromChars: text.length, toChars: ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS }, | |
| 'truncating agent_context carryover', | |
| ); | |
| text = text.slice(-ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS); | |
| } | |
| this.updateOptions({ modelOptions: { agent_context: text } as STTOptions<TModel> }); | |
| } | |
| } | |
| override _pushConversationItem(ev: ConversationItemAddedEvent): void { | |
| if (!this.capabilities.chatContext) return; | |
| const chatItem = ev.item; | |
| if (chatItem instanceof ChatMessage && chatItem.role === 'assistant' && chatItem.textContent) { | |
| let text = chatItem.textContent; | |
| if (text.length > ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS) { | |
| this.#logger.debug( | |
| { fromChars: text.length, toChars: ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS }, | |
| 'truncating agent_context carryover', | |
| ); | |
| text = text.slice(-ASSEMBLYAI_MAX_AGENT_CONTEXT_CHARS); | |
| } | |
| this.updateOptions({ modelOptions: { agent_context: text } as STTOptions<TModel> }); | |
| } | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
agent_contextcarryover for supported U3 Pro family models.agentContextCarryoverconstructor control,previous_context_n_turns: 0default suppression, and assistant reply truncation to the last 1750 chars._pushConversationItemtests into the existing inference STT Vitest suite.Source diff coverage
livekit-agents/livekit/agents/inference/stt.pyagents/src/inference/stt.tsports the AssemblyAI U3 Pro carryover model gate, max context size update,previous_context_n_turnsoption,agentContextCarryoverconstructor option,chatContextcapability wiring, unsupported-model warning, and assistant-message_pushConversationItemforwarding viaagent_context. Adapted Python overloads/TypedDicts into TypeScript option interfaces and constructor option handling.tests/test_inference_stt_context.pyagents/src/inference/stt.test.tsports the added tests for default capability gating, unsupported-model warnings, explicit opt-out/override,previous_context_n_turns: 0, assistant text forwarding, 1750-char tail truncation, ignored user/textless/handoff items, and explicitagent_contextpreservation/overwrite. Adapted pytest/caplog assertions to Vitest and the targetChatMessage/AgentHandoffItemevent helpers.No source diff files were omitted. No target infrastructure gap was found; the target already had STT
chatContextcapability plumbing and AgentSession conversation-item forwarding.Validation
pnpm exec prettier --write agents/src/inference/stt.ts agents/src/inference/stt.test.tspnpm test agents/src/inference/stt.test.tspassed: 64 passed, 1 skipped.pnpm --filter @livekit/agents typecheckpassed.pnpm --filter @livekit/agents lintpassed with existing warnings.pnpm --filter @livekit/agents buildpassed.pnpm buildpassed.pnpm lintpassed with existing warnings.cue-clitext-mode runtime validation passed: a temporary JS agent usinginference.STT({ model: 'assemblyai/universal-3-5-pro' })emitteddebug_messagewithchatContext: trueandagentContext: "Carry this assistant reply into STT context."after an assistant turn.Known Test Failure
pnpm test agentsfailed due to 11 pre-existing/baseline failures inagents/src/voice/amd.test.ts.pnpm test agents/src/voice/amd.test.ts, which failed the same 11 AMD assertions/timeouts while the ported inference STT test file passes.Ported from livekit/agents#6466
Original PR description
already introduced in the assembly ai plugin