Skip to content

feat(telemetry): interruption detail, handoff span, fallback events, text input - #7137

Merged
davidzhao merged 10 commits into
dz/telemetry-rpcfrom
dz/telemetry-coverage
Sep 12, 2026
Merged

davidzhao merged 10 commits into
dz/telemetry-rpcfrom
dz/telemetry-coverage

Conversation

@davidzhao

@davidzhao davidzhao commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

What

Smaller coverage gaps on existing spans, each of which came up when reading a trace and not being able to answer a question from it.

Interruptions

agent_turn carried a single lk.interrupted boolean. It now also carries lk.interruption.source, set by the caller that knows the cause:

source meaning
audio_activity barge-in from VAD / STT activity
user_turn a committed user turn preempting the reply
programmatic session.interrupt(), a tool, teardown

The pipeline and say paths also stamp lk.playout.position, how many seconds had actually played when the user cut in. No events: the UI renders spans and attributes, so the interruption detail lives on the turn.

Agent handoff

update_agent() spans a new update_agent bar (parent agent_session) with lk.previous_agent_label and lk.agent_label. The old agent's drain_agent_activity (with on_exit inside) and the new agent's start_agent_activity nest under it. The initial start stays under session_start.

Fallback adapters (LLM and TTS)

The attempt span carries lk.fallback.label / lk.fallback.index for the provider that served the request; a failover shows as gen_ai.request.model (expected) differing from gen_ai.response.model (served) on the request span. STT fallback is left for the deferred STT-events work since STT has no request span to hang these on yet.

Text input

  • keyterm_detection span around the keyterm-detection LLM pass (stt_context_options.keyterm_detection), nested under the agent_turn that answers the user message (the conversation event fires from that reply: it is the agent's work on the turn), falling back to agent_session when there is no reply. Its llm_request used to sit directly under agent_turn, looking like a second inference step. Attributes are counts only (lk.keyterms.count/added/removed) plus model and provider; the terms themselves stay in the session report as lk.pii.keyterms.

  • Fallback adapters report who serves. FallbackAdapter.model / .provider (LLM, TTS, STT) follow the active instance: the primary before any traffic, then whichever last served, like metrics_metadata already did. llm_node, tts_node, start_agent_activity and the adapter's own request span therefore name a real model instead of FallbackAdapter/livekit. Because request attributes are stamped when a request starts, a failover mid-request is recorded on the response side: gen_ai.response.model and gen_ai.provider.name of the instance that answered are set on the adapter request span and on llm_node/tts_node when the node finishes. The serving run carries the instance's gen_ai.request.model and gen_ai.provider.name next to lk.fallback.label/index. Usage metrics needed no change: the inner streams already emit them with their own model and the adapters suppress their own.

Tests

tests/test_coverage_spans.py: barge-in source and playout position through a fake session, the update_agent span with drain/on_exit/start nested, the LLM fallback serving provider, and interrupt-source precedence.

Not in this layer

STT events on the turn (deferred behind eou_wait feedback), realtime session lifecycle, and IPC high-ping / CPU pressure events.

Stacked on #7134.

🤖 Generated with Claude Code

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from d274cf0 to f35af04 Compare September 5, 2026 20:42
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from f35af04 to dd9b4df Compare September 5, 2026 20:48
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from dd9b4df to 2d2ae2a Compare September 5, 2026 21:00
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 2d2ae2a to 40ce794 Compare September 5, 2026 21:07
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 40ce794 to 8f34d94 Compare September 5, 2026 21:10
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from d491a27 to 9e40051 Compare September 6, 2026 00:31
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from 2bdd416 to def301b Compare September 6, 2026 01:09
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from def301b to 87874d3 Compare September 6, 2026 01:28
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 87874d3 to 14c2b7c Compare September 6, 2026 01:37
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from 3731e28 to 9289b47 Compare September 6, 2026 05:23
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 6113f9f to 895ab3d Compare September 6, 2026 05:42
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 4bedb0b to 02b24cd Compare September 6, 2026 23:04
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from a2849a9 to b00f8eb Compare September 7, 2026 00:10
@davidzhao
davidzhao marked this pull request as ready for review September 7, 2026 03:40
@davidzhao
davidzhao requested a review from a team as a code owner September 7, 2026 03:40
devin-ai-integration[bot]

This comment was marked as resolved.

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from b00f8eb to 4801016 Compare September 7, 2026 03:54
devin-ai-integration[bot]

This comment was marked as resolved.

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 1f2a1e1 to fc68466 Compare September 7, 2026 05:05
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from fc68466 to 28c70f8 Compare September 7, 2026 05:48
self._authorized_at: float | None = None
# telemetry: who interrupted this speech, recorded by the caller that knows
self._interrupt_source: str | None = None
self._interruption_recorded = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think this is used?

Comment thread livekit-agents/livekit/agents/voice/agent_session.py Outdated
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@chenghao-mou chenghao-mou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, two nits

"""Priority for important messages that should be played before others."""

def __init__(
self, *, speech_id: str, allow_interruptions: bool, input_details: InputDetails

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should expose the source as a parameter here. We still call .interrupt() in the following cases:

if output_emitter.pushed_duration() > 0.0:
logger.warning(
f"{tts.label} already synthesized of audio, ignoring the current segment for the tts fallback" # noqa: E501
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should record here too if this has pushed some audio.

davidzhao and others added 10 commits September 11, 2026 15:01
…text input

Smaller coverage gaps on existing spans, each of which came up when reading a
trace and not being able to answer a question from it.

Interruptions: agent_turn carried a single lk.interrupted boolean. It now
records an `interrupted` event with lk.interruption.source, set by the caller
that knows the cause: `audio_activity` (barge-in), `user_turn` (a committed
turn preempting the reply), or `programmatic` (session.interrupt(), a tool,
teardown). The pipeline and say paths also stamp lk.playout.position, how many
seconds had actually played when the user cut in. The false-interruption path
records `playout_paused` when a barge-in pauses playout and a
`false_interruption` event with lk.false_interruption.resumed when the timer
resolves it.

Agent handoff: update_agent() spans a new `update_agent` bar (parent:
agent_session) with lk.previous_agent_label and lk.agent_label; the old agent's
drain_agent_activity (with on_exit inside) and the new agent's
start_agent_activity nest under it. The initial start stays under session_start.

Fallback adapters (LLM and TTS): the request span gets a
`fallback_provider_failed` event per provider that failed (lk.fallback.label,
lk.fallback.index) and, on success, lk.fallback.label / lk.fallback.index for
the provider that served the request. Previously the adapter only renamed the
request span, so a silent failover was invisible.

Text input: a `text_input` event on agent_session (participant, byte size) when
a lk.chat text stream arrives, so text-mode conversations show their input edge.

Tests: tests/test_coverage_spans.py covers barge-in source and playout position
through a fake session, the update_agent span and its children, LLM fallback
events, and interrupt-source precedence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The keyterm-detection LLM pass ran in the reply task's context, so its
llm_request appeared directly under agent_turn as if it were a second
inference step. It is STT context for later turns, not part of any reply: a
keyterm_detection span under agent_session, with model, provider and keyterm
counts (the terms themselves stay in the session report as lk.pii.keyterms).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ecord the model that answered

FallbackAdapter.model/provider (LLM, TTS, STT) now follow the active instance,
the primary before any traffic and then whichever last served, as
metrics_metadata already did. llm_node, tts_node, start_agent_activity and the
adapter's own request span name a real model instead of the adapter. Since
request attributes are stamped at request start, a failover mid-request is
recorded on the response side: gen_ai.response.model and gen_ai.provider.name
of the instance that answered, on the adapter request span and on the node
when it finishes. Fallback events carry the instance's model and provider next
to its label. Usage metrics needed no change: inner streams emit them with
their own model and the adapters suppress their own.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the message

The conversation event that starts a detection pass fires from the reply
answering the user message, so the pass is that agent's work on the turn.
Parent it to the current span at trigger time (the agent_turn), falling back to
the session root when nothing is current (a skipped reply, user code editing
the history).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ource before the pause cancel, handoff start under update_agent, text input size in bytes

- fallback adapters stamp the span the request was made under (llm_node/tts_node)
  with the instance that served, at the moment it serves, instead of the node
  reading the adapter's shared active instance when it finishes: concurrent
  requests on one adapter can be served by different instances
- a paused reply resolved by a user turn is interrupted by the pause cancel
  before its cause was named; name it first, and in the pause cancel itself
- a handoff's start_agent_activity takes the update_agent span as its parent
  explicitly rather than relying on the session start context being cleared
- lk.text_input.size counts UTF-8 bytes like the RPC payload sizes

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…alize the keyterm span's provider

model/provider read the first instance marked available (the primary once all
are down), so a recovered primary is reported again before it has served; the
instance that actually serves is stamped per request by the stream.
keyterm_detection now runs the provider through gen_ai_provider_name like every
other span, omitting it when unknown.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…andoff span is a parent, not current

The UI renders spans and attributes, so the interrupted, playout_paused,
false_interruption, fallback_provider_failed and text_input events go;
lk.interruption.source stays as an attribute on agent_turn, and a failover is
readable from the request span's expected vs served model.

update_agent was made current for the whole handoff, so the long-lived tasks
the new activity spawned inherited it and their spans nested under an ended
span; it is now current only around drain/pause/close and passed explicitly to
start and resume. The playout position is stamped only when this speech's own
audio played, since wait_for_playout returns the previous segment's event
otherwise.

Comments trimmed to what a reader of the code needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…me barge-in; record partially serving tts instance

- SpeechHandle.interrupt(source=...) and AgentActivity.interrupt(source=...) carry the
  cause of the interruption for lk.interruption.source; _set_interrupt_source removed
- realtime input_speech_started interrupts as audio_activity; superseded turns as user_turn
- tts fallback records the serving instance when it failed after pushing audio

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants