feat(telemetry): trace RPC calls made and handled - #7134
Merged
Merged
Conversation
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:07
2c854c3 to
fcf2513
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:42
fcf2513 to
6c9ac55
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:48
6c9ac55 to
9aa16ed
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 21:00
9aa16ed to
13e4d84
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 5, 2026 21:10
1306080 to
aa343e0
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:16
aa343e0 to
26a2704
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:31
26a2704 to
6632f30
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:56
6632f30 to
93134c5
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 01:09
93134c5 to
8c372ed
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 01:28
8c372ed to
0840636
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 6, 2026 05:23
3371ae3 to
c690ca5
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 6, 2026 05:56
3b4d43e to
78b2948
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 7, 2026 00:10
fe5b4a4 to
9fae149
Compare
davidzhao
marked this pull request as ready for review
September 7, 2026 03:38
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 7, 2026 05:05
d2fae93 to
6c714bf
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 7, 2026 06:50
8dc4304 to
a3701b2
Compare
chenghao-mou
reviewed
Sep 7, 2026
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 8, 2026 04:26
6e88aaa to
805a46d
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
chenghao-mou
approved these changes
Sep 8, 2026
jmcclanahan
approved these changes
Sep 9, 2026
Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. livekit-rtc is gaining an RpcInterceptor hook (add_rpc_interceptor on LocalParticipant) that wraps every perform_rpc call and every dispatched handler invocation; this installs a tracing interceptor on the job's local participant and turns each call into a span following the OpenTelemetry RPC semantic conventions: - rpc_call (SpanKind.CLIENT) for outgoing calls, parented to the current span so an RPC issued from a tool nests under function_tool; - rpc_handler (SpanKind.SERVER) for incoming invocations, parented to the primary session's root span so it lands on the session timeline. Attributes: rpc.system=livekit, rpc.method, lk.rpc.request_id, lk.pii.rpc.caller_identity / lk.pii.rpc.destination_identity, lk.rpc.payload_size, lk.pii.rpc.payload (truncated to 1 KiB), lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, and lk.rpc.handler_registered (false when a client called a method the agent never registered). RpcError and handler exceptions set error status. Installed from JobContext.connect() after the room connects, and from RoomIO.start() for a room connected elsewhere. On an SDK without the hook (the current livekit==1.1.17 pin) install is a no-op with one debug log, so this can land ahead of the SDK release; tracing activates once the pin moves to the release carrying RpcInterceptor. Tests exercise the interceptor with fake continuations (attributes, parenting, truncation, error codes, unregistered method, handler exceptions) and the install/degrade behavior; they pass against both the pinned SDK and the interceptor-capable SDK source tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rpc.system is always livekit here, so it carried no information. Responses are now recorded like requests: lk.pii.rpc.response, truncated to 1 KiB, alongside the existing size, so a client/agent disagreement about a message format is readable from both sides of the span. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Participant identities are application identifiers, not end-user data. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RoomIO.start only instrumented a room that was already connected; a room that connected later (ctx.connect() after session.start(), or one the user connects) got no RPC spans unless JobContext.connect() installed it. The install now lives in the connection-state handler, so every ordering and reconnects are covered; it is idempotent, one interceptor instance deduped by identity in the SDK. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…els the handler chain The SDK cancels the incoming chain when the caller's deadline passes or the room disconnects and maps it to an RpcError only after the interceptor has unwound; CancelledError is not an Exception, so the handler span ended UNSET. The interceptor now records RESPONSE_TIMEOUT or RECIPIENT_DISCONNECTED, by elapsed time against the deadline, with ERROR status. Also: the handler's parent comes from session_context.session_root_context instead of a local copy, payload truncation is a plain slice, and comments are trimmed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… elapsed time The SDK's deadline clock starts before the interceptor's, so a deadline cancel could read as a disconnect, and a CancelledError raised inside the chain (an APPLICATION_ERROR to the caller) was labelled as either. The SDK now sets invocation.cancel_reason before it cancels the chain; the interceptor records that, APPLICATION_ERROR when it is unset, and no code on an SDK without the field. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.
Depends on the
RpcInterceptorhook inlivekit-rtc(livekit/python-sdks#806, released in 1.1.18); the stack pinslivekit==1.1.18in #7128.installstill degrades to a no-op with one debug log on an older SDK.Spans
rpc_callfunction_toolrpc_handlerAttributes:
rpc.method,lk.rpc.request_id,lk.rpc.caller_identity,lk.rpc.destination_identity,lk.rpc.payload_size,lk.rpc.response_size,lk.rpc.response_timeout,lk.rpc.error_code,lk.rpc.handler_registered(false when a client called a method the agent never registered). Request and response bodies are under PII keys,lk.pii.rpc.payloadandlk.pii.rpc.response, truncated to 1 KiB.RpcErrorand handler exceptions set error status.Where
telemetry/rpc.py:TracingRpcInterceptor(subclassesrtc.RpcInterceptorwhen present) andinstall(local_participant).JobContext.connect()after the room connects, and fromRoomIO.start()for a room connected elsewhere. One singleton interceptor; the SDK dedups registrations by identity, so repeated installs are no-ops.Tests
tests/test_rpc_tracing.pydrives the interceptor with fake continuations: attributes and parenting for both directions, request and response truncation, error codes and status, the unregistered-method flag, handler exceptions, andinstallregistering once or degrading. Passes against both the pinned SDK and the interceptor-capable SDK source tree.Stacked on #7131.
🤖 Generated with Claude Code