feat(telemetry): trace RPC calls made and handled - #2498
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: 47f0ad0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
davidzhao
added this pull request to stack #2502
September 15, 2026 06:32
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 15, 2026 06:46
f87c0f2 to
c0a9e8c
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 16, 2026 04:28
c0a9e8c to
4976e41
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 16, 2026 04:34
4976e41 to
c64e497
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 20, 2026 05:58
1b7f8c3 to
e492fe1
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 20, 2026 07:21
3ab4fbf to
1849338
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 20, 2026 17:17
1849338 to
369136a
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 20, 2026 18:02
0d9cd70 to
87f7817
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 20, 2026 23:39
87f7817 to
c7f6e6c
Compare
davidzhao
marked this pull request as ready for review
September 21, 2026 03:27
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 21, 2026 03:28
c7f6e6c to
2618dcc
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 21, 2026 15:35
2618dcc to
07838b1
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 23, 2026 15:10
8f8a4a1 to
b10e66b
Compare
chenghao-mou
approved these changes
Sep 23, 2026
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 26, 2026 01:56
b10e66b to
9e4c3ad
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 26, 2026 05:26
9ca3a8a to
88adbc6
Compare
Port of livekit/agents#7134. `telemetry.rpc` installs one tracing interceptor on the job's local participant and turns each RPC into a span following the OpenTelemetry RPC semantic conventions: `rpc_call` (CLIENT, under the current span, so an RPC issued from a tool nests under `function_tool`) and `rpc_handler` (SERVER, under the primary session's root span). Attributes cover method, ids, identities, payload and response sizes, the response timeout in seconds, the RpcError code and whether a handler was registered; request and response bodies are recorded truncated to 1 KiB under `lk.pii.rpc.*` keys. Installed from JobContext.connect() after the room connects and from RoomIO on every connected transition; both are idempotent, the SDK dedups the singleton interceptor by identity. `StartSpanOptions` gains `kind`. Adaptation: the installed @livekit/rtc-node predates LocalParticipant.addRpcInterceptor (it is in an open SDK PR), so the interceptor and call shapes are declared structurally here and `install` feature-detects the hook, degrading to one debug log and a no-op like the Python layer on an older livekit-rtc. Not ported: the Python handling of asyncio cancellation of the handler chain (promises are not cancelled in Node; the SDK maps timeouts on the caller's side). Incoming invocations are dispatched by the SDK from its FFI event path, outside the job's AsyncLocalStorage, so the session and job roots did not resolve there and rpc_handler spans came out as roots of their own (seen in a live run). install() now captures the job (passed explicitly by JobContext.connect() and RoomIO, which itself runs on the SDK's event path) and the interceptor runs the handler chain inside it, which also gives the user's handler getJobContext(), as a Python handler has. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s room connects - an interceptor is bound to the job it is installed for, one per job, so a process hosting several jobs routes each participant's invocations to its own job rather than the last one installed - the job installs on its room's connected transition, whoever connected it: ctx.connect(), a session, or the entrypoint connecting ctx.room itself. RoomIO no longer installs; the job covers it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…acing The warm transfer lifecycle tests stand in a local participant without addRpcInterceptor; the install threw inside the room's connection handler and stalled the session. Tracing steps aside instead: an older room SDK or a stand-in participant gets no spans, never a broken connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tall Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 26, 2026 05:54
025ddc9 to
47f0ad0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of livekit/agents#7134. Stacked on #2497.
Description
Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.
rpc_callfunction_toolrpc_handlerjob_entrypoint) before or after the sessionAttributes:
rpc.method,lk.rpc.request_id,lk.rpc.caller_identity,lk.rpc.destination_identity,lk.rpc.payload_size,lk.rpc.response_size,lk.rpc.response_timeout,lk.rpc.error_code. Request and response bodies are under PII keyslk.pii.rpc.payload/lk.pii.rpc.response, truncated to 1 KiB.RpcErrorand handler exceptions set error status.Installed from
JobContext.connect()after the room connects, and fromRoomIO.start()on every connect for a room connected elsewhere. One interceptor per process; the SDK dedups registrations by identity.The SDK dispatches incoming invocations from its FFI event path, outside the job's AsyncLocalStorage, so
installcaptures the job (passed explicitly byJobContext.connect()and by RoomIO, which runs on that event path too) and the interceptor runs the handler chain inside it. That is what parentsrpc_handlerto the session root, and it gives the user's handlergetJobContext(), as a Python handler has.Depends on the SDK
The interceptor hook is
LocalParticipant.addRpcInterceptor, released in@livekit/rtc-node1.1.0 (livekit/node-sdks#724). The workspace catalog now pins@livekit/rtc-nodeto^1.1.0, andtelemetry/rpc.tsuses the SDK's ownRpcInterceptor,RpcCallInfoandRpcInvocationDatatypes directly: there is no structural fallback or feature detection,installtakes aLocalParticipantand the invocation'smethodis always present.Note:
@livekit/noise-cancellation-node0.1.9 (used only by the private examples package) still declares a^0.13peer on@livekit/rtc-node, sopnpm installprints one unmet-peer warning until that package publishes a release accepting 1.x.The SDK core answers an RPC for an unregistered method with
UNSUPPORTED_METHODbefore it reaches the interceptor chain, so such calls produce no span. Python'slk.rpc.handler_registeredattribute, which istrueon every span for that reason, is deliberately not ported.Changes Made
telemetry/rpc.ts(new):TracingRpcInterceptor,install; exported astelemetry.rpc.telemetry/traces.ts:StartSpanOptions.kindpassed throughstartSpan/startActiveSpan/startActiveSpanSync.telemetry/trace_types.ts: eleven new attributes (the twolk.pii.rpc.*keys are PII).job.ts,voice/room_io/room_io.ts: install points.Adaptations from the Python source
CancelledError/cancel_reasonbranch and its tests are not ported: Node promises are not cancelled and the SDK never cancels the handler chain.install(undefined)is tolerated sinceroom.localParticipantis optional in the JS SDK types.Testing
telemetry/rpc.test.ts(11 tests) with fake continuations: attributes and parenting both directions, truncation and byte sizes, error codes and status, the unregistered flag on the defensive path, handler exceptions, install once / degrade;room_io.test.tsinstall-on-connect case.agentssuite green; build, typecheck, lint, API report updated.🤖 Generated with Claude Code