feat(telemetry): add client telemetry through the shared Rust core by pblazej · Pull Request #1021 · livekit/client-sdk-android

pblazej · 2026-09-15T18:27:27Z

Client telemetry for Android, on top of the shared Rust core (livekit/rust-sdks#1396). Every Room reports its spans, RTC statistics, SDK warnings/errors and device state to its LiveKit Cloud project (only when the token carries the observability grant), for 1,160 lines of Kotlin and no new public types.

Public API

API	Scope	Notes
`LiveKit.disableTelemetry()`	process	TODO: final shape pending the token/consent discussion
`Room.emitTelemetryEvent(name, attributes = emptyMap())`	Room	string name + string attributes; limits enforced by the core
`Room.setTelemetryAttribute(key, value)`	Room	correlation ids for matching with app data; `null` removes

Nothing else is public. Configuration, instruments, transport and the UniFFI types stay internal; every tuning value (60 s export, 60 s windows, stats poll interval) is the core's default.

Platform code

What this platform adds on top of Rust (everything else — destination, token handling, retries, cache, holds, stats mapping, span state — is in the core).

Three files, 822 lines; 1,160 lines in total including the wiring in existing files and the build.

Files and responsibilities

File	LOC	Responsibility
`telemetry/Telemetry.kt`	285	installs the pipeline synchronously with the first Room (fail-open, a missing native library included); synchronous opt-out (in effect when `disableTelemetry()` returns: no stats request or submit starts after it; a Room created afterwards also deletes a previous launch's cache); SDK log records (ambient span, else the ambient Room, else the process); WebRTC errors; OkHttp transport that returns the raw answer and follows no redirect; enum mappings
`telemetry/DeviceTelemetry.kt`	337	thermal status, battery saver, Data Saver, memory trim, default network, battery, app state → `DeviceState`, drained from one channel on its own serial dispatcher (a failing change is skipped, never ends the stream); audio route / focus (audio switch: one listener pair per handler, however many Rooms share it, released by the last) and camera / microphone failures → device events
`telemetry/RTCTelemetry.kt`	200	Room events observed before the join, for the core's `lk.subscribe` (join-announced tracks reconciled at connect and after a full reconnect); one `getStats()` per peer connection every `statsPollIntervalMs()` (a track appearing re-reads the interval but keeps the deadline) → `recordPeerStats`; cancelled at the opt-out; a failing core call is logged and skipped, never the app's crash; report flattening
wiring in existing files and the build (below)	338
Total	1160	(non-test, non-generated)

Changes in existing code

Wiring only: every existing public API, the console logging and per-track statistics behave as on main. Every telemetry call in existing code is wrapped in guarded {}: a core failure is reported to the console on a best-effort basis (even a throwing app logger is swallowed) and never skips the SDK's own teardown or rollback.

Changed files

Where	Change	Behaviour for existing apps
`Room`	takes its scope and RTC instrument at creation (the first Room also installs the pipeline, synchronously: loading the native core and registering the device receivers, once per process); releases its share of the audio listeners in `release()`; hands the scope the app's URL and token as soon as `connect()` accepts the attempt; one `lk.connect` span per `connect()` (engine, ICE and room-connected checkpoints); starts the RTC instrument before the join; `setRoom` at join and room update, `disconnected` at clean-up	unchanged
`RTCEngine` (+ detekt baseline)	hands the scope the URL the app gave and its latest token at join and on every token refresh (room move is not handled by the SDK on main — `ROOM_MOVED` is a TODO in `SignalClient` — so there is no move to forward); a server Leave's protocol reason and a reconnect that gave up (`reconnect_failed`) reach `lk.room.disconnected`; suspending `peerStats()` for the poller; one `lk.reconnect` span per reconnect cycle with its reason, attempts as checkpoints; `signal` / `join_recv` / `pc_created` checkpoints	unchanged: `reconnect()` keeps its signature
`SignalClient`	`ws_open` / `offer_sent` / `answer_sent` checkpoints; carries the Room's scope on its coroutines	unchanged
`LocalParticipant`	one `lk.publish` span per publish attempt, nested under a still-running ambient span and ambient itself while the publish runs	unchanged
`RemoteTrackPublication.setSubscribed`	a manual subscribe opens `lk.subscribe` and wakes the stats poller	unchanged
`LKLog`	warnings and errors also go to the core, whatever the console level	unchanged: the app's logger gets exactly what it got before
`RTCModule`, `CameraCapturerUtils`	WebRTC error lines, microphone and camera failures also go to the core; with `enableWebRTCLogging` the console forward writes to the app's logger directly, so WebRTC lines are captured once	unchanged: the console gets the same lines
`LiveKit`	`disableTelemetry()`, synchronous	additive
`settings.gradle`	`-PlivekitUniffiVersion=<version>` (or `LIVEKIT_UNIFFI_VERSION`) resolves livekit-uniffi-android at that version from Maven Local (`cargo make android-package-local` in rust-sdks/livekit-uniffi publishes `0.0.1`), else the released artifact from `libs.versions.toml`	unchanged for consumers. A property/env switch rather than a committed Maven Local repository: nothing machine-specific lands in the build. Until a livekit-uniffi release carries the telemetry bindings, a build without the switch fails to compile, so this PR's CI stays red until then
`livekit-android-test`	`MockPeerConnection.statsReport`; JNA's JVM natives and `-PlivekitUniffiLibraryPath` (or `LIVEKIT_UNIFFI_LIBRARY_PATH`) to run the real core under Robolectric; `RoomTest` verifies the network-change reconnect reason and gives its mocked participant a real participant's empty sid	test only
`.github/workflows/android.yml`	one `Telemetry E2E test` step after the existing build-and-test step: builds `livekit-uniffi` for the host at the rust-sdks tag matching `gradle/libs.versions.toml` (`livekit-uniffi/v<version>`), starts otelcol-contrib 0.162.0 (SHA-256 checked) on :4319, and runs the telemetry package's tests (`TelemetryMockE2ETest` and the platform tests that need the core) with `LK_TELEMETRY_ENDPOINT`; a collector that never listens, or a skipped `TelemetryMockE2ETest`, fails the job	CI only; adds a debug Rust build (a few minutes) to the job. It first runs once the release that carries the telemetry bindings is in `libs.versions.toml`: until then the earlier build step fails to compile

Events

10 of 19 SPEC signals fully covered, 8 partially (platform limits or pending core capture), 1 skipped (smoke-test only).

Event coverage table

Signal (SPEC name)	Status	Source on this platform / why skipped
`lk.connect` span (+ checkpoints)	✅	`Room.connect`: `ws_open` · `signal` · `join_recv` · `pc_created` · `offer_sent` · `answer_sent` · `engine` · `pc_connected` · `room_connected` (`signal`/`join_recv` and the last three are stamped together: the SDK has no separate moment for them)
`lk.reconnect` span	✅	`RTCEngine.reconnect` cycle; reason from the trigger (signal close, primary/publisher ICE, network change); `attempt <n> quick\|full` per attempt
`lk.publish` span	✅	`LocalParticipant` publish, nested under a still-running ambient span; its warnings point at it
`lk.subscribe` span	✅	intent: remote publish under autoSubscribe, tracks the join announced (at connect), or `setSubscribed(true)` (wakes the poller); subscribed / failed / unsubscribed / unpublished; first media seen by the core
`lk.rtc.stats.sample`	✅	one `getStats()` per peer connection (publisher + subscriber), paced by the core
`lk.room.disconnected`	✅	Room clean-up; reason by its protocol number (a server Leave's own, so `agent_error` too), or `reconnect_failed` when a reconnect cycle gives up
`lk.telemetry.report`	✅	core
`custom.<name>`	✅	`Room.emitTelemetryEvent`
log records (warn/error)	⚠️ partial	SDK (`LKLog`) and WebRTC errors. The Rust core's own warnings/errors are not captured: Android does not install the Rust log forwarder (`logForwardBootstrap`), which is what makes the core copy them itself; it never passes Rust log entries to `telemetryLog`, so nothing is counted twice
`lk.device.thermal.changed`	⚠️ partial	`PowerManager` thermal status on API 29+; reported as unknown below (not exposed by the OS)
`lk.device.low_power.changed`	✅	battery saver (`ACTION_POWER_SAVE_MODE_CHANGED`); unknown without a power service
`lk.device.app_state.changed`	⚠️ partial	background when every activity is hidden (`TRIM_MEMORY_UI_HIDDEN`), foreground at an activity start; no lifecycle dependency
`lk.device.memory.changed`	⚠️ partial	`onTrimMemory` / `onLowMemory`; the OS never signals the end of pressure, so it counts as normal again when an activity starts; API 34+ no longer delivers the running-app levels
`lk.device.network.changed`	⚠️ partial	default network callback (API 24+; unknown below): wifi / cell / wired / VPN / bluetooth / other / unavailable, metered, Data Saver
`lk.device.battery.changed`	✅	`ACTION_BATTERY_CHANGED`
`lk.device.audio_route.changed`	⚠️ partial	the audio switch's selected device (default `AudioSwitchHandler` only); it names no reason
`lk.device.audio.interruption`	⚠️ partial	audio focus loss / gain (default `AudioSwitchHandler` only)
`lk.device.capture.failed`	⚠️ partial	camera errors and disconnects from the capturer, microphone init/start/record errors from the audio device module; a denied permission is not told apart (`other`)
`lk.ping`	❌ skipped	pipeline smoke test, never emitted in production paths

Local testing

Run the Android end-to-end test against a local OTel backend (LGTM) and look at a whole call in the local LGTM UI on :3000, in a few minutes.

Commands

The test (TelemetryMockE2ETest) runs the real Rust core on the SDK's mocks and reads back what its own collector wrote, so the core talks to that collector on :4319 and an overlay config (otelcol-lgtm.yaml) makes it forward everything to LGTM on :4318 as well.

# 1. Local OTel backend (OTLP/HTTP on :4318, UI on :3000)
docker run -d --name lk-lgtm -p 3000:3000 -p 4318:4318 grafana/otel-lgtm

# 2. Local LiveKit server — only for trying a sample app; the e2e test runs on mocks
livekit-server --dev

# 3. Unreleased Rust bindings (dev wiring explained above): the AAR in Maven Local, plus a
#    host build of the core for Robolectric
export JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home  # any JDK 17
(cd <rust>/livekit-uniffi && ANDROID_SIZE_LIMIT_BYTES=2097152 cargo make android-package-local)
(cd <rust> && cargo build -p livekit-uniffi)

# 4. The test's collector (otelcol-contrib), with the overlay that also forwards to LGTM
otelcol-contrib --config livekit-android-test/src/test/resources/telemetry/otelcol.yaml \
  --config livekit-android-test/src/test/resources/telemetry/otelcol-lgtm.yaml &

# 5. Point the core at the collector and run the e2e test
LK_TELEMETRY_ENDPOINT=http://127.0.0.1:4319 ./gradlew :livekit-android-test:testDebugUnitTest \
  -PlivekitUniffiVersion=0.0.1 -PlivekitUniffiLibraryPath=<rust>/target/debug \
  --tests '*TelemetryMockE2ETest*' --rerun

Then open http://localhost:3000 → Explore:

Tempo: {resource.service.name="livekit-client-android"} → the call's lk.connect, lk.reconnect, lk.publish, lk.subscribe spans, with their checkpoints as span events.
Loki: {service_name="livekit-client-android"} → lk.rtc.stats.sample windows, lk.device.* events, custom.e2e.checkpoint, lk.room.disconnected and the warning/error records; filter by the session's trace id to follow one Room.

Pointing LK_TELEMETRY_ENDPOINT straight at http://localhost:4318 works for a sample app, but the e2e test then skips: it asserts on its own collector's output.

changeset-bot · 2026-09-15T18:27:31Z

🦋 Changeset detected

Latest commit: 3da1b73

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
client-sdk-android	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-09-15T18:28:44Z

Dependency diff:

Bridge to livekit-telemetry via UniFFI, with process-wide opt-out, attribute setting, and custom event emission. Pipeline initializes at first Room creation and shuts down at opt-out. Production warnings and errors are forwarded to telemetry.

Instrument Room connect, LocalParticipant publish, and RemoteTrackPublication subscribe with telemetry spans.

Instrument reconnect cycles in RTCEngine, and connect checkpoints with context propagation in SignalClient.

Track OS device signals (thermal state, memory pressure, network connectivity, battery level, audio route changes, app lifecycle) and capture failures for observability.

Periodic WebRTC stats export for connected peer connections.

Expose disableTelemetry on LiveKit object, and emitTelemetryEvent/setTelemetryAttribute on Room.

Robolectric tests exercise full lifecycle against local OTLP collector, verifying span export and attribute propagation.

Add a telemetry E2E step to CI with a host Rust build and a local collector, add the changeset entry, and update the detekt baseline.

devin-ai-integration

Devin Review found 6 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-10-01T14:41:49Z

storageDirectory

+    fun disable() {
+        synchronized(this) { disabled = true } // after this, ifCollecting starts nothing
+        try {
+            telemetryDisable()
+        } catch (e: Throwable) { // a missing native library included: the opt-out never fails the app
+            diagnose(e, "The opt-out did not reach the core; Rooms created from now on still collect nothing.")
+        }


🟡 Early opt-out leaves cached batches

When disableTelemetry() runs before any Room exists, Telemetry.disable leaves previous-launch batches on disk. Only a later Telemetry.scope deletes that directory, so an app creating no Room retains the batches.

Learn more

The process opt-out is available before the first Room. A previous process can have left unsent telemetry under storageDirectory. In this case no pipeline is installed, and the opt-out only removes that directory on a later scope call. An app that opts out and does not create a Room retains the previous launch's batches.

Example: The app launches after an offline call left livekit-telemetry batches, calls LiveKit.disableTelemetry() in startup, and never creates a Room. The batches remain in its cache directory.

Recommended fix: Retain an application context or provide one to the early opt-out so Telemetry.disable can delete storageDirectory immediately, including when the core has not been configured; keep deletion synchronized with installation.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T14:41:50Z

trackEnded

            build()
        }
        participant.signalClient.sendUpdateSubscription(isDesired, participantTracks)
+        if (subscribed) guarded { participant.signalClient.rtcTelemetry?.subscribeIntent(this, participant) }


🟡 Cancelled subscriptions retain open spans

After setSubscribed(true) starts a pending subscribe, setSubscribed(false) never ends it. The publication remains registered, so cancelled subscriptions appear pending until track removal or disconnection.

Learn more

The manual subscription hook starts a core subscribe span when the app requests a track. Unsubscribing changes isDesired and sends a signal update, but the core only receives trackEnded for track removal events. setSubscribed does not remove the publication, so a pending subscription remains open.

Example: An app requests an unsubscribed camera, then cancels the request before the camera arrives. The lk.subscribe span keeps measuring a request the app no longer wants.

Recommended fix: Notify the RTC telemetry instrument on the transition to false and end/cancel the pending subscribe for that SID. Keep this distinct from a later server-driven track removal.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T14:41:52Z

pollStats

+
+        @Suppress("InjectDispatcher")
+        val poller = connection.launch(Dispatchers.Default + failOpen) {
+            pollStats(interval = { scope.statsPollIntervalMs().toLong() }, wake = wake) {


🟡 Poll interval failure stops RTC reporting

If statsPollIntervalMs() throws, pollStats terminates outside the poller's error handler. That Room never submits another RTC statistics report during the connection.

Learn more

The core supplies a polling interval for the next RTC stats request. The interval is evaluated by pollStats, outside the catch surrounding recordPeerStats. A UniFFI exception ends the poller coroutine; CoroutineExceptionHandler logs it but does not restart the worker. Other telemetry failures in the poll are explicitly swallowed so future samples continue.

Example: A core call fails while reading the next interval after a track appears. Every subsequent RTC window for the connected Room is missing, although the peer connections still work.

Recommended fix: Catch failures from scope.statsPollIntervalMs() inside the polling loop and use a bounded fallback interval, then retry the core on the next iteration.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T14:41:53Z

memoryCallbacks

+            val state = DeviceState(
+                thermal = ThermalState.UNKNOWN,
+                lowPowerMode = null,
+                appState = AppState.FOREGROUND,


🟡 Background sessions report foreground state

When a background service creates the first Room, DeviceTelemetry starts with appState set to foreground. The earlier UI-hidden callback cannot reach its newly registered listener, so the session misreports background activity.

Learn more

The first Room installs the device instrument and registers memoryCallbacks afterwards. That listener only changes app state to background when a UI-hidden trim callback arrives. If an Android service first creates a Room after all activities are already hidden, that callback has already passed, while the initial state is foreground. No subsequent trim is needed to keep the process backgrounded, so telemetry reports foreground until a later lifecycle transition.

Example: The user leaves the app, then a background service creates its first Room for an audio call. Its device records begin with foreground app state and remain there while the app stays hidden.

Recommended fix: Initialize app state from a process-level activity tracker that is active before the first Room, or represent the initial state as unknown until a reliable lifecycle signal arrives.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T14:41:54Z

sendRequestImpl

+            if (console || Telemetry.captures(loggingLevel)) {
+                val text = message()
+                if (console) {
+                    logger?.log(loggingLevel, t, text)
+                }
+                Telemetry.log(loggingLevel, t, text)


🟥 Signaling tokens can enter telemetry logs

When signaling fails, Telemetry.log exports SDK warnings regardless of console level. sendRequestImpl logs the full request, so token-bearing requests can reach telemetry.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-01T14:41:56Z

+            Request.Builder()
+                .url(request.url)
+                .post(request.body.toRequestBody())
+                .apply { request.headers.forEach { (name, value) -> header(name, value) } }
+                .build()


🟨 Bearer credentials can use plaintext transport

If telemetry supplies an HTTP URL, OkHttpTelemetryTransport sends its authorization headers without requiring HTTPS. The Room's bearer credential then travels in cleartext.

Was this helpful? React with 👍 or 👎 to provide feedback.

This was referenced Sep 30, 2026

feat(uniffi): expose the telemetry core livekit/rust-sdks#1396

Open

feat(telemetry): add the crate with its event model and OTLP encoding livekit/rust-sdks#1479

Open

pblazej changed the title ~~Telemetry~~ feat(telemetry): add client telemetry through the shared Rust core Oct 1, 2026

pblazej added 8 commits October 1, 2026 13:11

feat(telemetry): add connect, publish and subscribe spans to Room

0d006f7

Instrument Room connect, LocalParticipant publish, and RemoteTrackPublication subscribe with telemetry spans.

feat(telemetry): add reconnect and engine spans

d4f8318

Instrument reconnect cycles in RTCEngine, and connect checkpoints with context propagation in SignalClient.

feat(telemetry): add device state instrumentation

d7db333

Track OS device signals (thermal state, memory pressure, network connectivity, battery level, audio route changes, app lifecycle) and capture failures for observability.

feat(telemetry): collect RTC stats per peer connection

d6fceac

Periodic WebRTC stats export for connected peer connections.

feat(telemetry): expose opt-out, custom events and attributes

2e3029c

Expose disableTelemetry on LiveKit object, and emitTelemetryEvent/setTelemetryAttribute on Room.

test(telemetry): add e2e story against a local collector

d08c5d5

Robolectric tests exercise full lifecycle against local OTLP collector, verifying span export and attribute propagation.

chore(telemetry): wire CI, changeset and detekt baseline

3da1b73

Add a telemetry E2E step to CI with a host Rust build and a local collector, add the changeset entry, and update the detekt baseline.

pblazej force-pushed the blaze/telemetry branch from aff8d93 to 3da1b73 Compare October 1, 2026 13:53

pblazej marked this pull request as ready for review October 1, 2026 14:36

pblazej requested review from MaxHeimbrock, davidliu and xianshijing-lk as code owners October 1, 2026 14:36

devin-ai-integration Bot reviewed Oct 1, 2026

View reviewed changes

Navigation Menu

feat(telemetry): add client telemetry through the shared Rust core - #1021

feat(telemetry): add client telemetry through the shared Rust core#1021
pblazej wants to merge 8 commits into
mainfrom
blaze/telemetry

pblazej commented Sep 15, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pblazej commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Public API

Platform code

Changes in existing code

Events

Local testing

Uh oh!

changeset-bot Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pblazej commented Sep 15, 2026 •

edited

Loading

changeset-bot Bot commented Sep 15, 2026 •

edited

Loading

github-actions Bot commented Sep 15, 2026 •

edited

Loading