tests: trace-shape schema and checker

[davidzhao]

Port of livekit/agents#7148. Stacked on #2500.

Description

The nesting of spans is an emergent property of many call sites. A refactor can move a span under the wrong parent while every existing test passes, because each test asserts the one edge it was written for. This layer writes the rules down once and checks every span in every full-session test against them, and applies the same rules to a real run's export.

agents/src/telemetry/testing/trace_schema.ts (test support, not exported from the package):

• SPAN_PARENTS: for every span name the JS framework emits, the parents it may have (ROOT for none, ANY for spans that follow their caller: rpc_call, event_loop_blocked). An unknown name is a violation, so a new span must be registered.
• MAY_OUTLIVE_PARENT: the child/parent edges where the child may end after its parent, each with its reason (the startup spans session.start() does not wait for, on_enter, and stalls whose end is one heartbeat late). Everything else must sit inside its parent, with 2 ms of slack.
• checkTrace(): one trace id, every parent present, parent allowed by the schema, bounds, and the per-turn invariants: one agent_turn per lk.speech_id, lk.generation_count equal to the number of generation events, every eou_wait with an outcome.
• Two span sources, same rules: fromReadableSpans for an in-memory exporter, fromOtlpJson for an export downloaded from LiveKit Cloud. As a CLI, pnpm exec tsx agents/src/telemetry/testing/trace_schema.ts traces.json prints the span summary and the violations, tolerating orphans in a partial export.

assertTraceWellFormed(exporter.getFinishedSpans()) now ends the full-session tests of the earlier layers: agent_turn_span, coverage_spans, eou_wait_span, session_lifecycle_span.

Schema differences from Python

All from what the JS code emits: answering_machine_detection is JS's name for amd; JS has no llm_fallback_adapter / tts_fallback_adapter / tts_stream_adapter spans (its adapters emit the plain request spans, which nest under the attempt's *_request_run), no wait_for_video_track, no judge_evaluation; rpc_handler may be a root, since without a session the SDK dispatches on a context carrying no span. MAY_OUTLIVE_PARENT is identical to Python's.

Testing

• New telemetry/testing/trace_schema.test.ts (9 tests): schema self-consistency; a sound trace passes; a misparented llm_request, an eou_detection outside its wait, an unknown span, a missing parent, a child outside its parent, a duplicated speech turn, a generation-count mismatch and a second trace id are each reported; the deliberate overruns pass; OTLP and in-memory sources agree; a full fake session is well-formed.
• Test hardening: the eou_wait full-session helper decided the turn 20 ms after the fake STT final was due, which under load opened a second user turn; the endpointing delay now leaves a 170 ms margin.
• No shipped code changes. Full agents suite green.

Review follow-ups

• lk.generation_count is checked against every generation event on the turn, matching feat(telemetry): one agent_turn span per speech handle #2500 where the count is the turn's and includes a discarded preemptive attempt's generation. The handoff itself is still validated separately: generations after the last preemptive_generation_discarded event must carry the finishing speech's id (a realtime tool reply continues the tool call's id, so it passes).
• An empty export is a violation (0 traces), not a sound trace.
• A generation event without lk.generation_id is a violation wherever it sits on the turn, a discarded attempt's before the handoff included: the framework always stamps one. Only the ownership check (the id is the finishing speech's) is limited to generations after the last handoff.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 474081a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chenghao-mou]

I don't think we install tsx automatically.

[devin-ai-integration]

Devin Review found 2 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)