Skip to content

openai_agents OTel bridge: started trace/span replicas corrupt OpenInference parents and leak per task #1852

Description

@DABH

OpenAIAgentsPlugin(use_otel_instrumentation=True) propagates Agents SDK context across Temporal boundaries by recreating the caller's trace and span on the receiving side (trace_context_from_header_contents, span_context_from_header_contents) and starting them (start_traces=True). Starting them makes OpenInferenceTracingProcessor register OTel spans for these replicas under the caller's Agents SDK ids (_root_spans[trace_id], _otel_spans[span_id]), with OTel ids seeded to match the caller's. The replicas are never finished.

The processor is one per process and keyed by Agents SDK id, so a replica overwrites the caller's own entry whenever caller and receiver share a process:

  • Worker running workflows and activities (the default). The activity-side replica of temporal:startActivity replaces the workflow-side span in _otel_spans. When the workflow-side span finishes, on_span_end ends the replica instead, whose OTel parent is the never-exported replica root. Every temporal:startActivity is exported with a parent_id that is not in the trace, and the model-call and tool subtrees render detached from the agent turns. max_cached_workflows=0 hides it, because each replay re-registers the workflow-side span after the overwrite.
  • Client and worker in one process. The replica trace root replaces _root_spans[trace_id], so on_trace_end ends the last replica instead of the client's root. Client-side spans lose their parent and attributes set on get_current_span() are lost.
  • Worker-side _root_spans and _otel_spans grow by one entry per task and are never released.

Repro (single process, dev server on 7233, no API key):

import asyncio, uuid
from datetime import timedelta
import opentelemetry.trace
from agents import Agent, Runner, trace as agents_trace
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from temporalio import workflow
from temporalio.client import Client
from temporalio.contrib.openai_agents import ModelActivityParameters
from temporalio.contrib.openai_agents.testing import AgentEnvironment, ResponseBuilders, TestModel
from temporalio.contrib.opentelemetry import create_tracer_provider
from temporalio.worker import UnsandboxedWorkflowRunner, Worker

@workflow.defn
class Hello:
    @workflow.run
    async def run(self) -> str:
        return str((await Runner.run(Agent(name="a", instructions="hi"), input="hi")).final_output)

async def main():
    exporter = InMemorySpanExporter()
    provider = create_tracer_provider(); provider.add_span_processor(SimpleSpanProcessor(exporter))
    opentelemetry.trace.set_tracer_provider(provider)
    client = await Client.connect("localhost:7233")
    async with AgentEnvironment(model=TestModel.returning_responses([ResponseBuilders.output_message("hello")]),
                                use_otel_instrumentation=True, add_temporal_spans=True,
                                model_params=ModelActivityParameters(start_to_close_timeout=timedelta(seconds=30))) as env:
        c = env.applied_on_client(client)
        tq = str(uuid.uuid4())
        async with Worker(c, task_queue=tq, workflows=[Hello], workflow_runner=UnsandboxedWorkflowRunner()):
            with env.openai_agents_plugin.tracing_context():
                with agents_trace("t"):
                    root = opentelemetry.trace.get_current_span()
                    await c.execute_workflow(Hello.run, id=str(uuid.uuid4()), task_queue=tq)
    spans = exporter.get_finished_spans()
    ids = {s.context.span_id for s in spans}
    print("client root exported:", root.get_span_context().span_id in ids)                      # False
    print("spans with a parent that was never exported:",
          [s.name for s in spans if s.parent and s.parent.span_id not in ids])                # ['temporal:startWorkflow:Hello']

asyncio.run(main())

In a worker-only process (client elsewhere) the same check flags every temporal:startActivity span instead; arize_tracing/verify_trace.py --scenario agents in temporalio/samples-python#365 reports it for the sample's agents scenario, and its worker.py --role workflows|activities split is the workaround.

Proposed fix: do not start the replicas. Set them as the current Agents SDK trace and span without start(), as the interceptor already does when start_traces=False, and give the OTel side its parent by attaching the propagated span context (otelTraceId/otelSpanId, or simply the W3C header the OpenTelemetryInterceptor already propagates) as the current OpenTelemetry context for the duration of the task. OpenInferenceTracingProcessor falls back to the current OTel context when a parent is not in its maps, so children parent correctly, the id seeding becomes unnecessary, and nothing accumulates per task.

Found while validating the Arize/Phoenix sample (temporalio/samples-python#365). Related but separate: #1855.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions