OpenAIAgentsPlugin(use_otel_instrumentation=True) propagates Agents SDK context across Temporal boundaries by recreating the caller's trace and span on the receiving side (trace_context_from_header_contents, span_context_from_header_contents) and starting them (start_traces=True). Starting them makes OpenInferenceTracingProcessor register OTel spans for these replicas under the caller's Agents SDK ids (_root_spans[trace_id], _otel_spans[span_id]), with OTel ids seeded to match the caller's. The replicas are never finished.
The processor is one per process and keyed by Agents SDK id, so a replica overwrites the caller's own entry whenever caller and receiver share a process:
- Worker running workflows and activities (the default). The activity-side replica of
temporal:startActivity replaces the workflow-side span in _otel_spans. When the workflow-side span finishes, on_span_end ends the replica instead, whose OTel parent is the never-exported replica root. Every temporal:startActivity is exported with a parent_id that is not in the trace, and the model-call and tool subtrees render detached from the agent turns. max_cached_workflows=0 hides it, because each replay re-registers the workflow-side span after the overwrite.
- Client and worker in one process. The replica trace root replaces
_root_spans[trace_id], so on_trace_end ends the last replica instead of the client's root. Client-side spans lose their parent and attributes set on get_current_span() are lost.
- Worker-side
_root_spans and _otel_spans grow by one entry per task and are never released.
Repro (single process, dev server on 7233, no API key):
import asyncio, uuid
from datetime import timedelta
import opentelemetry.trace
from agents import Agent, Runner, trace as agents_trace
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from temporalio import workflow
from temporalio.client import Client
from temporalio.contrib.openai_agents import ModelActivityParameters
from temporalio.contrib.openai_agents.testing import AgentEnvironment, ResponseBuilders, TestModel
from temporalio.contrib.opentelemetry import create_tracer_provider
from temporalio.worker import UnsandboxedWorkflowRunner, Worker
@workflow.defn
class Hello:
@workflow.run
async def run(self) -> str:
return str((await Runner.run(Agent(name="a", instructions="hi"), input="hi")).final_output)
async def main():
exporter = InMemorySpanExporter()
provider = create_tracer_provider(); provider.add_span_processor(SimpleSpanProcessor(exporter))
opentelemetry.trace.set_tracer_provider(provider)
client = await Client.connect("localhost:7233")
async with AgentEnvironment(model=TestModel.returning_responses([ResponseBuilders.output_message("hello")]),
use_otel_instrumentation=True, add_temporal_spans=True,
model_params=ModelActivityParameters(start_to_close_timeout=timedelta(seconds=30))) as env:
c = env.applied_on_client(client)
tq = str(uuid.uuid4())
async with Worker(c, task_queue=tq, workflows=[Hello], workflow_runner=UnsandboxedWorkflowRunner()):
with env.openai_agents_plugin.tracing_context():
with agents_trace("t"):
root = opentelemetry.trace.get_current_span()
await c.execute_workflow(Hello.run, id=str(uuid.uuid4()), task_queue=tq)
spans = exporter.get_finished_spans()
ids = {s.context.span_id for s in spans}
print("client root exported:", root.get_span_context().span_id in ids) # False
print("spans with a parent that was never exported:",
[s.name for s in spans if s.parent and s.parent.span_id not in ids]) # ['temporal:startWorkflow:Hello']
asyncio.run(main())
In a worker-only process (client elsewhere) the same check flags every temporal:startActivity span instead; arize_tracing/verify_trace.py --scenario agents in temporalio/samples-python#365 reports it for the sample's agents scenario, and its worker.py --role workflows|activities split is the workaround.
Proposed fix: do not start the replicas. Set them as the current Agents SDK trace and span without start(), as the interceptor already does when start_traces=False, and give the OTel side its parent by attaching the propagated span context (otelTraceId/otelSpanId, or simply the W3C header the OpenTelemetryInterceptor already propagates) as the current OpenTelemetry context for the duration of the task. OpenInferenceTracingProcessor falls back to the current OTel context when a parent is not in its maps, so children parent correctly, the id seeding becomes unnecessary, and nothing accumulates per task.
Found while validating the Arize/Phoenix sample (temporalio/samples-python#365). Related but separate: #1855.
OpenAIAgentsPlugin(use_otel_instrumentation=True)propagates Agents SDK context across Temporal boundaries by recreating the caller's trace and span on the receiving side (trace_context_from_header_contents,span_context_from_header_contents) and starting them (start_traces=True). Starting them makesOpenInferenceTracingProcessorregister OTel spans for these replicas under the caller's Agents SDK ids (_root_spans[trace_id],_otel_spans[span_id]), with OTel ids seeded to match the caller's. The replicas are never finished.The processor is one per process and keyed by Agents SDK id, so a replica overwrites the caller's own entry whenever caller and receiver share a process:
temporal:startActivityreplaces the workflow-side span in_otel_spans. When the workflow-side span finishes,on_span_endends the replica instead, whose OTel parent is the never-exported replica root. Everytemporal:startActivityis exported with aparent_idthat is not in the trace, and the model-call and tool subtrees render detached from the agent turns.max_cached_workflows=0hides it, because each replay re-registers the workflow-side span after the overwrite._root_spans[trace_id], soon_trace_endends the last replica instead of the client's root. Client-side spans lose their parent and attributes set onget_current_span()are lost._root_spansand_otel_spansgrow by one entry per task and are never released.Repro (single process, dev server on 7233, no API key):
In a worker-only process (client elsewhere) the same check flags every
temporal:startActivityspan instead;arize_tracing/verify_trace.py --scenario agentsin temporalio/samples-python#365 reports it for the sample's agents scenario, and itsworker.py --role workflows|activitiessplit is the workaround.Proposed fix: do not start the replicas. Set them as the current Agents SDK trace and span without
start(), as the interceptor already does whenstart_traces=False, and give the OTel side its parent by attaching the propagated span context (otelTraceId/otelSpanId, or simply the W3C header theOpenTelemetryInterceptoralready propagates) as the current OpenTelemetry context for the duration of the task.OpenInferenceTracingProcessorfalls back to the current OTel context when a parent is not in its maps, so children parent correctly, the id seeding becomes unnecessary, and nothing accumulates per task.Found while validating the Arize/Phoenix sample (temporalio/samples-python#365). Related but separate: #1855.