Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,12 @@ extra:
| LlamaIndex | `pip install "agentx-python[llamaindex]"` | `AgentXLlamaIndexHandler` |
| AutoGen | `pip install "agentx-python[autogen]"` | `AgentXAutoGenObserver` |

Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework required.
Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework
required. Tracing is **platform agnostic**: each integration stamps its platform label
automatically, a plain trace auto-detects the one orchestration framework imported in the
process, and `framework="any-name"` labels platforms AgentX has never heard of - the label
drives the dashboard's framework filter and Monitor's Platforms chart. See
[Platform detection](TRACING.md#platform-detection).

Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces - see [TRACING.md](TRACING.md) for the full pattern.

Expand Down
40 changes: 39 additions & 1 deletion TRACING.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ with tracer.trace("agent-name", framework="langchain") as span:
| Parameter | Type | Required | Description |
|---|---|---|---|
| `name` | `str` | ✓ | Agent or operation label shown in the UI |
| `framework` | `str` | - | Framework identifier: `"langchain"`, `"crewai"`, `"openai-agents"`, `"anthropic"`, or custom |
| `framework` | `str` | - | Platform label - any string, including custom platform names. Auto-filled when omitted: see [Platform detection](#platform-detection) |
| `model` | `str` | - | LLM model used, e.g. `"gpt-4o"`, `"claude-sonnet-4-6"` |
| `session_id` | `str` | - | Groups traces from the same user session or thread |
| `metadata` | `dict` | - | Arbitrary key-value metadata (not indexed, max 16 KB) |
Expand All @@ -113,6 +113,44 @@ with tracer.trace("agent-name", framework="langchain") as span:

---

## Platform detection

Tracing is **platform agnostic**: every trace carries a platform label, and any agent runtime
works. The label resolves in priority order:

1. **Explicit** - `framework="..."` on `trace()`. Any string is valid, including platforms
AgentX has no integration for: `framework="my-inhouse-runner"` charts and filters like any
built-in name. (The engine folds labels to lowercase, so `"LangChain"` and `"langchain"`
are one platform.)
2. **Integration** - every AgentX integration stamps its literal automatically, no parameter
needed:

| Integration | Label |
|---|---|
| `AgentXCallbackHandler` (LangChain/LangGraph) | `langchain` |
| `AgentXCrewObserver` | `crewai` |
| `AgentXTracingProcessor` (OpenAI Agents SDK) | `openai-agents` |
| `patch_openai_client` | `openai` |
| `patch_anthropic_client` | `anthropic` |
| `patch_genai_client` | `google-genai` |
| `AgentXADKPlugin` | `google-adk` |
| `AgentXLiteLLMLogger` | `litellm` |
| `AgentXLlamaIndexHandler` | `llamaindex` |
| `AgentXAutoGenObserver` | `autogen` |
| `MoveworksImporter` | `moveworks` |
| `DatabricksTraceImporter` | `databricks` |

3. **Auto-detection** - a plain `@tracer.trace(...)` with neither of the above looks at which
known orchestration framework is actually imported in the process (LangChain/LangGraph,
CrewAI, LlamaIndex, AutoGen, OpenAI Agents SDK, Google ADK, Semantic Kernel, Haystack,
Pydantic AI, smolagents, DSPy) and labels the span when exactly one is loaded. Ambiguous or
unknown means no label - the trace still ingests fine and buckets as "Other / custom" in the
dashboard, never mislabeled.

The label powers the Live Traces framework filter and Monitor's **Platforms** chart
(`GET /agent-monitoring/metrics` - `byFramework` buckets, `frameworks` totals, and a
`framework=` filter).

## Framework examples

### LangChain
Expand Down
3 changes: 3 additions & 0 deletions agentx/integrations/_traced_call.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,9 @@ def finish_llm_call(
input=input_repr,
output=output,
model=model,
# Stamp the provider literal on a span opened without one (adoption keeps an
# explicit framework= or a framework integration's label winning over this).
framework=framework,
input_tokens=input_tokens,
output_tokens=output_tokens,
cache_read_tokens=cache_read_tokens,
Expand Down
22 changes: 22 additions & 0 deletions agentx/monitor/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,28 @@ def kpis(self, window: str = "7d") -> dict:
plus deltas vs the prior window and the run-outcome breakdown."""
return self._request("GET", "/kpis", params={"window": window})

def metrics(
self,
window: str = "1d",
*,
agent: Optional[str] = None,
model: Optional[str] = None,
tool: Optional[str] = None,
framework: Optional[str] = None,
status: Optional[str] = None,
) -> dict:
"""The Monitor metrics grid's data over a window ("1h".."90d") - bucketed spans by kind,
latency percentiles, tokens/cost, tool executions and failures, and platform attribution
(`frameworks` window totals + per-bucket `byFramework` - the Platforms chart). Optional
filters scope every number the way the dashboard's filter chips do; `framework` matches
the platform label traces carry (see TRACING.md's Platform detection), with "other"
selecting unlabeled traffic."""
params = {"window": window}
for key, value in (("agent", agent), ("model", model), ("tool", tool), ("framework", framework), ("status", status)):
if value is not None:
params[key] = value
return self._request("GET", "/metrics", params=params)

def topics(self, window: str = "7d") -> dict:
"""The Topics view's data over a window ("24h", "7d", "30d"): LLM-classified themes of
sampled production traffic with per-topic counts and sentiment. Empty until Topics is
Expand Down
64 changes: 64 additions & 0 deletions agentx/tracing/framework_detect.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""Best-effort agent-framework auto-detection (the "platform agnostic" story).

A span whose framework was neither passed explicitly (``tracer.trace(...,
framework="...")``) nor adopted from a framework integration (callback handler,
observer, patched client - see ``_TraceSpan._captured_framework``) gets labeled
by looking at which known ORCHESTRATION framework is actually imported in this
process. ``sys.modules`` is the signal - imported, not merely installed - so a
machine with ten frameworks pip-installed but one in use still resolves.

Only unambiguous answers are given: zero or more than one known framework
loaded means ``None``, and the span goes out unlabeled rather than mislabeled.
The user's explicit ``framework=`` always wins, including totally custom names
for platforms this table has never heard of.

Raw provider SDKs (openai, anthropic, google-genai, ...) are deliberately NOT
in this table: they are transitive dependencies of nearly every framework, so
their presence says nothing about what orchestrates the agent - and their
patched-client integrations already stamp the provider literal on the spans
they create.
"""

from __future__ import annotations

import sys
from typing import Optional

# Top-level module name -> the wire literal the matching integration emits.
# Multiple modules may map to one literal (langgraph is the LangChain family).
_ORCHESTRATOR_MODULES = {
"langchain": "langchain",
"langchain_core": "langchain",
"langgraph": "langchain",
"crewai": "crewai",
"llama_index": "llamaindex",
"autogen": "autogen",
"autogen_agentchat": "autogen",
"agents": "openai-agents", # the OpenAI Agents SDK's import name
"google.adk": "google-adk",
"semantic_kernel": "semantic-kernel",
"haystack": "haystack",
"pydantic_ai": "pydantic-ai",
"smolagents": "smolagents",
"dspy": "dspy",
}


def _looks_like_openai_agents_sdk() -> bool:
# "agents" is a name any user package could claim - only trust it when the
# OpenAI Agents SDK's own submodules are loaded alongside it.
return "agents.run" in sys.modules or "agents.tracing" in sys.modules


def detect_framework() -> Optional[str]:
"""The single unambiguous orchestration framework imported right now, else None."""
found: set = set()
for module, literal in _ORCHESTRATOR_MODULES.items():
if module not in sys.modules:
continue
if module == "agents" and not _looks_like_openai_agents_sdk():
continue
found.add(literal)
if len(found) > 1:
return None
return found.pop() if len(found) == 1 else None
48 changes: 39 additions & 9 deletions agentx/tracing/tracer.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from agentx.tracing.ingest_client import IngestClient
from agentx.tracing.ci_types import CIRun, CIRunResult, CIRunStatus, CIQuestionScore
from agentx.tracing.eval_scope import EVAL_RUN_SOURCE, current_eval_run_id
from agentx.tracing.framework_detect import detect_framework

F = TypeVar("F", bound=Callable[..., Any])

Expand Down Expand Up @@ -125,6 +126,10 @@ def __init__(
# Adopted from a merged child run (e.g. AgentXCallbackHandler) when this span itself
# wasn't opened with an explicit framework= - see _merge_child_run below.
self._captured_framework: Optional[str] = None
# Best-effort auto-detection (framework_detect.py), resolved once at span open so child
# spans emitted mid-flight carry the same label the root will. Lowest precedence:
# explicit framework= > integration adoption > this. None when ambiguous.
self._detected_framework: Optional[str] = None if framework else detect_framework()
self._input_tokens: int = 0
self._output_tokens: int = 0
# Subsets of _input_tokens (not additional tokens) - a prompt-caching write/read, when the
Expand Down Expand Up @@ -191,7 +196,7 @@ def __exit__(self, exc_type, exc_val, tb):
latency_ms=latency_ms,
error=self._error,
metadata=metadata,
framework=self._framework or self._captured_framework,
framework=self._framework or self._captured_framework or self._detected_framework or detect_framework(),
model=self._model or self._captured_model,
tool_calls=self.tool_calls or None,
session_id=self._session_id,
Expand Down Expand Up @@ -235,14 +240,18 @@ def _record_llm_call(
input: Any = None,
output: Any = None,
model: Optional[str] = None,
framework: Optional[str] = None,
input_tokens: Optional[int] = None,
output_tokens: Optional[int] = None,
cache_read_tokens: Optional[int] = None,
cache_write_tokens: Optional[int] = None,
) -> None:
"""Record one LLM-call child span (e.g. one patched Anthropic call) under this span -
name left unset so _merge_child_run auto-numbers it "LLM Call N"."""
name left unset so _merge_child_run auto-numbers it "LLM Call N". ``framework`` lets the
patched client stamp its provider literal on a span the user opened without one - the
adoption in _merge_child_run keeps explicit/integration labels winning."""
self._merge_child_run(
framework=framework,
execution_steps=[{
"duration_ms": duration_ms,
"start_time": start_time,
Expand Down Expand Up @@ -301,7 +310,7 @@ def child_span(
child = _TraceSpan(
tracer=self._tracer,
name=name,
framework=framework or self._framework or self._captured_framework,
framework=framework or self._framework or self._captured_framework or self._detected_framework,
model=model,
session_id=self._session_id,
)
Expand Down Expand Up @@ -397,6 +406,13 @@ def _merge_child_run(
under this span).
"""
with self._merge_lock:
# Adopt framework/model BEFORE emitting child spans: child_span resolves its
# framework from this span's fields, so adopting after the emission loops used to
# send every CrewAI/AutoGen child out unlabeled while only the root got stamped.
if model and not self._captured_model:
self._captured_model = model
if framework and not self._captured_framework:
self._captured_framework = framework
for step in [] if not emit_steps else (execution_steps or []):
self._child_span_count += 1
self.child_span(
Expand Down Expand Up @@ -463,10 +479,6 @@ def _merge_child_run(
self.input = input
if output is not None:
self.output = output
if model and not self._captured_model:
self._captured_model = model
if framework and not self._captured_framework:
self._captured_framework = framework
if input_tokens:
self._input_tokens += input_tokens
if output_tokens:
Expand Down Expand Up @@ -520,7 +532,8 @@ def wrapper(*args, **kwargs):
# to be called from inside another active span.
span = self._tracer.trace(
self.name, metadata=self._metadata, framework=self._framework, model=self._model,
session_id=self._session_id,
session_id=self._session_id, sync=self._sync, monitor=self._monitor,
pattern_ids=self._pattern_ids, agent_id=self._agent_id, span_kind=self._span_kind,
)
span.__enter__()
try:
Expand All @@ -542,7 +555,8 @@ async def wrapper(*args, **kwargs):
# See _wrap_sync's comment - same "fresh span per call" reasoning applies here.
span = self._tracer.trace(
self.name, metadata=self._metadata, framework=self._framework, model=self._model,
session_id=self._session_id,
session_id=self._session_id, sync=self._sync, monitor=self._monitor,
pattern_ids=self._pattern_ids, agent_id=self._agent_id, span_kind=self._span_kind,
)
span.__enter__()
try:
Expand Down Expand Up @@ -911,6 +925,22 @@ def trace(

with client.tracer.trace("support-agent", agent_id="ag_123", sync=True) as span:
span.output = call_llm(...)

``framework`` names the platform the agent runs on - tracing is platform agnostic, and
this label is how the dashboard's framework filter and Monitor's Platforms chart group
traffic. Three ways it gets set, strongest first:

1. **Explicit**: ``framework="langchain"`` - any string works, including platforms
AgentX has no integration for (``framework="my-inhouse-runner"``).
2. **Integration**: every integration stamps its own literal automatically -
``langchain``, ``crewai``, ``openai-agents``, ``openai``, ``anthropic``,
``google-genai``, ``google-adk``, ``litellm``, ``llamaindex``, ``autogen``,
``moveworks``, ``databricks``.
3. **Auto-detection**: with neither of the above, the SDK labels the span with the one
known orchestration framework imported in the process (LangChain/LangGraph, CrewAI,
LlamaIndex, AutoGen, OpenAI Agents SDK, Google ADK, Semantic Kernel, Haystack,
Pydantic AI, smolagents, DSPy). Ambiguous (several imported) or unknown -> the span
goes out unlabeled rather than mislabeled.
"""
return _TraceSpan(
tracer=self,
Expand Down
Loading
Loading