Skip to content

Forge Runtime Tracing section for OpenTelemetry and Agent Reliability OTEL Labels - #242

Merged
ericksondelacruz merged 4 commits into
mainfrom
feat/forge-runtime-tracing
Oct 12, 2026
Merged

ericksondelacruz merged 4 commits into
mainfrom
feat/forge-runtime-tracing

Conversation

@ericksondelacruz

@ericksondelacruz ericksondelacruz commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Forge Runtime Tracing section for OpenTelemetry and Agent Reliability OTEL Labels

Summary

trustabl forge now tells the coding model how to make an agent observable at
run time. The generated SKILL.md gains a "Runtime Tracing" section built
around agent-reliability-otel-labels, the library that adds process labels
(run identity, step order, tool-call fingerprints, attempt count, side-effect
class, exit reason) to the OpenTelemetry spans an agent already emits.

Forge also keeps each SDK's rule section to the languages that SDK is used in,
so a Go-only repo is no longer handed Python and TypeScript rules.

This PR changes forge only. It adds no scan rules, changes no rule packs, and
leaves schema_version unchanged. The scan-side label rules (OBS-006..012)
ship separately.

Runtime Tracing section

It comes right after the apply loop, in four steps:

  1. Emit spans with OpenTelemetry.
  2. Add the labels with agent-reliability-otel-labels.
  3. Choose the values (attempt, side_effect, error_class, exit_reason).
  4. Check before moving on.

Content:

  • One block per language (Python, TypeScript, Go), plus ten framework recipes:
    Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph, MCP client
    sessions, OpenAI Agents SDK, Pydantic AI), TypeScript (OpenAI Agents SDK,
    Vercel AI SDK), Go (Google ADK).
  • Each language's setup registers its tracer provider globally, because
    framework instrumentation starts its spans on the global provider.
  • The Google ADK Go recipe keeps one Run per runner.Run on the context.
    ADK builds the agent, its tools and its callbacks once, so a Run they
    closed over would be shared by every run, which is the defect OBS-009
    flags.
  • The text is embedded markdown under internal/forge/tracingtext/.

Selection:

  • Forge detects the language each SDK is declared in from the manifest that
    declares it (pyproject.toml / requirements.txt / Pipfile /
    poetry.lock → Python, package.json → TypeScript, go.mod → Go). It keeps
    (category, language) pairs as detected, with no cross product.
  • --lang python,typescript,go adds a language. An unknown value exits 1.
  • The section is omitted, with a one-line stderr note, when no supported
    language is selected. It is omitted without a note when claude_skill is
    the only category: skills are SKILL.md files, not agent code.

Rule sections filtered by language

SelectRuleLanguages decides which languages each category's rules are kept
for. GenerateCombined then drops each rule whose language the SDK is not
used in, using the scanner's own language gate (RuleDef.AppliesToLanguage).
TypeScript keeps JavaScript rules, and a rule with no language is always kept.

  • A listed language that no rule of the pack targets gets a one-line note
    under that SDK's heading. Today that is a Go-only Google ADK repo, which gets
    the two language-independent ADK rules and the note.
  • A category keeps every rule when forge does not know every language it is
    used in: one given only through --policy, or one also declared in a
    manifest forge cannot place, such as composer.json (Detection.Unplaced).
  • --lang adds its language to every listed category, so it can bring rules
    back but never remove them.

This also changes output for single-language repos of other SDKs. A
Python-only OpenAI Agents repo now gets its 33 Python rules and 2
language-independent rules, not the TypeScript rules as well.

CLI

  • --lang help says it does not select a rule pack, so a repo with no
    manifest yet needs --policy too. When --lang is given and nothing is
    detected, the exit-1 message says the same.
  • forge check's regenerate hint reminds you to pass the --policy and
    --lang flags used originally, since the stamp does not record them.

Supporting changes

  • Recon gains a go.mod needle for Google ADK for Go. Its recon-only name
    google-adk-go is not mapped to an SDK for scan, so it never emits
    META-002.
  • TemplateVersion is 3, so forge check reports older generated skills as
    outdated.
  • A drift guard against the library: a vendored copy of its key manifest, a
    golden list of the 36 identifiers the text uses, tests for both, and
    scripts/check-otel-sync.sh for checking against the real repo. The script
    counts only declarations in source files, so a stale __pycache__/*.pyc, a
    comment, or another package's function with the same name (rand.Read)
    cannot hide a rename.

How to verify

  • go test ./... passes.
  • OTEL_REPO=../agent-reliability-otel-labels scripts/check-otel-sync.sh
    passes: 36 identifiers checked.
  • RULES_REPO=<agent-reliability-rules at main> scripts/check-rules-sync.sh
    passes (128 files). This PR touches no rule files.
  • Unit tests cover:
    • the ADK Go recipe: its Go block is parsed, and the Run must be created
      only in a per-run function whose result reaches runner.Run;
    • global provider registration in all three language setups;
    • the rule-language selection and filter, and the coverage note;
    • the skill-only and --lang CLI paths.
  • Run by hand against the published 0.1.0 API; these harnesses are not
    committed:
    • ADK Go v1.7.0: the recipe's code, taken from a generated SKILL.md,
      ran on one agent built once, 5 runs with 3 at once. It gave 5 distinct run
      IDs, and step indexes restart at 0 in every run. The pattern this
      replaces gave 1 shared run ID.
    • OpenTelemetry JS 2.x: with the TypeScript setup from a generated
      SKILL.md, a span started through the global API is exported and
      labelled. Without the global registration it is dropped.
    • check-otel-sync.sh: each identifier was renamed in a copy of the
      library. The script catches all 36 renames and 9 declaration-only renames;
      the earlier version missed 6 and 7 of them.

Known limits

  • Forge text proves nothing at run time. With no current span inside a tool,
    every label call is a silent no-op.
  • Non-OpenTelemetry tracing stacks are not covered. A framework with no
    recipe gets its language block only, and is told to turn on its own
    OpenTelemetry instrumentation.
  • In the ADK Go recipe, ADK skips the after-agent callback when the caller
    stops iterating a run's events, so a run that stops on its first failure
    carries no trustabl.exit_reason. The recipe text says so.
  • No Go-specific Google ADK rules exist, so a Go ADK repo gets the
    language-independent ADK rules only.

Follow-ups

  • Add a CI job for check-otel-sync.sh. The library repo is now public.

…t-reliability-otel-labels

trustabl forge now emits a "Runtime Tracing" section in the generated
SKILL.md, right after the apply loop. It tells the coding model how to make
the agent emit OpenTelemetry spans (Step 1), how to add the agent-reliability-otel-labels
labels to those spans (Step 2), how to choose their values (Step 3), and
what to check before moving on (Step 4).

Selection:
- Forge detects the language each SDK is declared in from the manifest
  that declares it (Python, TypeScript, Go) and keeps (category, language)
  pairs as detected, with no cross product.
- --lang python,typescript,go adds a language; an explicit --policy or
  --lang widens the selection only for what detection did not already
  place. An unknown --lang value exits 1.
- The section is omitted, with a one-line note on stderr, when no
  supported language is selected.

Content:
- One block per language, plus ten framework recipes copied from scripts
  that ran: Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph,
  MCP client sessions, OpenAI Agents SDK, Pydantic AI), TypeScript
  (OpenAI Agents SDK, Vercel AI SDK), Go (Google ADK).
- Text is embedded markdown under internal/forge/tracingtext/, indexed by
  the languageBlocks and frameworkRecipes tables.

Supporting changes:
- Recon gains a go.mod needle for Google ADK for Go (recon-only name
  google-adk-go, deliberately not mapped in scanner.depNameToSDK) and a
  shared manifest-to-language table.
- TemplateVersion is 3, so forge check reports older generated skills as
  outdated; its regenerate hint now says to pass the original --policy and
  --lang flags again.
- A drift guard against agent-reliability-otel-labels: a vendored copy of its key
  manifest, a golden list of the binding identifiers the text uses, tests
  for both, and scripts/check-otel-sync.sh for checking against the real
  repo. The CI job for it is not added yet because that repo is not
  publicly cloneable.
- README.md, ARCHITECTURE.md and COVERAGE.md describe the new behaviour.
@ericksondelacruz
ericksondelacruz force-pushed the feat/forge-runtime-tracing branch from 4f8f6bd to de6a241 Compare October 6, 2026 00:38
…language

Review fixes for the forge Runtime Tracing section.

- ADK Go recipe: the agent, its tools and its callbacks are built once, so a
  Run they close over was shared by every run (the defect OBS-009 flags).
  The recipe now carries one run state per runner.Run on the context.
  Checked against ADK Go v1.7.0: 5 runs of one agent, 3 concurrent, now get
  5 distinct run IDs and step indexes that restart at 0.
- Rule sections are filtered to the languages each SDK is used in, through
  the scanner's own language gate (RuleDef.AppliesToLanguage). A Go-only ADK
  repo no longer gets Python and TypeScript rules; it gets the
  language-independent ones and a note that no Go rules exist yet. A category
  from --policy alone, or also declared in a manifest forge cannot place
  (Detection.Unplaced), keeps every rule.
- TypeScript setup registers its provider globally, so framework
  instrumentation that starts spans on the global API is exported and labelled.
- claude_skill alone gets no tracing section and no "pass --lang" note;
  --lang without --policy in a repo with no manifest now says --lang does not
  select a rule pack.
- scripts/check-otel-sync.sh counts only declarations in source files, so a
  stale __pycache__/*.pyc, a comment, or rand.Read no longer hides a rename
  (the old check missed 7 of 9 declaration renames).
- Cleanups: one flag-value parser for --policy and --lang; tracing recipes
  are selected once per section and each labels file is read once.
@ericksondelacruz
ericksondelacruz merged commit ee440bf into main Oct 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants