Repository navigation
Forge Runtime Tracing section for OpenTelemetry and Agent Reliability OTEL Labels - #242
Merged
Merged
Conversation
sairenchristianbuerano
approved these changes
Oct 5, 2026
…t-reliability-otel-labels trustabl forge now emits a "Runtime Tracing" section in the generated SKILL.md, right after the apply loop. It tells the coding model how to make the agent emit OpenTelemetry spans (Step 1), how to add the agent-reliability-otel-labels labels to those spans (Step 2), how to choose their values (Step 3), and what to check before moving on (Step 4). Selection: - Forge detects the language each SDK is declared in from the manifest that declares it (Python, TypeScript, Go) and keeps (category, language) pairs as detected, with no cross product. - --lang python,typescript,go adds a language; an explicit --policy or --lang widens the selection only for what detection did not already place. An unknown --lang value exits 1. - The section is omitted, with a one-line note on stderr, when no supported language is selected. Content: - One block per language, plus ten framework recipes copied from scripts that ran: Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph, MCP client sessions, OpenAI Agents SDK, Pydantic AI), TypeScript (OpenAI Agents SDK, Vercel AI SDK), Go (Google ADK). - Text is embedded markdown under internal/forge/tracingtext/, indexed by the languageBlocks and frameworkRecipes tables. Supporting changes: - Recon gains a go.mod needle for Google ADK for Go (recon-only name google-adk-go, deliberately not mapped in scanner.depNameToSDK) and a shared manifest-to-language table. - TemplateVersion is 3, so forge check reports older generated skills as outdated; its regenerate hint now says to pass the original --policy and --lang flags again. - A drift guard against agent-reliability-otel-labels: a vendored copy of its key manifest, a golden list of the binding identifiers the text uses, tests for both, and scripts/check-otel-sync.sh for checking against the real repo. The CI job for it is not added yet because that repo is not publicly cloneable. - README.md, ARCHITECTURE.md and COVERAGE.md describe the new behaviour.
ericksondelacruz
force-pushed
the
feat/forge-runtime-tracing
branch
from
October 6, 2026 00:38
4f8f6bd to
de6a241
Compare
…acing # Conflicts: # COVERAGE.md
…acing # Conflicts: # COVERAGE.md
…language Review fixes for the forge Runtime Tracing section. - ADK Go recipe: the agent, its tools and its callbacks are built once, so a Run they close over was shared by every run (the defect OBS-009 flags). The recipe now carries one run state per runner.Run on the context. Checked against ADK Go v1.7.0: 5 runs of one agent, 3 concurrent, now get 5 distinct run IDs and step indexes that restart at 0. - Rule sections are filtered to the languages each SDK is used in, through the scanner's own language gate (RuleDef.AppliesToLanguage). A Go-only ADK repo no longer gets Python and TypeScript rules; it gets the language-independent ones and a note that no Go rules exist yet. A category from --policy alone, or also declared in a manifest forge cannot place (Detection.Unplaced), keeps every rule. - TypeScript setup registers its provider globally, so framework instrumentation that starts spans on the global API is exported and labelled. - claude_skill alone gets no tracing section and no "pass --lang" note; --lang without --policy in a repo with no manifest now says --lang does not select a rule pack. - scripts/check-otel-sync.sh counts only declarations in source files, so a stale __pycache__/*.pyc, a comment, or rand.Read no longer hides a rename (the old check missed 7 of 9 declaration renames). - Cleanups: one flag-value parser for --policy and --lang; tracing recipes are selected once per section and each labels file is read once.
4 of 5 tasks
sairenchristianbuerano
approved these changes
Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forge Runtime Tracing section for OpenTelemetry and Agent Reliability OTEL Labels
Summary
trustabl forgenow tells the coding model how to make an agent observable atrun time. The generated
SKILL.mdgains a "Runtime Tracing" section builtaround
agent-reliability-otel-labels, the library that adds process labels(run identity, step order, tool-call fingerprints, attempt count, side-effect
class, exit reason) to the OpenTelemetry spans an agent already emits.
Forge also keeps each SDK's rule section to the languages that SDK is used in,
so a Go-only repo is no longer handed Python and TypeScript rules.
This PR changes
forgeonly. It adds no scan rules, changes no rule packs, andleaves
schema_versionunchanged. The scan-side label rules (OBS-006..012)ship separately.
Runtime Tracing section
It comes right after the apply loop, in four steps:
agent-reliability-otel-labels.attempt,side_effect,error_class,exit_reason).Content:
Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph, MCP client
sessions, OpenAI Agents SDK, Pydantic AI), TypeScript (OpenAI Agents SDK,
Vercel AI SDK), Go (Google ADK).
framework instrumentation starts its spans on the global provider.
Runperrunner.Runon the context.ADK builds the agent, its tools and its callbacks once, so a
Runtheyclosed over would be shared by every run, which is the defect
OBS-009flags.
internal/forge/tracingtext/.Selection:
declares it (
pyproject.toml/requirements.txt/Pipfile/poetry.lock→ Python,package.json→ TypeScript,go.mod→ Go). It keeps(category, language)pairs as detected, with no cross product.--lang python,typescript,goadds a language. An unknown value exits 1.language is selected. It is omitted without a note when
claude_skillisthe only category: skills are
SKILL.mdfiles, not agent code.Rule sections filtered by language
SelectRuleLanguagesdecides which languages each category's rules are keptfor.
GenerateCombinedthen drops each rule whoselanguagethe SDK is notused in, using the scanner's own language gate (
RuleDef.AppliesToLanguage).TypeScript keeps JavaScript rules, and a rule with no language is always kept.
under that SDK's heading. Today that is a Go-only Google ADK repo, which gets
the two language-independent ADK rules and the note.
used in: one given only through
--policy, or one also declared in amanifest forge cannot place, such as
composer.json(Detection.Unplaced).--langadds its language to every listed category, so it can bring rulesback but never remove them.
This also changes output for single-language repos of other SDKs. A
Python-only OpenAI Agents repo now gets its 33 Python rules and 2
language-independent rules, not the TypeScript rules as well.
CLI
--langhelp says it does not select a rule pack, so a repo with nomanifest yet needs
--policytoo. When--langis given and nothing isdetected, the exit-1 message says the same.
forge check's regenerate hint reminds you to pass the--policyand--langflags used originally, since the stamp does not record them.Supporting changes
go.modneedle for Google ADK for Go. Its recon-only namegoogle-adk-gois not mapped to an SDK forscan, so it never emitsMETA-002.TemplateVersionis 3, soforge checkreports older generated skills asoutdated.
golden list of the 36 identifiers the text uses, tests for both, and
scripts/check-otel-sync.shfor checking against the real repo. The scriptcounts only declarations in source files, so a stale
__pycache__/*.pyc, acomment, or another package's function with the same name (
rand.Read)cannot hide a rename.
How to verify
go test ./...passes.OTEL_REPO=../agent-reliability-otel-labels scripts/check-otel-sync.shpasses: 36 identifiers checked.
RULES_REPO=<agent-reliability-rules at main> scripts/check-rules-sync.shpasses (128 files). This PR touches no rule files.
Runmust be createdonly in a per-run function whose result reaches
runner.Run;--langCLI paths.committed:
SKILL.md,ran on one agent built once, 5 runs with 3 at once. It gave 5 distinct run
IDs, and step indexes restart at 0 in every run. The pattern this
replaces gave 1 shared run ID.
SKILL.md, a span started through the global API is exported andlabelled. Without the global registration it is dropped.
check-otel-sync.sh: each identifier was renamed in a copy of thelibrary. The script catches all 36 renames and 9 declaration-only renames;
the earlier version missed 6 and 7 of them.
Known limits
every label call is a silent no-op.
recipe gets its language block only, and is told to turn on its own
OpenTelemetry instrumentation.
stops iterating a run's events, so a run that stops on its first failure
carries no
trustabl.exit_reason. The recipe text says so.language-independent ADK rules only.
Follow-ups
check-otel-sync.sh. The library repo is now public.