Repository navigation
Add OBS-006...008 for agent-reliability-otel-labels - #258
Merged
Merged
Conversation
…t-reliability-otel-labels trustabl forge now emits a "Runtime Tracing" section in the generated SKILL.md, right after the apply loop. It tells the coding model how to make the agent emit OpenTelemetry spans (Step 1), how to add the agent-reliability-otel-labels labels to those spans (Step 2), how to choose their values (Step 3), and what to check before moving on (Step 4). Selection: - Forge detects the language each SDK is declared in from the manifest that declares it (Python, TypeScript, Go) and keeps (category, language) pairs as detected, with no cross product. - --lang python,typescript,go adds a language; an explicit --policy or --lang widens the selection only for what detection did not already place. An unknown --lang value exits 1. - The section is omitted, with a one-line note on stderr, when no supported language is selected. Content: - One block per language, plus ten framework recipes copied from scripts that ran: Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph, MCP client sessions, OpenAI Agents SDK, Pydantic AI), TypeScript (OpenAI Agents SDK, Vercel AI SDK), Go (Google ADK). - Text is embedded markdown under internal/forge/tracingtext/, indexed by the languageBlocks and frameworkRecipes tables. Supporting changes: - Recon gains a go.mod needle for Google ADK for Go (recon-only name google-adk-go, deliberately not mapped in scanner.depNameToSDK) and a shared manifest-to-language table. - TemplateVersion is 3, so forge check reports older generated skills as outdated; its regenerate hint now says to pass the original --policy and --lang flags again. - A drift guard against agent-reliability-otel-labels: a vendored copy of its key manifest, a golden list of the binding identifiers the text uses, tests for both, and scripts/check-otel-sync.sh for checking against the real repo. The CI job for it is not added yet because that repo is not publicly cloneable. - README.md, ARCHITECTURE.md and COVERAGE.md describe the new behaviour.
…acing # Conflicts: # COVERAGE.md
…acing # Conflicts: # COVERAGE.md
…language Review fixes for the forge Runtime Tracing section. - ADK Go recipe: the agent, its tools and its callbacks are built once, so a Run they close over was shared by every run (the defect OBS-009 flags). The recipe now carries one run state per runner.Run on the context. Checked against ADK Go v1.7.0: 5 runs of one agent, 3 concurrent, now get 5 distinct run IDs and step indexes that restart at 0. - Rule sections are filtered to the languages each SDK is used in, through the scanner's own language gate (RuleDef.AppliesToLanguage). A Go-only ADK repo no longer gets Python and TypeScript rules; it gets the language-independent ones and a note that no Go rules exist yet. A category from --policy alone, or also declared in a manifest forge cannot place (Detection.Unplaced), keeps every rule. - TypeScript setup registers its provider globally, so framework instrumentation that starts spans on the global API is exported and labelled. - claude_skill alone gets no tracing section and no "pass --lang" note; --lang without --policy in a repo with no manifest now says --lang does not select a rule pack. - scripts/check-otel-sync.sh counts only declarations in source files, so a stale __pycache__/*.pyc, a comment, or rand.Read no longer hides a rename (the old check missed 7 of 9 declaration renames). - Cleanups: one flag-value parser for --policy and --lang; tracing recipes are selected once per section and each labels file is read once.
…-rules The rules repo was renamed from trustabl-rules. check-rules-sync.sh now defaults RULES_REPO to ../agent-reliability-rules, and the "trustabl rules validate" example uses the new name.
Three repo-scope rules check whether an instrumented agent uses agent-reliability-otel-labels. They form one ladder; each fires at the first missing rung, so at most one fires per repo: - OBS-006 (low): tracing is initialized, but the library is neither declared nor imported. - OBS-007 (medium): the library is installed, but no Run's mark_start is called. - OBS-008 (medium): a run is started, but mark_run_end is missing, or the repo has tools and no mark_tool_span call. All three are gated on repo_observability_initialized: true. Engine changes: - Recon: RepoProfile.OtelLabelsDeclaredIn records the Python and npm package names, kept separate from ObsDeps. - Inventory: analysis.DiscoverOtelLabels emits OtelLabelsSignal records (import, run_start, tool_span, run_end) for Python and TypeScript/ JavaScript, kept separate from ObservabilitySignals so the library cannot satisfy its own prerequisite. The Go binding is not inspected. - Predicates: repo_otel_labels_installed, repo_otel_labels_initialized, repo_otel_labels_implemented. - SupportedSchemaVersion and the fixture manifest go from 20 to 21. Not covered: whether the labels are correct (a shared Run, a constant attempt, labelling outside finally, an understated side_effect, unmarked handoffs), hand-written trustabl.* keys, and aliased imports. The fixture mirrors agent-reliability-rules branch feat/otel-labels-rules. Do not merge the rules before the library is published on PyPI and npm.
ericksondelacruz
requested review from
ivanpaghubasan,
jhumel-code and
sairenchristianbuerano
October 11, 2026 15:36
sairenchristianbuerano
approved these changes
Oct 11, 2026
ivanpaghubasan
approved these changes
Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three repo-scope rules that check whether an instrumented agent uses
agent-reliability-otel-labels
to label the spans it already emits. They form one ladder; each fires at the first
missing rung, so at most one fires per repo:
Run'smark_startis ever calledmark_run_endis missing, or the repo has tools and nomark_tool_spancallAll three are gated on
repo_observability_initialized: true, because the libraryonly labels spans that something else already creates.
schema_version20 → 21.Engine changes
RepoProfile.OtelLabelsDeclaredInrecords where the library isdeclared (
agent-reliability-otel-labels/agent_reliability_otel_labelsin thePython root manifests,
@trustabl/agent-reliability-otel-labelsinpackage.json).This is kept out of
ObsDeps.analysis.DiscoverOtelLabelsemits typedOtelLabelsSignalrecords(
import,run_start,tool_span,run_end,attr_literal) for Python andTypeScript/JavaScript. They are kept out of
ObservabilitySignals, because thelibrary never creates a span and must not satisfy its own prerequisite.
mark_tool_span as mtsandmarkToolSpan as labelstill count.trustabl.*keysitself, via
set_attribute/setAttributeor a literal key ofset_attributes/setAttributes, gets credit rung by rung:trustabl.run_id→ run startedtrustabl.exit_reason→ run endtrustabl.tool.input_fp/trustabl.tool.attempt→ tool spanTestOrigin. Test-file signals still creditOBS-006..008, because a test importing the library is real evidence it is
installed.
repo_otel_labels_installed,repo_otel_labels_initialized,repo_otel_labels_implemented.check-rules-sync.shand thetrustabl rules validatehelp text nowpoint at
agent-reliability-rules(the repo was renamed fromtrustabl-rules).Not covered
Run, labels written only on thesuccess path, a constant
attempt, a run end recorded only on success. These areOBS-009..012, on a follow-up branch that waits for library 0.1.1.
passed as
key=, or built from a template string is not credited.Depends on / merge order
Opened on top of #242 (
feat/forge-runtime-tracing). Until #242 merges, this diffalso shows #242's commits.
rules: scans pull the rules repo's
main.main, therules-syncjob checksagainst the rules repo's
main, so merging this first turnsmainred.On this PR,
rules-syncchecks against the rules repo's branch of the same name(
feat/otel-labels-rules).Older engines load the schema-21 pack leniently: they skip OBS-006..008 and report
a META-005 info finding.
Test plan
go vet ./...go test ./...RULES_REPO=../agent-reliability-rules scripts/check-rules-sync.sh(with therules repo on
feat/otel-labels-rules): in sync, 128 files, 333 rules