Skip to content

Add OBS-006...008 for agent-reliability-otel-labels - #258

Merged
ericksondelacruz merged 8 commits into
mainfrom
feat/otel-labels-rules
Oct 12, 2026
Merged

ericksondelacruz merged 8 commits into
mainfrom
feat/otel-labels-rules

Conversation

@ericksondelacruz

Copy link
Copy Markdown
Collaborator

Summary

Adds three repo-scope rules that check whether an instrumented agent uses
agent-reliability-otel-labels
to label the spans it already emits. They form one ladder; each fires at the first
missing rung, so at most one fires per repo:

Rule Fires when Severity Confidence
OBS-006 Tracing is initialized, but the library is neither declared nor imported low 0.7
OBS-007 The library is installed, but no Run's mark_start is ever called medium 0.65
OBS-008 A run is started, but mark_run_end is missing, or the repo has tools and no mark_tool_span call medium 0.6

All three are gated on repo_observability_initialized: true, because the library
only labels spans that something else already creates.

schema_version 20 → 21.

Engine changes

  • Recon: RepoProfile.OtelLabelsDeclaredIn records where the library is
    declared (agent-reliability-otel-labels / agent_reliability_otel_labels in the
    Python root manifests, @trustabl/agent-reliability-otel-labels in package.json).
    This is kept out of ObsDeps.
  • Inventory: analysis.DiscoverOtelLabels emits typed OtelLabelsSignal records
    (import, run_start, tool_span, run_end, attr_literal) for Python and
    TypeScript/JavaScript. They are kept out of ObservabilitySignals, because the
    library never creates a span and must not satisfy its own prerequisite.
  • Renamed imports are followed: each file's own imports are resolved, so
    mark_tool_span as mts and markToolSpan as label still count.
  • Hand-written labels are credited: a repo that writes the trustabl.* keys
    itself, via set_attribute / setAttribute or a literal key of set_attributes /
    setAttributes, gets credit rung by rung:
    • any key → installed
    • trustabl.run_id → run started
    • trustabl.exit_reason → run end
    • trustabl.tool.input_fp / trustabl.tool.attempt → tool span
  • Test files: every signal carries TestOrigin. Test-file signals still credit
    OBS-006..008, because a test importing the library is real evidence it is
    installed.
  • Predicates: repo_otel_labels_installed, repo_otel_labels_initialized,
    repo_otel_labels_implemented.
  • Chore: check-rules-sync.sh and the trustabl rules validate help text now
    point at agent-reliability-rules (the repo was renamed from trustabl-rules).

Not covered

  • Whether the labels are correct: a shared Run, labels written only on the
    success path, a constant attempt, a run end recorded only on success. These are
    OBS-009..012, on a follow-up branch that waits for library 0.1.1.
  • Hand-written keys that aren't string literals: a key held in a constant,
    passed as key=, or built from a template string is not credited.
  • The Go binding: not inspected.

Depends on / merge order

Opened on top of #242 (feat/forge-runtime-tracing). Until #242 merges, this diff
also shows #242's commits.

  1. Forge Runtime Tracing section for OpenTelemetry and Agent Reliability OTEL Labels #242 (forge Runtime Tracing)
  2. trustabl-rulebook #__ (OBS-006..008 rationale)
  3. agent-reliability-rules #__ (OBS-006..008 YAML). Merging this one releases the
    rules: scans pull the rules repo's main.
  4. This PR, right after the rules PR. On main, the rules-sync job checks
    against the rules repo's main, so merging this first turns main red.

On this PR, rules-sync checks against the rules repo's branch of the same name
(feat/otel-labels-rules).

Older engines load the schema-21 pack leniently: they skip OBS-006..008 and report
a META-005 info finding.

Test plan

  • go vet ./...
  • go test ./...
  • Every commit builds on its own
  • RULES_REPO=../agent-reliability-rules scripts/check-rules-sync.sh (with the
    rules repo on feat/otel-labels-rules): in sync, 128 files, 333 rules
  • CI green

…t-reliability-otel-labels

trustabl forge now emits a "Runtime Tracing" section in the generated
SKILL.md, right after the apply loop. It tells the coding model how to make
the agent emit OpenTelemetry spans (Step 1), how to add the agent-reliability-otel-labels
labels to those spans (Step 2), how to choose their values (Step 3), and
what to check before moving on (Step 4).

Selection:
- Forge detects the language each SDK is declared in from the manifest
  that declares it (Python, TypeScript, Go) and keeps (category, language)
  pairs as detected, with no cross product.
- --lang python,typescript,go adds a language; an explicit --policy or
  --lang widens the selection only for what detection did not already
  place. An unknown --lang value exits 1.
- The section is omitted, with a one-line note on stderr, when no
  supported language is selected.

Content:
- One block per language, plus ten framework recipes copied from scripts
  that ran: Python (AutoGen, CrewAI, Google ADK, LangChain / LangGraph,
  MCP client sessions, OpenAI Agents SDK, Pydantic AI), TypeScript
  (OpenAI Agents SDK, Vercel AI SDK), Go (Google ADK).
- Text is embedded markdown under internal/forge/tracingtext/, indexed by
  the languageBlocks and frameworkRecipes tables.

Supporting changes:
- Recon gains a go.mod needle for Google ADK for Go (recon-only name
  google-adk-go, deliberately not mapped in scanner.depNameToSDK) and a
  shared manifest-to-language table.
- TemplateVersion is 3, so forge check reports older generated skills as
  outdated; its regenerate hint now says to pass the original --policy and
  --lang flags again.
- A drift guard against agent-reliability-otel-labels: a vendored copy of its key
  manifest, a golden list of the binding identifiers the text uses, tests
  for both, and scripts/check-otel-sync.sh for checking against the real
  repo. The CI job for it is not added yet because that repo is not
  publicly cloneable.
- README.md, ARCHITECTURE.md and COVERAGE.md describe the new behaviour.
…language

Review fixes for the forge Runtime Tracing section.

- ADK Go recipe: the agent, its tools and its callbacks are built once, so a
  Run they close over was shared by every run (the defect OBS-009 flags).
  The recipe now carries one run state per runner.Run on the context.
  Checked against ADK Go v1.7.0: 5 runs of one agent, 3 concurrent, now get
  5 distinct run IDs and step indexes that restart at 0.
- Rule sections are filtered to the languages each SDK is used in, through
  the scanner's own language gate (RuleDef.AppliesToLanguage). A Go-only ADK
  repo no longer gets Python and TypeScript rules; it gets the
  language-independent ones and a note that no Go rules exist yet. A category
  from --policy alone, or also declared in a manifest forge cannot place
  (Detection.Unplaced), keeps every rule.
- TypeScript setup registers its provider globally, so framework
  instrumentation that starts spans on the global API is exported and labelled.
- claude_skill alone gets no tracing section and no "pass --lang" note;
  --lang without --policy in a repo with no manifest now says --lang does not
  select a rule pack.
- scripts/check-otel-sync.sh counts only declarations in source files, so a
  stale __pycache__/*.pyc, a comment, or rand.Read no longer hides a rename
  (the old check missed 7 of 9 declaration renames).
- Cleanups: one flag-value parser for --policy and --lang; tracing recipes
  are selected once per section and each labels file is read once.
…-rules

The rules repo was renamed from trustabl-rules. check-rules-sync.sh now
defaults RULES_REPO to ../agent-reliability-rules, and the
"trustabl rules validate" example uses the new name.
Three repo-scope rules check whether an instrumented agent uses
agent-reliability-otel-labels. They form one ladder; each fires at the first
missing rung, so at most one fires per repo:

- OBS-006 (low): tracing is initialized, but the library is neither
  declared nor imported.
- OBS-007 (medium): the library is installed, but no Run's mark_start is
  called.
- OBS-008 (medium): a run is started, but mark_run_end is missing, or the
  repo has tools and no mark_tool_span call.

All three are gated on repo_observability_initialized: true.

Engine changes:
- Recon: RepoProfile.OtelLabelsDeclaredIn records the Python and npm
  package names, kept separate from ObsDeps.
- Inventory: analysis.DiscoverOtelLabels emits OtelLabelsSignal records
  (import, run_start, tool_span, run_end) for Python and TypeScript/
  JavaScript, kept separate from ObservabilitySignals so the library
  cannot satisfy its own prerequisite. The Go binding is not inspected.
- Predicates: repo_otel_labels_installed, repo_otel_labels_initialized,
  repo_otel_labels_implemented.
- SupportedSchemaVersion and the fixture manifest go from 20 to 21.

Not covered: whether the labels are correct (a shared Run, a constant
attempt, labelling outside finally, an understated side_effect, unmarked
handoffs), hand-written trustabl.* keys, and aliased imports.

The fixture mirrors agent-reliability-rules branch feat/otel-labels-rules.
Do not merge the rules before the library is published on PyPI and npm.
@ericksondelacruz
ericksondelacruz merged commit 44b2062 into main Oct 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants