feat(libsy): add OTel metrics, tracing spans, and structured logging

[eric-liu-nvidia]

Summary

Implements SWITCH-928: the observability layer for the libsy algorithm layer, built on OpenTelemetry per the discussion on the ticket (opentelemetry-rust as the metrics sink; tracing for spans and structured logs, which hosts bridge into OTel with tracing-opentelemetry).

All instrumentation lives in the libsy core at the Decision hook and the offload boundary, so algorithms carry no telemetry code beyond the new Algorithm::name() label:

• Metrics (OTel global meter provider, scope libsy; the library owns no exporter): counters libsy.runs, libsy.llm_calls, libsy.decisions, libsy.{input,output,total,reasoning}_tokens; histograms libsy.run_duration_ms, libsy.llm_call_duration_ms. Attributes are algorithm / selected_model / outcome (ok|error), so failure rates fall out of the outcome split. Instrument naming and cardinality bounds follow the switchyard OTel conventions.
• Spans (tracing): one libsy.run per request carrying the Metadata correlation ids and outcome, with one child libsy.llm_call per offloaded call carrying the selected model, latency, outcome, and token counts.
• Structured logs (tracing, target libsy): an info event per published Decision with its reasoning; warn events for failed calls and runs.

API changes: new required Algorithm::name(&self) -> &str (implemented for the three reference algorithms), Usage re-exported from libsy, and the unused Driver::default() removed (drivers now carry the algorithm label).

With no OTel SDK or tracing subscriber installed, all instrumentation is a no-op.

Testing

• New integration tests in crates/libsy/tests/observability.rs assert the success path (counter/histogram values, exact token counts, span parentage and fields, decision log) and the failure path (outcome=error at run and call level, error text on spans, warn logs) via an in-memory OTel exporter and a capturing tracing layer.
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace all pass (50/50 test binaries).

Fixes SWITCH-928

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added observability for algorithm runs, model calls, and routing decisions through structured tracing, logs, and OpenTelemetry metrics.
  • Captures run outcomes, durations, model selections, errors, and token usage when available.
  • Added stable names for built-in algorithms to improve telemetry labeling.

• Documentation

  • Expanded guidance for exploring the Rust libraries.
  • Documented algorithm naming and observability setup, including available metrics and tracing behavior.

• Tests

  • Added coverage for successful and failed runs, telemetry data, spans, metrics, and structured logs.

[coderabbitai]

Walkthrough

libsy adds a required algorithm name, instruments algorithm runs and LLM calls with tracing and OpenTelemetry metrics, records routing decisions, re-exports Usage, updates examples, and adds integration tests for successful and failed telemetry paths.

Changes

libsy observability

Layer / File(s)	Summary
Observability contract and dependencies crates/libsy/Cargo.toml, crates/libsy/src/lib.rs, crates/libsy/README.md, .agents/skills/...	The Algorithm contract gains name, Usage is re-exported, telemetry dependencies and documentation are added, and exploration guidance points to the Rust algorithm crates.
Telemetry helpers and recording crates/libsy/src/observability.rs	Run and LLM-call spans, outcome and token metrics, structured logs, and routing-decision counters are implemented.
Algorithm naming and execution wiring crates/libsy/src/lib.rs, crates/libsy-examples/src/*.rs	Driver and run_stream propagate algorithm names, instrument execution, record decisions, and update test and example algorithms with stable identifiers.
Observability integration validation crates/libsy/tests/observability.rs	In-memory telemetry tests validate successful and failed runs, metrics, span relationships and fields, token handling, and structured logs.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

I’m a rabbit with metrics tucked under my ear,
Spans hop through runs so outcomes stay clear.
Tokens leave tracks, decisions softly glow,
Errors thump warnings wherever they go.
“Name every algorithm!” I cheer from the burrow below.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding OpenTelemetry metrics, tracing spans, and structured logging to libsy.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

crates/libsy/tests/observability.rs (1)

194-253: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the shared traversal logic between u64_counter_value and f64_histogram_count.

Both functions repeat the same snapshot → scope-name-filter → metric-name-filter loop, differing only in the AggregatedMetrics variant and the value extracted. Worth factoring into one generic helper to avoid the two implementations drifting apart.

♻️ Sketch of a shared helper

fn matching_data_points<'a, T>(
    snapshots: &'a [ResourceMetrics],
    name: &str,
    wanted: &[(&str, &str)],
    extract: impl Fn(&AggregatedMetrics) -> Option<&dyn Iterator<Item = &'a T>>,
) -> Option<u64> {
    // shared scope/metric-name filtering, delegate value extraction to callers
    // (or simply parameterize on an `impl Fn(&Metric) -> Option<u64>` per data point)
}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/libsy/tests/observability.rs` around lines 194 - 253, Extract the
duplicated snapshot, scope-name, metric-name, and attribute filtering from
u64_counter_value and f64_histogram_count into a shared helper. Parameterize the
helper with the metric-data/value extraction needed for each AggregatedMetrics
variant, then have both functions reuse it while preserving their existing
cumulative maximum behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/libsy/README.md`:
- Around line 153-154: Update the impl Algorithm for LlmClassifierOrchAlgo
example to implement the required name method, returning the classifier’s stable
telemetry label, so the documented snippet compiles.

In `@crates/libsy/src/lib.rs`:
- Around line 298-300: Update the info method to await
self.driver.info(decision) before calling observability::record_decision, and
only record the decision after the driver returns success; propagate any driver
error without recording it.

---

Nitpick comments:
In `@crates/libsy/tests/observability.rs`:
- Around line 194-253: Extract the duplicated snapshot, scope-name, metric-name,
and attribute filtering from u64_counter_value and f64_histogram_count into a
shared helper. Parameterize the helper with the metric-data/value extraction
needed for each AggregatedMetrics variant, then have both functions reuse it
while preserving their existing cumulative maximum behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3c15084e-1e09-4aa7-8c59-f212393feaa9

📥 Commits

Reviewing files that changed from the base of the PR and between e62fe46 and 96d2593.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock, !Cargo.lock

📒 Files selected for processing (9)

• .agents/skills/switchyard-codebase-exploration/SKILL.md
• crates/libsy-examples/src/ensemble.rs
• crates/libsy-examples/src/llm_class.rs
• crates/libsy-examples/src/rand.rs
• crates/libsy/Cargo.toml
• crates/libsy/README.md
• crates/libsy/src/lib.rs
• crates/libsy/src/observability.rs
• crates/libsy/tests/observability.rs

[grahamking]

I think this PR tries to do too much. Should we start by defining what metrics libsy should report? That's probably a conversation for Slack.

Then we could add one in a small PR (just the new dependency and the one metric line), and go from there.