Skip to content

feat(providers): add session discovery + evaluate_sessions batch helper (#143) - #391

Open
lifelmy wants to merge 1 commit into
strands-agents:mainfrom
lifelmy:agent-tasks/143
Open

feat(providers): add session discovery + evaluate_sessions batch helper (#143)#391
lifelmy wants to merge 1 commit into
strands-agents:mainfrom
lifelmy:agent-tasks/143

Conversation

@lifelmy

@lifelmy lifelmy commented Sep 1, 2026

Copy link
Copy Markdown

Description

Closes #143.

Extends TraceProvider with optional session discovery and adds a batch helper so callers can evaluate every session matching a filter in one call, instead of hand-writing the discover → build cases → run boilerplate.

What changed

  • SessionFilter (Pydantic model in providers/trace_provider.py): universal start_time / end_time / limit fields plus additional_fields: dict[str, Any] for provider-specific criteria.
  • TraceProvider.list_sessions(session_filter=None) -> Iterator[str]: intentionally not abstract. Providers override it only when their backend supports discovery; the default raises NotImplementedError pointing at the known-session_id path (get_evaluation_data). This keeps the bar low for providers that only accept externally-supplied session IDs — matching alternative Add InteractionsEvaluator for multi-agent and more complicated sequence-based evaluations. #1 rejected in the issue.
  • strands_evals.batch.evaluate_sessions(provider, evaluators, session_filter=None, *, max_workers=1): composes list_sessions → one Case per session (keyed on session_id) → Experiment, using provider.as_task() as the per-case task. A free function, not a method on Experiment/TraceProvider, to avoid the coupling/circular-dependency issues the issue calls out (alternatives Add styling fixes and license docs. #3 and Add support for dataset generation with strands agents #4).
  • Exported SessionFilter and evaluate_sessions from the package root and providers package.

Deviation from the issue sketch

The issue sketched evaluate_sessions(...) -> list[EvaluationReport] with a locally-defined Experiment + task. The current public API is Experiment.run_evaluations, which returns a single EvaluationReport flattened across evaluators (each row tagged via cases[i]["evaluator"]). To match today's API I return one EvaluationReport. Happy to change the shape if you'd prefer otherwise. max_workers defaults to 1 (sequential) because a TracedHandler shares one exporter and some providers share a single network client.

Related Issues

#143

Testing

  • hatch test tests/strands_evals/providers/ tests/strands_evals/test_batch.py107 passed (includes the existing provider suites, plus new SessionFilter / list_sessions / evaluate_sessions tests).
  • hatch test tests/strands_evals/providers/test_trace_provider.py tests/strands_evals/test_batch.py21 passed.
  • hatch fmt --formatter clean; hatch fmt --linter → all checks passed.
  • hatch run hatch-static-analysis:mypy -p src → Success, no issues in 236 source files.

New tests: tests/strands_evals/test_batch.py (batch fan-out, per-session output routing, filter forwarding, empty discovery, NotImplementedError path) and added TestSessionFilter / TestListSessions to tests/strands_evals/providers/test_trace_provider.py.

Checklist

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix/feature works
  • New and existing unit tests pass locally
  • I have reviewed and understand every line of code in this PR

I use a coding assistant to help implement, and I review and take responsibility for the final change.

…er (strands-agents#143)

Extend TraceProvider with an optional, non-abstract list_sessions(SessionFilter)
so backends that can enumerate sessions expose discovery, while the default
raises NotImplementedError pointing at the known-session-id path. Add a
SessionFilter pydantic model (start_time/end_time/limit + additional_fields).

Add strands_evals.batch.evaluate_sessions(provider, evaluators, session_filter)
that composes list_sessions -> Case-per-session -> Experiment.run, removing the
discover/build/run boilerplate every provider user rewrites. It returns a single
EvaluationReport to match the current Experiment API (flattened across
evaluators), rather than the list[EvaluationReport] in the original sketch.

Export SessionFilter and evaluate_sessions from the package root.
@github-actions github-actions Bot added area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL enhancement New feature or request labels Sep 1, 2026
@lifelmy
lifelmy marked this pull request as ready for review September 3, 2026 09:27
@lifelmy
lifelmy requested a review from a team as a code owner September 3, 2026 09:27
@lifelmy
lifelmy requested a review from mehtarac September 3, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Extend TraceProvider

1 participant