feat(providers): add session discovery + evaluate_sessions batch helper (#143) - #391
Open
lifelmy wants to merge 1 commit into
Open
feat(providers): add session discovery + evaluate_sessions batch helper (#143)#391lifelmy wants to merge 1 commit into
lifelmy wants to merge 1 commit into
Conversation
…er (strands-agents#143) Extend TraceProvider with an optional, non-abstract list_sessions(SessionFilter) so backends that can enumerate sessions expose discovery, while the default raises NotImplementedError pointing at the known-session-id path. Add a SessionFilter pydantic model (start_time/end_time/limit + additional_fields). Add strands_evals.batch.evaluate_sessions(provider, evaluators, session_filter) that composes list_sessions -> Case-per-session -> Experiment.run, removing the discover/build/run boilerplate every provider user rewrites. It returns a single EvaluationReport to match the current Experiment API (flattened across evaluators), rather than the list[EvaluationReport] in the original sketch. Export SessionFilter and evaluate_sessions from the package root.
lifelmy
requested a deployment
to
manual-approval
September 1, 2026 06:32 — with
GitHub Actions
Waiting
lifelmy
requested a deployment
to
manual-approval
September 1, 2026 06:32 — with
GitHub Actions
Waiting
lifelmy
marked this pull request as ready for review
September 3, 2026 09:27
lifelmy
requested a deployment
to
manual-approval
September 3, 2026 09:30 — with
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #143.
Extends
TraceProviderwith optional session discovery and adds a batch helper so callers can evaluate every session matching a filter in one call, instead of hand-writing the discover → build cases → run boilerplate.What changed
SessionFilter(Pydantic model inproviders/trace_provider.py): universalstart_time/end_time/limitfields plusadditional_fields: dict[str, Any]for provider-specific criteria.TraceProvider.list_sessions(session_filter=None) -> Iterator[str]: intentionally not abstract. Providers override it only when their backend supports discovery; the default raisesNotImplementedErrorpointing at the known-session_idpath (get_evaluation_data). This keeps the bar low for providers that only accept externally-supplied session IDs — matching alternative Add InteractionsEvaluator for multi-agent and more complicated sequence-based evaluations. #1 rejected in the issue.strands_evals.batch.evaluate_sessions(provider, evaluators, session_filter=None, *, max_workers=1): composeslist_sessions→ oneCaseper session (keyed onsession_id) →Experiment, usingprovider.as_task()as the per-case task. A free function, not a method onExperiment/TraceProvider, to avoid the coupling/circular-dependency issues the issue calls out (alternatives Add styling fixes and license docs. #3 and Add support for dataset generation with strands agents #4).SessionFilterandevaluate_sessionsfrom the package root andproviderspackage.Deviation from the issue sketch
The issue sketched
evaluate_sessions(...) -> list[EvaluationReport]with a locally-definedExperiment+task. The current public API isExperiment.run_evaluations, which returns a singleEvaluationReportflattened across evaluators (each row tagged viacases[i]["evaluator"]). To match today's API I return oneEvaluationReport. Happy to change the shape if you'd prefer otherwise.max_workersdefaults to1(sequential) because aTracedHandlershares one exporter and some providers share a single network client.Related Issues
#143
Testing
hatch test tests/strands_evals/providers/ tests/strands_evals/test_batch.py→ 107 passed (includes the existing provider suites, plus newSessionFilter/list_sessions/evaluate_sessionstests).hatch test tests/strands_evals/providers/test_trace_provider.py tests/strands_evals/test_batch.py→ 21 passed.hatch fmt --formatterclean;hatch fmt --linter→ all checks passed.hatch run hatch-static-analysis:mypy -p src→ Success, no issues in 236 source files.New tests:
tests/strands_evals/test_batch.py(batch fan-out, per-session output routing, filter forwarding, empty discovery, NotImplementedError path) and addedTestSessionFilter/TestListSessionstotests/strands_evals/providers/test_trace_provider.py.Checklist
I use a coding assistant to help implement, and I review and take responsibility for the final change.