feat(experiments): Allow an evaluation to belong to multiple experiment groups#759
Draft
shanaiabuggy wants to merge 6 commits into
Draft
feat(experiments): Allow an evaluation to belong to multiple experiment groups#759shanaiabuggy wants to merge 6 commits into
shanaiabuggy wants to merge 6 commits into
Conversation
Adds a CONTAINS ("$contains") comparison operator to the entity-store filter
engine so callers can query "value is an element of a JSON array field" — the
prerequisite for modeling an experiment's group membership as a list (ASE-559).
- filter_ops.py: new FilterOperator.CONTAINS, a default contains() on the
FilterRepository ABC (additive — existing repos need not implement it), and
dispatch in ComparisonOperation.apply. The JSON filter parser is generic over
the enum, so $contains parses with no parser change.
- SQLAlchemyFilterRepository.contains: portable across SQLite (JSON) and
PostgreSQL (JSONB) via a quote-delimited match on the serialized array text;
wildcard-escaped and prefix-collision-safe ("g1" never matches ["g10"]).
- InMemoryFilterRepository.contains: native membership, matching SQL semantics.
- Parity harness extended with $contains cases incl. a prefix near-miss.
Validated: 32 parity cases pass (SQLite + in-memory agree); 113 entities +
167 plugin filter tests pass; ruff + ty clean. The Postgres path is portable by
design; Docker-backed integration tests confirm it in CI.
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Relax the one-group-per-evaluation constraint to many-to-many. An evaluation's single experiment_group_id becomes an experiment_ids list (>=1); everything that renders a group's runs (list view, rollups, sort) now spans the membership set. - Entity: Experiment.experiment_group_id -> experiment_ids: list[str] (min 1), with a model_validator(mode="before") coercing legacy single-id rows to [id] on read (no data migration). parent_experiment_id (lineage) untouched. - Create/update API: EvaluationRequest accepts experiment_ids (>=1); experiment_group_id kept as a deprecated coalesced alias. EvaluationResponse exposes experiment_ids with experiment_group_id as a deprecated computed alias (= experiment_ids[0]). Each group id is validated to exist + be live. - Membership queries: a shared _group_membership_filter() matches "$contains experiment_ids OR legacy scalar experiment_group_id", used by the list filter (rewriting the exposed experiment_group_id param), both count helpers, and the delete cascade — so un-migrated rows stay queryable. - Delete cascade is now reference-counted: a member whose sole membership was the deleted group is soft-deleted; a member shared with another group just drops that membership and survives there. Tests: multi-group create appears in both leaderboards; empty experiment_ids rejected; reference-counted cascade (shared member survives, sole member gone); existing single-group + deprecated-alias paths still pass. Full intake experiment/evaluation suite green (SQLite). Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
…ASE-559] Address review: the create request should not force the new field on clients. - EvaluationRequest.experiment_ids is now optional (default_factory=list) rather than required, so the request stays purely additive; experiment_group_id remains an accepted (deprecated) field. A model_validator requires at least one of the two and coalesces the deprecated single id into experiment_ids. - experiment_group_id is unchanged in the response body (a deprecated computed alias = experiment_ids[0]), consistent with the existing parent_experiment_id deprecation pattern. Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Contributor
|
Curate existing runs into comparison groups (e.g. an all-benchmarks board)
after creation:
- POST /evaluations/{name}/groups/{group_id} — add a membership. Validates the
group exists + is live; idempotent when already a member.
- DELETE /evaluations/{name}/groups/{group_id} — remove a membership. Rejects
removing the evaluation's last group with 409 (>=1-group invariant; delete the
evaluation instead); idempotent when not a member.
Modeled on the pin/unpin endpoints. Tests cover add (idempotent + appears on the
target board), add-to-unknown-group (400), remove (drops the membership), and
remove-last-group (409).
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
…p [ASE-559]
Regenerated via `make refresh-openapi` after the many-to-many API changes:
- EvaluationRequest/EvaluationResponse gain experiment_ids; experiment_group_id
retained as a deprecated field/alias.
- New POST/DELETE /evaluations/{name}/groups/{group_id} membership endpoints.
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
…559]
A per-row "Add to group" action (FolderPlus, next to the pin toggle) in the
ExperimentGroupDataView opens a modal with a searchable list of the workspace's
experiment groups — excluding the ones the evaluation already belongs to — and
adds the run to the selected group so it appears on that board too. Mirrors the
pin action's mutation + scoped query invalidation, with success/error toasts.
Lets users assemble a cross-benchmark ("all-benchmarks") board from existing
runs. Uses the generated useAddEvaluationToGroup / useListExperimentGroups hooks.
- AddToGroupModal.tsx (+ .test.tsx, 4 cases: lists/excludes members, filters by
search, calls the add hook, empty state)
- ExperimentGroupDataView: row-action button + modal wiring (pin logic unchanged)
typecheck + component test + eslint clean.
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.