[donei003]

Overview

Implements the SDK side of the idempotent offline-eval create plan (docs/idempotent-eval-create.md).

run(key=…) now reconciles via public …/offline-evaluations:

1. GET …/offline-evaluations/{key}
2. 404 → POST create-only with required key + name (name defaults to key); on 409 race, re-GET and continue
3. 200 (or after 409 re-GET) → always PATCH the full desired config snapshot; the server decides whether anything changed (no client-side equality)
4. POST …/evaluations/{id}/runs (runs stay on the legacy by-id prefix until nested offline-evaluations runs ship server-side)

No PUT. Legacy POST …/evaluations create-always remains on the server for SDK 1.0 clients; this release no longer calls it.

Create/PATCH bodies always include tools and criteria (including []) so a PATCH can clear prior pins.

Depends on server stack: ai-evaluator#922 (schema), ai-evaluator#923 (API).

Contract update: companion draft on ai-sdks-monorepo for TESTING.md §8.4–§8.5.

Version / docs

• README: stable keys; document >=1.1.0 for this reconcile behavior (release-please will bump from feat(evaluations): on merge).
• Examples use key="support-qa" instead of timestamp-suffixed keys.

Testing strategy

uv run pytest \
  packages/client/tests/test_evaluations_run.py \
  packages/client/tests/test_evaluations.py \
  packages/client/tests/test_evaluations_spans.py -q

211 passed in test_evaluations_run.py, including:

• first mint (GET 404 → POST)
• steady-state reuse always PATCHes (GET 200 → PATCH)
• POST 409 → re-GET → PATCH race
• LDApiClient.patch JSON body

Other

Out of scope here: moving runs/summary/upload under …/offline-evaluations/{id}/… (server follow-on); soft-hide null-key UI; Mintlify docs.

  

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Retry overwrites newer evaluation settings

When another writer updates a key during a failed PATCH retry, the retry reapplies an older snapshot. LDApiClient retries 5xx responses without checking the evaluation version, so the newer settings disappear.

Learn more

LDApiClient.request retries an idempotent PATCH after 5xx or a transport timeout without reading the record again (retry loop). A full desired snapshot is safe to replay only while no other writer has changed that evaluation. If the first PATCH took effect but its response was lost, a second writer can update the key before this client's backoff ends. Replaying the first body then treats the second writer's configuration as a change and replaces it. The retry can also change the evaluation version returned to the run.

Example: Worker A PATCHes support-qa to model gpt-4o and receives 503 after application. Worker B PATCHes it to claude-sonnet; A's automatic retry restores gpt-4o.

Recommended fix: Keep this PATCH non-retryable on ambiguous failures, or add server-enforced conditional writes using a version/ETag so a replay fails when another update intervenes. If retries are necessary, re-read and reconcile under that server-side concurrency constraint.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[XieX]

Should fix before merge

1. PATCH sends name: key, so every CI run resets a renamed evaluation.
2. The key format isn't checked before network I/O. _validate_run_args (module.py:769) only rejects blank keys. The server's create rule is ^[a-z0-9][a-z0-9._-]*$, at most 128 characters (the GET path allows uppercase). So "Support-QA" gets a 404 on GET and then a 400 on POST, and a key with : or a space gets a 400 on the GET. These failures come after the tool and dataset requests have already run. Before this change the key was only used as name, so any string worked, and timestamp keys from the old README with colons will now fail. TESTING.md says "All validation happens before any network I/O". May need to be called out as a behaviour change in the release notes.
3. A PATCH 409 isn't handled. The PATCH route returns 409 when two updates collide (launchdarkly/ai-evaluator#923 test_returns_409_on_optimistic_lock), which is the likely failure when concurrent CI jobs share a key. Right now it surfaces as a raw LDApiError. Consider retrying once after a re-GET, or at least raising an EvaluationsError that explains it.

Smaller points

• Missing version. _parse_evaluation_ref (runner.py:630) turns a missing or non-integer version into None, and that goes out as evaluationVersion. TESTING.md and JS both treat it as an error. This predates the PR, but the PATCH response is now the source of the version, so it's a good moment to fix. Note that isinstance(True, int) also accepts a bool.
• Response "key": null. raw.get("key", raw.get("name", ...)) returns None when the response has an explicit "key": null; JS falls back to name, then label, then the caller's key. Same for parameters, prompt_snippets and messages passed explicitly as None: Python sends null, while JS sends {} or []. Would be good to be consistent across both.

---

Drafted with Claude Code and cross-checked against the server and gonfalon code.