feat(frontend): migrate evaluator playground to workflow evaluators

[mmabrouk]

Summary

• Migrates evaluator configs CRUD to use workflow evaluator endpoints
• Migrates evaluator run/invoke to use workflow-based invocation
• Includes API fixes for schema persistence and builtin evaluator hydration

Stacked on #3572. Merges into docs/migrate-evaluator-playground-plan which targets main.

Previously merged as #3576 and #3577 into the wrong base branch (chore/migrate-evaluators).

---

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Feb 16, 2026 1:21pm

[devin-ai-integration]

Devin Review found 1 potential issue.

View 7 additional findings in Devin Review.

[devin-ai-integration]

🟡 Shallow merge in _ensure_builtin_evaluator_data can silently discard hydrated output schemas

When the existing simple_evaluator_data has a schemas field set to a value that does not contain an outputs key (e.g., schemas: {} or schemas: {"inputs": {...}}), the shallow dict merge {**hydrated_data_dict, **existing_data_dict} completely replaces the hydrated schemas (which contains the computed outputs) with the existing incomplete schemas, thereby losing the output schema the function was supposed to inject.

Root Cause & Impact

The function's purpose is to fill in missing output schemas for builtin evaluators. At api/oss/src/core/evaluators/service.py:848-853, the merge is:

return SimpleEvaluatorData(
    **{
        **hydrated_data_dict,   # has schemas.outputs
        **existing_data_dict,   # may have schemas WITHOUT outputs
    }
)

Because Python dict unpacking is a shallow merge, the entire schemas key from existing_data_dict replaces the one from hydrated_data_dict. For example:

• hydrated_data_dict["schemas"] = {"outputs": {"type": "object", ...}}
• existing_data_dict["schemas"] = {} (or {"inputs": {...}})
• Result: schemas = {} — the outputs schema is lost

The _has_outputs_schema guard at line 817 only checks if outputs already exist and returns early. It does not prevent the merge from discarding the hydrated outputs when schemas exists but lacks the outputs key.

Impact: Evaluator revisions can be persisted without the expected data.schemas.outputs, which may break downstream consumers that rely on knowing the evaluator's output shape (e.g., workflow invocation, result rendering).

Prompt for agents

In api/oss/src/core/evaluators/service.py, the _ensure_builtin_evaluator_data method (around line 848-853) performs a shallow dict merge that can lose the hydrated schemas.outputs. Replace the shallow merge with a deep merge that preserves nested dict keys. Specifically, when both hydrated_data_dict and existing_data_dict have a 'schemas' key, the merge should combine them so that existing schemas keys take precedence but missing keys (like 'outputs') are filled in from the hydrated dict. For example:

merged = {**hydrated_data_dict, **existing_data_dict}
if 'schemas' in hydrated_data_dict and 'schemas' in existing_data_dict:
    merged['schemas'] = {**hydrated_data_dict['schemas'], **existing_data_dict['schemas']}

Then construct SimpleEvaluatorData(**merged).

---

Was this helpful? React with 👍 or 👎 to provide feedback.