feat: evals runner with vercel workflows, sandbox

[mattrossman]

Incomplete proof of concept using Vercel Workflows to orchestrate Sandboxes for running evals outside of GitHub Actions. Workflows can be dispatched via a new Nitro app in apps/runner, the deployment of which has Deployment Protection so you can use a bypass secret to authenticate.

I started with a single workflow runEvalWorkflow composed of 3 steps:

• createEvalSandbox
• runEvalInSandbox
• deleteEvalSandbox

Idea being that sandbox creation and deletion could fail, so this will retry + continue on step failures as needed, and gracefully rollback sandbox creation on intermediate errors.

A second workflow runEvalsWorkflow (plural) uses the same steps, but adds loops / retry logic to run a batch of multiple eval runs.

I added logic so that agents write their stream of output to the workflow logs (i.e. tool calls, assistant message parts), as well as the normal framework logs (eval run status) to 2 stream "namespaces" for each run.

Not yet included:

• A corresponding GH Action to dispatch the Vercel Workflow and collect results
• Discovery of runs via eval / experiment suites (currently you pass an explicit list of eval x experiments to run)

Note the majority of this diff is pnpm-lock.yaml

Related: #114

Ref AI-912

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored		Aug 10, 2026 8:20pm