feat: add Cursor CLI agent harness and experiments - #166
Conversation
Enable Composer 2.5 and Grok 4.5 evals via cursor-agent with stream-json parsing, MCP/skills support, and benchmark/no-skills experiment variants. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@vivek100 is attempting to deploy a commit to the Supabase Team on Vercel. A member of the Team first needs to authorize it. |
|
Hey @vivek100 , thanks for looking into this. We're definitely interested in adding support for more harnesses including Cursor (it looks like this work overlaps with existing PR #76). Disclaimer, the blocker here is our eval runner infra which current lives on GitHub Actions with a 256 job limit. We're working to move to a more scalable solution that will give us the headroom to support more agent variations in our benchmark matrix. |
|
Hi @mattrossman thanks for the response. I should have looked at the open PRs first :P , PR #76 does implement the same thing. I was working on similar evals using harbor framework and was trying to see if there are any upsides or trying the supabase eval setup. Will mark this PR close as its not needed, 76 can be used when the infra setup is updated. |
Summary
packages/core/src/agents/cursor/) with pinned install, headlessstream-jsonrunner, transcript parser, andcursorAgent()factory wired into the registry.cursor-composer-2.5,cursor-composer-2.5-no-skills,cursor-grok-4.5-high,cursor-grok-4.5-high-no-skills(Supabase MCP on; skills vs none).CURSOR_API_KEYin.env.example/ README and label Cursor in the results UI.Fixes #167
How to run
Model ids (from live
cursor-agent --list-models):composer-2.5,cursor-grok-4.5-high.Verification
pnpm exec vitest run src/agents/cursor/(core package)pnpm typecheck,pnpm format:checkpnpm eval:dryfor all four Cursor experimentscursor-composer-2.5×resolve-dataapi-001-empty-results— agent completed with Supabase MCP (list_tables,execute_sql,apply_migration, …) and skills loaded. Scorer judge step needsOPENAI_API_KEY(eval usesjudge()in addition to deterministic checks).Test plan
pnpm --filter @supabase-evals/core test(includes Cursor runner/parser tests)pnpm typecheckrun-evalslabel if maintainers want full benchmark CI