Skip to content

feat: add Cursor CLI agent harness and experiments - #166

Closed
vivek100 wants to merge 1 commit into
supabase:mainfrom
vivek100:feat/cursor-agent-support
Closed

feat: add Cursor CLI agent harness and experiments#166
vivek100 wants to merge 1 commit into
supabase:mainfrom
vivek100:feat/cursor-agent-support

Conversation

@vivek100

@vivek100 vivek100 commented Aug 4, 2026

Copy link
Copy Markdown

Summary

  • Add a Cursor CLI harness (packages/core/src/agents/cursor/) with pinned install, headless stream-json runner, transcript parser, and cursorAgent() factory wired into the registry.
  • Add four experiments mirroring Claude Code: cursor-composer-2.5, cursor-composer-2.5-no-skills, cursor-grok-4.5-high, cursor-grok-4.5-high-no-skills (Supabase MCP on; skills vs none).
  • Document CURSOR_API_KEY in .env.example / README and label Cursor in the results UI.

Fixes #167

How to run

# .env needs CURSOR_API_KEY; OPENAI_API_KEY for rubric/judge checks
pnpm eval -- --experiment cursor-composer-2.5 --eval resolve-dataapi-001-empty-results

Model ids (from live cursor-agent --list-models): composer-2.5, cursor-grok-4.5-high.

Verification

  • Unit: pnpm exec vitest run src/agents/cursor/ (core package)
  • Typecheck / format: pnpm typecheck, pnpm format:check
  • Discovery: pnpm eval:dry for all four Cursor experiments
  • Tools-mode smoke: cursor-composer-2.5 × resolve-dataapi-001-empty-results — agent completed with Supabase MCP (list_tables, execute_sql, apply_migration, …) and skills loaded. Scorer judge step needs OPENAI_API_KEY (eval uses judge() in addition to deterministic checks).

Test plan

  • pnpm --filter @supabase-evals/core test (includes Cursor runner/parser tests)
  • pnpm typecheck
  • Dry-run one Cursor experiment
  • Optional: add run-evals label if maintainers want full benchmark CI
  • Optional: targeted eval-refresh for the new Cursor experiments

Enable Composer 2.5 and Grok 4.5 evals via cursor-agent with stream-json
parsing, MCP/skills support, and benchmark/no-skills experiment variants.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vivek100
vivek100 requested a review from a team August 4, 2026 20:39
@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown

@vivek100 is attempting to deploy a commit to the Supabase Team on Vercel.

A member of the Team first needs to authorize it.

@mattrossman

Copy link
Copy Markdown
Collaborator

Hey @vivek100 , thanks for looking into this.

We're definitely interested in adding support for more harnesses including Cursor (it looks like this work overlaps with existing PR #76).

Disclaimer, the blocker here is our eval runner infra which current lives on GitHub Actions with a 256 job limit. We're working to move to a more scalable solution that will give us the headroom to support more agent variations in our benchmark matrix.

@vivek100

vivek100 commented Aug 5, 2026

Copy link
Copy Markdown
Author

Hi @mattrossman thanks for the response.

I should have looked at the open PRs first :P , PR #76 does implement the same thing.

I was working on similar evals using harbor framework and was trying to see if there are any upsides or trying the supabase eval setup.

Will mark this PR close as its not needed, 76 can be used when the infra setup is updated.

@vivek100 vivek100 closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Cursor CLI agent support (Composer 2.5 / Grok 4.5)

2 participants