Skip to content

Commit 184bf8d

Browse files
yunhao-jclaude
andcommitted
CoderCup — initial public release
The continuous public benchmark for AI coding agents, refereed by TestSprite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0 parents  commit 184bf8d

394 files changed

Lines changed: 103843 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/ISSUE_TEMPLATE/bug.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
name: 🐛 Bug report
3+
about: Something on codercup.ai broken or wrong
4+
title: 'Bug: '
5+
labels: bug
6+
---
7+
8+
**Where**
9+
10+
<!-- Page URL or component -->
11+
12+
**What happens**
13+
14+
<!-- One paragraph -->
15+
16+
**What should happen**
17+
18+
<!-- One paragraph -->
19+
20+
**Repro**
21+
22+
<!-- Steps; browser + viewport if relevant -->
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
name: 🤖 New agent driver
3+
about: Propose adding a new coding agent to CoderCup
4+
title: 'New driver: <agent-name>'
5+
labels: new-driver
6+
---
7+
8+
<!--
9+
CoderCup is open to any AI coding agent that runs on a sandboxed
10+
Linux host through a CLI. Onboarding is a ~30-line driver entry
11+
in runners/drivers/<agent-slug>/ + contract review.
12+
13+
See runners/README.md#adding-a-new-driver for the full process.
14+
-->
15+
16+
**Agent name + CLI invocation**
17+
18+
<!-- e.g., "Aider — `aider --message-file <prompt>`" -->
19+
20+
**Authentication model**
21+
22+
<!-- Subscription / API key / OAuth / other -->
23+
24+
**Public pricing**
25+
26+
<!-- Link to the model's public rate card so we can populate
27+
scoring/rates.ts -->
28+
29+
**Streaming behavior**
30+
31+
<!-- Does the CLI emit JSONL events to stdout? structured logs?
32+
just plain text? -->
33+
34+
**Maintainer**
35+
36+
<!-- Who will be on call if this driver breaks? -->
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
name: 📋 Task suggestion
3+
about: Suggest a new requirement or surface to add to a CoderCup task spec
4+
title: 'Task suggestion: '
5+
labels: task-suggestion
6+
---
7+
8+
<!--
9+
The current task is World Cup 2026 (v1 → v2 in progress). New
10+
requirements get folded into the next spec version + scored on the
11+
next agent cohort run.
12+
13+
See task-spec/ for the current spec versions and tests/world-cup-v1/
14+
for the registered TestSprite plans.
15+
-->
16+
17+
**Surface to add**
18+
19+
<!-- One sentence: what should the agent's deployed app expose? -->
20+
21+
**Why this matters**
22+
23+
<!-- One paragraph: what user need does it address, or what
24+
engineering capability does it differentiate? -->
25+
26+
**Suggested test plan (optional)**
27+
28+
<!-- If you can sketch a TestSprite plan in the world-cup-v1 format:
29+
{
30+
"type": "frontend",
31+
"name": "...",
32+
"description": "...",
33+
"priority": "p2",
34+
"planSteps": [
35+
{ "type": "action", "description": "..." },
36+
{ "type": "assertion", "description": "..." }
37+
]
38+
}
39+
-->
40+
41+
**Acceptance**
42+
43+
<!-- Concrete: when does this requirement count as met? -->

.github/workflows/ci.yml

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
name: Unit tests + Next.js build
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- uses: actions/setup-node@v4
17+
with:
18+
node-version: '20.10.0'
19+
cache: 'npm'
20+
21+
- name: Install dependencies
22+
run: npm ci
23+
24+
- name: Type check + run unit tests
25+
run: npm test -- --run
26+
27+
- name: Build the leaderboard (Next.js static export)
28+
run: npm run build
29+
30+
# CDK tsc-only check (no synth — RunnerStack does a VPC lookup that needs
31+
# AWS credentials, and we don't want to gate CI on OIDC setup right now).
32+
cdk-typecheck:
33+
name: CDK type check (infra)
34+
runs-on: ubuntu-latest
35+
defaults:
36+
run:
37+
working-directory: infra/cdk
38+
steps:
39+
- uses: actions/checkout@v4
40+
41+
- uses: actions/setup-node@v4
42+
with:
43+
node-version: '20.10.0'
44+
45+
- name: Install CDK deps
46+
run: npm ci
47+
48+
- name: tsc --noEmit
49+
run: npx tsc --noEmit

.gitignore

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# dependencies
2+
/node_modules
3+
/.pnp
4+
.pnp.*
5+
.yarn/*
6+
!.yarn/patches
7+
!.yarn/plugins
8+
!.yarn/releases
9+
!.yarn/versions
10+
11+
# testing
12+
/coverage
13+
14+
# next.js
15+
/.next/
16+
/out/
17+
18+
# production
19+
/build
20+
21+
# misc
22+
.DS_Store
23+
*.pem
24+
25+
# debug
26+
npm-debug.log*
27+
yarn-debug.log*
28+
yarn-error.log*
29+
.pnpm-debug.log*
30+
31+
# env files (DO NOT COMMIT SECRETS)
32+
.env
33+
.env*.local
34+
.env.development
35+
.env.production
36+
37+
# typescript
38+
*.tsbuildinfo
39+
next-env.d.ts
40+
41+
# editor / OS
42+
.idea
43+
.vscode
44+
.cursor
45+
*.swp
46+
47+
# AWS CDK
48+
infra/cdk.out
49+
infra/cdk.context.json
50+
51+
# runner artifacts (local-only; production drops to S3)
52+
/runs/
53+
54+
# Local Claude scheduler state
55+
.claude/
56+
57+
# TestSprite local run artifacts (videos/snapshots — large, not source)
58+
.testsprite/
59+
err.txt
60+
61+
# internal ops docs + orchestration harness (not part of the open-source release)
62+
docs-internal/

0 commit comments

Comments
 (0)