Skip to content

docs(qa): record non-Claude browser benchmark pilot - #200

Merged
SarthakWade merged 2 commits into
mainfrom
docs/non-claude-browser-benchmark-pilot
Sep 19, 2026
Merged

SarthakWade merged 2 commits into
mainfrom
docs/non-claude-browser-benchmark-pilot

Conversation

@yashranaway

Copy link
Copy Markdown
Collaborator

What this changes

Records the non-Claude portion of the agent browser-tool benchmark proposed in #161:

Runner Tool Exact success Median wall time Median browser calls
Codex Headless 3/3 218.749 s 34
Codex Playwright MCP 3/3 138.761 s 25
OpenCode Headless 3/3 92.418 s 34
OpenCode Playwright MCP 2/3 129.991 s 24

The PR adds architecture decision 29, the deterministic fixture and validator, exact prompts, aggregate machine-readable results, all 12 raw runner logs, and checksums. CI now verifies nested evidence manifests.

Related to #161. This does not close it. The separate lab repository, full task taxonomy, startup parity, resource metrics, and larger sample remain outstanding.

Why

Issue #161 requires a reproducible multi-run benchmark while preserving the existing deterministic regression gate. The result needs an explicit repository boundary before a maintained lab is created. This PR records that boundary and preserves the pilot without presenting it as a general product ranking.

How it was verified

  • pnpm test was not run locally because this Linux host has no Swift toolchain. apps/headless/build-linux.sh passed all 67 protocol tests in Docker before the measured runs.
  • pnpm test:runtime
  • Repository shell syntax check
  • Existing and new QA evidence checksum manifests
  • git diff --check
  • pnpm test:e2e:linux is not required because this changes no host, transport, runtime, or artifact code.
  • pnpm test:e2e:mac is not required because this changes no macOS host or capture code.

Contracts

No browser protocol or host behavior changes. The pilot explicitly records prompt-injection handling, origin restrictions, approval behavior, and known setup differences.

Housekeeping

  • Architecture-decision entry added
  • QA documentation updated
  • Evidence is under docs/qa/evidence/ with a checksum manifest
  • Existing in-repository benchmark remains unchanged

@SarthakWade SarthakWade left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes before merge. The benchmark fixture passed manual happy-path and failure-path checks, checksums verify, and both pnpm test and pnpm test:runtime pass locally. Three blockers remain: required CI is red, published raw logs expose contributor-local paths and private local file contents, and this branch conflicts with the approved stacked PRs once they land. GitHub currently reports no conflict against the present main, but git merge-tree against stack #202 confirms a content conflict in the architecture decision file.

Comment thread docs/qa/evidence/agent-browser-benchmark-2026-09-19/playwright-adapter.txt Outdated
Comment thread docs/roadmap/architecture-decisions.md Outdated

@SarthakWade SarthakWade left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the updated branch after resolving the main conflict and addressing the requested privacy and CI fixes. Local protocol, runtime, fixture, checksum, static, and production build checks pass. All required GitHub Actions checks pass.

@SarthakWade
SarthakWade merged commit 80d89e6 into main Sep 19, 2026
16 checks passed
@SarthakWade
SarthakWade deleted the docs/non-claude-browser-benchmark-pilot branch September 19, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants