docs(qa): record non-Claude browser benchmark pilot - #200
Conversation
SarthakWade
left a comment
There was a problem hiding this comment.
Requesting changes before merge. The benchmark fixture passed manual happy-path and failure-path checks, checksums verify, and both pnpm test and pnpm test:runtime pass locally. Three blockers remain: required CI is red, published raw logs expose contributor-local paths and private local file contents, and this branch conflicts with the approved stacked PRs once they land. GitHub currently reports no conflict against the present main, but git merge-tree against stack #202 confirms a content conflict in the architecture decision file.
SarthakWade
left a comment
There was a problem hiding this comment.
Reviewed the updated branch after resolving the main conflict and addressing the requested privacy and CI fixes. Local protocol, runtime, fixture, checksum, static, and production build checks pass. All required GitHub Actions checks pass.
What this changes
Records the non-Claude portion of the agent browser-tool benchmark proposed in #161:
The PR adds architecture decision 29, the deterministic fixture and validator, exact prompts, aggregate machine-readable results, all 12 raw runner logs, and checksums. CI now verifies nested evidence manifests.
Related to #161. This does not close it. The separate lab repository, full task taxonomy, startup parity, resource metrics, and larger sample remain outstanding.
Why
Issue #161 requires a reproducible multi-run benchmark while preserving the existing deterministic regression gate. The result needs an explicit repository boundary before a maintained lab is created. This PR records that boundary and preserves the pilot without presenting it as a general product ranking.
How it was verified
pnpm testwas not run locally because this Linux host has no Swift toolchain.apps/headless/build-linux.shpassed all 67 protocol tests in Docker before the measured runs.pnpm test:runtimegit diff --checkpnpm test:e2e:linuxis not required because this changes no host, transport, runtime, or artifact code.pnpm test:e2e:macis not required because this changes no macOS host or capture code.Contracts
No browser protocol or host behavior changes. The pilot explicitly records prompt-injection handling, origin restrictions, approval behavior, and known setup differences.
Housekeeping
docs/qa/evidence/with a checksum manifest