Add retail-cart benchmark: guest add-to-cart outcomes on bot-protected sites

.github/workflows/ci.yml

@@ -24,6 +24,8 @@ jobs:
       - run: uv run pytest -q
       - name: Prepare every static arm for one task
         run: uv run bench prepare kernel-mcp-clawbench --arms codex-kernel-mcp,claude-kernel-mcp --tasks v2-1134-chapter-finder-redcross --out "$RUNNER_TEMP/prepared"
+      - name: Prepare the static retail-cart arm
+        run: uv run bench prepare retail-cart --arms codex-repl --out "$RUNNER_TEMP/prepared-retail-cart"
 
   report:
     runs-on: ubuntu-latest

.github/workflows/retail-cart.yml

@@ -0,0 +1,45 @@
+name: Retail cart flows with the browser REPL
+
+on:
+  workflow_dispatch:
+    inputs:
+      arms:
+        description: Comma-separated arm names (empty runs every arm in benchmarks/retail-cart.toml)
+        required: false
+        default: ""
+        type: string
+      tasks:
+        description: Comma-separated task IDs, or all
+        required: false
+        default: all
+        type: string
+      attempts:
+        description: Trials per task and arm
+        required: false
+        default: "10"
+        type: string
+      concurrency:
+        description: Concurrent trials per arm
+        required: false
+        default: "16"
+        type: string
+      kernel_mcp_server_ref:
+        description: kernel/kernel-mcp-server git ref to build (empty builds main)
+        required: false
+        default: ""
+        type: string
+
+permissions:
+  contents: read
+
+jobs:
+  benchmark:
+    uses: ./.github/workflows/_run.yml
+    secrets: inherit
+    with:
+      benchmark: retail-cart
+      arms: ${{ inputs.arms }}
+      tasks: ${{ inputs.tasks }}
+      attempts: ${{ inputs.attempts }}
+      concurrency: ${{ inputs.concurrency }}
+      toolset_ref: ${{ inputs.kernel_mcp_server_ref }}

README.md

@@ -119,6 +119,26 @@ The Kernel MCP server on ClawBench.
 
 ClawBench's primary score is the lenient reward. `kernel_mcp_valid` records whether the agent called Kernel MCP's `execute_playwright_code` and used its assigned browser on every call, so a failed task can be told apart from a trial that never used the toolset.
 
+### `retail-cart`
+
+Guest add-to-cart tasks on 16 retail and industrial-supply sites with heavy bot protection. Each task starts on the site's home page and asks the agent to put one item in the site's cart as a guest, by whatever route works, and show the cart.
+
+| Arm | Agent | Browser |
+|---|---|---|
+| `codex-repl` | Codex, `gpt-6-luna`, Kernel MCP's `browser_repl` only | `stealth: true` and no proxy, so each browser gets its own egress IP from the stealth proxy pool |
+| `codex-repl-site-config` | Same; `gpt-6-luna` also picks the site and proxy for the configurator | From Kernel's config registry per site, plus its guidance. Each trial gets its own proxy. |
+| `codex-repl-pinned` | Same, plus Kernel MCP's `computer_action` | A fixed per-site table (`configurators/pinned/sites.toml`): stealth, GPU, a new US ISP or residential proxy per trial, a macOS persona or VM setup on some sites, and per-site guidance |
+
+The agent ends each task by calling the `bench` MCP server's `finalize` tool with its evidence and an outcome: `success`, `failed`, `blocked` (bot protection stopped it), `website_error` or `kernel_error`. `reward` is 1 for a reported success, and `outcome_<name>` records the failure, so bot blocks are counted apart from the agent's own failures.
+
+The verifier then checks the cart itself. It reads the page the agent ended on if that is the cart, otherwise opens the task's `cart_url` or fetches its `cart_api`, and sets `cart_verified` when the item is listed. When it could not see the cart, because loading it failed or showed a bot check, it sets `cart_unverifiable` instead.
+
+Run several attempts per task to get a pass rate per site:
+
+```bash
+uv run bench run retail-cart --attempts 10
+```
+
 ## Extending
 
 ### Add an arm

arms/codex-repl-pinned.toml

@@ -0,0 +1,16 @@
+# Same as codex-repl, but each site's browser, proxy and persona come from the
+# fixed table in configurators/pinned/sites.toml, with a new proxy per trial.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "kernel-mcp-local"
+
+[agent_kwargs]
+disable_web_search = true
+
+[toolset_args]
+toolsets = ["repl", "computer"]
+
+[browser]
+config = { stealth = true }
+configurator = "pinned"

arms/codex-repl-site-config.toml

@@ -0,0 +1,21 @@
+# Same as codex-repl, but each task's browser and proxy come from Kernel's
+# config registry for the task's site, and the registry's guidance for that
+# site is added to the instruction.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "kernel-mcp-local"
+
+[agent_kwargs]
+disable_web_search = true
+
+[toolset_args]
+toolsets = ["repl"]
+
+[browser]
+config = { stealth = true }
+configurator = "site-config"
+
+[browser.configurator_args]
+model = "openai/gpt-6-luna"
+steering = true

arms/codex-repl.toml

@@ -0,0 +1,16 @@
+# Codex driving the browser only through Kernel MCP's browser REPL, with a
+# fixed stealth browser.
+agent = "codex"
+model = "gpt-6-luna"
+agent_version = "0.163.0"
+toolset = "kernel-mcp-local"
+
+# Codex's own web tool would let the agent answer without the browser.
+[agent_kwargs]
+disable_web_search = true
+
+[toolset_args]
+toolsets = ["repl"]
+
+[browser]
+config = { stealth = true }

benchmarks/retail-cart.toml

@@ -0,0 +1,5 @@
+# Guest add-to-cart flows on bot-protected retail and industrial-supply sites,
+# driven through the browser REPL, with a stealth browser and with the config
+# registry's per-site settings, and with a fixed per-site table.
+suite = "retail-cart"
+arms = ["codex-repl", "codex-repl-site-config", "codex-repl-pinned"]

configurators/pinned/configure

@@ -0,0 +1,50 @@
+#!/usr/bin/env python3
+"""Choose browser settings from the fixed per-site table in sites.toml.
+
+The task's site comes from `metadata.site` in its task.toml. Proxies are US
+ISP or US residential, created per trial. A `persona` of `macos` restarts
+Chromium with the macOS persona's launcher variables. `vm_setup` commands run as root
+in the browser's VM before the first page load.
+
+stdin:  {"task": {"config": str, "instruction": str}, "args": {}}
+stdout: {"browser": {...}, "steering": str | null, "provenance": {...}}
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+import tomllib
+from pathlib import Path
+
+TABLE = tomllib.loads((Path(__file__).parent / "sites.toml").read_text())
+
+
+def main() -> int:
+    task = json.load(sys.stdin)["task"]
+    site = tomllib.loads(task["config"]).get("metadata", {}).get("site")
+    entry = TABLE["sites"].get(site)
+    if entry is None:
+        raise SystemExit(f"No pinned configuration for site {site!r}")
+    browser = dict(entry["browser"])
+    browser["proxy_create"] = {"type": entry["proxy"], "config": {"country": "US"}}
+    persona = entry.get("persona")
+    if persona == "macos":
+        browser["chromium_env"] = TABLE["macos_persona"]
+    elif persona is not None:
+        raise SystemExit(f"Unknown persona {persona!r} for {site}")
+    if entry.get("vm_setup"):
+        browser["vm_setup"] = entry["vm_setup"]
+    json.dump(
+        {
+            "browser": browser,
+            "steering": entry.get("steering"),
+            "provenance": {"site": site, "proxy": entry["proxy"], "persona": persona},
+        },
+        sys.stdout,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

configurators/pinned/sites.toml

@@ -0,0 +1,109 @@
+# Browser settings per site, measured by hand on 2026-10-29 to 2026-11-01.
+# Each entry is merged over the arm's config. `proxy` gives every trial its own
+# proxy of that type; `persona = "macos"` restarts Chromium with the macOS
+# persona's launcher variables; `vm_setup` commands run as root in the
+# browser's VM before the first page load; `steering` is added to the
+# instruction.
+
+[macos_persona]
+KERNEL_MAC_PERSONA = "1"
+
+[sites."amazon.com"]
+browser = { stealth = true, gpu = true }
+proxy = "isp"
+persona = "macos"
+
+[sites."costco.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."dickssportinggoods.com"]
+browser = { stealth = true, gpu = true }
+proxy = "isp"
+persona = "macos"
+
+[sites."ebay.com"]
+browser = { stealth = true, gpu = true }
+proxy = "isp"
+
+[sites."guitarcenter.com"]
+# Measured passing only with Web Bot Auth signatures, which this project does not have.
+browser = { stealth = true }
+proxy = "isp"
+steering = """
+guitarcenter.com denies its home page (/), /search and /cart to automated browsers with an "Access Denied" page, but search results at /search/ with a trailing slash, product pages and /cart/index load. Search by opening https://www.guitarcenter.com/search/?Ntt=<words joined with +>, open the product, and click Add to Cart. Show the cart at https://www.guitarcenter.com/cart/index; if it does not list the item, report the add-to-cart confirmation the product page showed.
+"""
+
+[sites."harborfreight.com"]
+browser = { stealth = true, gpu = true }
+proxy = "isp"
+persona = "macos"
+
+[sites."homedepot.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."lowes.com"]
+browser = { stealth = false }
+proxy = "isp"
+
+[sites."mouser.com"]
+browser = { stealth = true }
+proxy = "residential"
+steering = """
+mouser.com denies its search results, category and product pages (/c/..., /ProductDetail/...) to automated browsers, so do not press Enter in the search box. Find the part number instead: type the item into the search box and read the suggestions without submitting, or search the web (for example DuckDuckGo) for "mouser <item>" and take the part number from the results. Then open https://www.mouser.com/en/Cart/, enter the manufacturer or Mouser part number and a quantity in its add-by-part-number row, and click Add.
+
+Pages may show a bot-check interstitial that clears by itself within about 12 seconds. Wait up to 30 seconds before treating a page as blocked.
+"""
+
+[sites."newark.com"]
+browser = { stealth = true }
+proxy = "isp"
+steering = """
+newark.com denies its search results, category and product pages (/search, /c/..., /dp/...) to automated browsers, so pressing Enter in the search box does nothing. The site's older store pages still work; use them:
+
+1. Type the item into the search box and read the suggestions without submitting, or open https://www.newark.com/webapp/wcs/stores/servlet/AjaxSearchLookAhead?searchTerm=<words>&storeId=10194&catalogId=15003&langId=-1. Suggestions link to /dp/<order code>; note the order code of a matching product.
+2. Open https://www.newark.com/webapp/wcs/stores/servlet/ProductDisplay?storeId=10194&catalogId=15003&langId=-1&partNumber=<order code>.
+3. Clicking its Add to Cart link does nothing; navigate to the link's href instead.
+4. Show the cart at https://www.newark.com/webapp/wcs/stores/servlet/OrderItemDisplay?storeId=10194&catalogId=15003&langId=-1. The /cart page stays empty.
+"""
+
+[sites."radwell.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."staples.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."target.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."walmart.com"]
+browser = { stealth = true }
+proxy = "isp"
+
+[sites."wbmason.com"]
+# The captcha-solver extension takes over the site's Cloudflare Turnstile check
+# and never completes it, so its Turnstile interceptor is emptied before the
+# first page load.
+browser = { stealth = true }
+proxy = "residential"
+vm_setup = ["printf '/* disabled */\\n' > /home/kernel/extensions/kernel/capmonster/interceptors/turnstileInterceptor.js"]
+steering = """
+wbmason.com shows a "Verify Access to Continue" page with a Cloudflare Turnstile check before the site. Do not reload or navigate while it is showing: that turns it into a hard block. Wait up to 30 seconds for it to pass by itself; if a checkbox appears, click it once with computer_action.
+
+On this site use computer_action for clicks and typing: scripted clicks and typing in the search box do not take effect.
+"""
+
+[sites."zoro.com"]
+# A fresh residential IP per trial and one reload passed 5/5 by hand; GPU and
+# the WebGL persona made no difference there and got a slider every time in
+# the benchmark.
+browser = { stealth = true }
+proxy = "residential"
+steering = """
+On zoro.com the first page load is often refused with a 403 or a bot-check page. That refusal sets a cookie, and loading the same URL once more usually shows the real site, so if the first load is blocked, load it again once. If the page says "Access is temporarily restricted", reloading will not help: report the task as blocked.
+"""
+

suites/retail-cart/image/Dockerfile

@@ -0,0 +1,13 @@
+# Task image for retail-cart. The browser is the Kernel VM this container runs
+# in, so the image only holds the finalize tool, the start-URL setup and the
+# scorer. Toolsets append their own layers.
+FROM node:22-bookworm-slim
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends ca-certificates curl git python3 \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY bench/ /opt/bench/
+RUN chmod +x /opt/bench/finalize-mcp /opt/bench/open-start-url /opt/bench/score
+
+WORKDIR /app