Skip to content

Ask retail-cart for the outcome, verify the cart, and add the pinned per-site arm - #11

Closed
rgarcia wants to merge 5 commits into
hypeship/retail-cartfrom
hypeship/retail-cart-pinned-configs
Closed

rgarcia wants to merge 5 commits into
hypeship/retail-cartfrom
hypeship/retail-cart-pinned-configs

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10.

Summary

Task and verifier (retail-cart):

  • The task asks for the outcome, not the route: put one item in the site's cart as a guest and show the cart. The agent may reach the item however works, including a web search to identify the exact product or part number; the cart must be on the site.
  • The verifier now checks the cart itself. It opens the task's cart_url, or fetches its cart_api from the page the agent ended on, and sets cart_verified when the text matches cart_match. cart-check.json keeps what it saw. reward is still the agent's own report.

Pinned arm (codex-repl-pinned): a pinned configurator gives each site fixed settings from configurators/pinned/sites.toml:

  • browser: stealth, gpu
  • proxy: a new US ISP or residential proxy per trial
  • persona: macOS or WebGL, applied by restarting Chromium with KERNEL_* launcher variables
  • vm_setup: shell commands run as root in the browser VM before the first page load. On wbmason.com this empties the captcha-solver extension's Turnstile interceptor, which otherwise takes over the site's Cloudflare Turnstile check and never completes it.
  • steering: per-site guidance added to the instruction, e.g. reload once on zoro.com, the order-code pages on newark.com, the trailing-slash search on guitarcenter.com, and add-by-part-number from the cart on mouser.com.

The arm also enables Kernel MCP's computer_action next to browser_repl.

Harness:

  • KernelTaskEnvironment runs the Chromium restart and vm_setup right after creating the browser. Either failing fails the trial with VmSetupError.
  • GPU browsers are created without a memory request, which the API rejects with gpu=true.
  • kernel-mcp-local accepts the computer toolset.

Results

10 attempts per site, both arms in the same window, 2026-11-01. "Cart" counts trials where the verifier found the item in the cart.

Site Pinned cart Pinned reported Stealth cart Stealth reported
costco.com 10/10 10/10 10/10 10/10
harborfreight.com 10/10 10/10 10/10 10/10
homedepot.com 10/10 10/10 10/10 10/10
staples.com 10/10 10/10 10/10 10/10
target.com 10/10 10/10 9/10 9/10
walmart.com 10/10 10/10 10/10 10/10
wbmason.com 10/10 10/10 5/10 5/10
dickssportinggoods.com 9/10 9/10 8/10 8/10
ebay.com 9/10 10/10 6/10 8/10
amazon.com 8/10 10/10 9/10 10/10
newark.com 8/10 9/10 0/10 0/10
mouser.com 7/10 8/10 0/10 0/10
guitarcenter.com 5/10 0/10 0/10 0/10
radwell.com 4/10 4/10 4/10 4/10
lowes.com 0/10 10/10 0/10 9/10
zoro.com 0/10 0/10 2/10 2/10
Total 120/160 130/160 93/160 105/160

Where the two counts disagree:

  • lowes.com: the agent added a Bostitch 27-pack metal kit; the task names a Stanley one, which the site no longer seems to list. The item needs updating.
  • amazon.com, ebay.com, mouser.com (4 trials): opening the cart page during verification hit a checkpoint, a captcha or a timeout. The check needs to tell "couldn't check" from "not in cart".
  • guitarcenter.com: the item was in the cart (cart API) in 5 trials where the agent reported failure, because the cart page never renders the line items.
  • newark.com: one reported success over a cart that showed 0 items.

Testing

  • ruff check, ruff format --check, pytest (26 passed)
  • smoke run of the five previously blocked sites plus costco, then the 10-attempt run above
  • checked the extension file path and command on a live browser before the run

@rgarcia rgarcia changed the title Add pinned per-site configurations to retail-cart, with Chromium launcher personas Ask retail-cart for the outcome, verify the cart, and add the pinned per-site arm Nov 1, 2026
@rgarcia

rgarcia commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #10 (retail-cart) and #9 (generic harness changes) so the stack is one generic PR and one benchmark PR.

@rgarcia rgarcia closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant