Skip to content

bench: amazon.com intents for #56 and a Haiku 5.5 pilot - #123

Merged
yummybomb merged 1 commit into
mainfrom
hypeship/bench-amazon
Oct 19, 2026
Merged

yummybomb merged 1 commit into
mainfrom
hypeship/bench-amazon

Conversation

@yummybomb

Copy link
Copy Markdown
Contributor

Summary

  • Adds five amazon.com intents that run the read-only tools from Add Amazon shopping and guarded retail cart tools #56 (search_products, get_product, get_best_sellers, get_cart). The bench registers only readOnlyHint: true tools, so Add Amazon shopping and guarded retail cart tools #56's cart-write tools stay out.
    • amazon-search-first-result (find): the first non-sponsored result for "usb c cable".
    • amazon-product-details (inspect): title, price, seller, rating and rating count for one ASIN.
    • amazon-best-sellers-top3 (find): the top three Kitchen & Dining Best Sellers, with an asin agreement check.
    • amazon-cheapest-well-rated (compare): the cheapest result rated 4.5 stars or more among the first 10 non-sponsored results, with a price lower-is-better agreement check.
    • amazon-return-window (gap): the standard return window, pinned to 30 days. No tool covers it, so it is the placebo.
  • Adds the answer schemas these intents need.
  • Changes the README pilot model to anthropic/claude-haiku-5-5 --thinking medium, and makes ANTHROPIC_API_KEY the example key.

Amazon shows a plain Kernel browser a "Continue shopping" interstitial, so run these intents with --stealth.

Runs

All three runs: --domain amazon.com --adoption 4 --pairs 5 --slots 4 --stealth, with #56 pinned at 12b5b74. Each run scored 54 trials and aborted none.

intent gpt-5.4 speedup / adoption haiku-5-5 medium haiku-5-5 low
search-first-result 3.2× faster, 100% 0.7×, 60% 0.7× slower, 60%
product-details 3.3× (no signal), 100% 0.6×, 40% 0.8×, 20%
best-sellers-top3 3.2× faster, 100% 1.0×, Δ correct −2 0.8×, Δ correct −2
cheapest-well-rated 3.5× faster, 100% 0.8×, 80% 0.8×, 80%
return-window (gap) 0.6×, 0% 0.9×, 0% 0.8×, Δ correct −1
spend $5.48 $0.24 $0.17
  • gpt-5.4: every trial was correct in both arms. Treatment invoked a tool in every trial on the four tool-covered intents and was faster in 5 of 5 pairs on three of them.
  • Haiku 5.5: control is already fast (median 13–60s per intent), so the extra turns spent listing and invoking tools make treatment no faster. It also skips the tools more often. Medium invoked a tool in more trials than low and got every gap-intent answer right, so the pilot uses medium.
  • Best Sellers: Haiku's treatment misses are all on this intent. get_best_sellers returns price: null for some ranked items, and Haiku passed the null through instead of calling get_product, as gpt-5.4 did.

Testing

  • node --test "bench/**/*.test.mjs": 97 pass.

@yummybomb
yummybomb marked this pull request as ready for review October 19, 2026 20:26
@yummybomb
yummybomb merged commit eefd79c into main Oct 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant