Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions code_puppy_core_plugins/browser_harness/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Browser Harness

This opt-in plugin lets Code Puppy drive **your real browser** — your logins,
cookies, extensions, and tabs — through
[browser-harness](https://github.com/browser-use/browser-harness), a small CDP
harness that keeps one tab attached across calls. It is a thin, honest wrapper:
Code Puppy shells out to the `browser-harness` CLI, which owns the browser
connection.

## Two installs, on purpose

1. The plugin itself ships with `code-puppy-core-plugins`; nothing to do.
2. The harness is a separate tool, because it owns its own daemon and upgrades:

```bash
uv tool install --python 3.12 --upgrade --force browser-harness
```

`/browser status` tells you which of the two is missing.

## Consent and configuration

Browser control is off by default; its tools are not exposed to the model until
you opt in, exactly like macOS Computer Use.

```text
/browser status # install state, consent, endpoints, browsers
/browser enable | disable # persisted one-time consent
/browser doctor # harness health, with the exact fix
/browser connect <devtools-url> # pin http://127.0.0.1:9222 or a wss:// URL
/browser disconnect # back to auto-discovery
/browser install # commands to install a drivable browser
/browser recordings [on|off] # the harness's own local trace recording
```

Disabling removes the tools again and blocks further calls. `/browser enable`
merges the tools into the running session, so no restart is needed.

## Tools

- `browser_harness(script, browser_name=None, timeout=120)` — run Python with
the harness helpers (`new_tab`, `js`, `click_at_xy`, `cdp`, `wait_for_load`,
…) pre-imported. stdout is the only channel back.
- `browser_screenshot(full=False, max_dim=1568)` — capture the attached tab. It
renders inline in Ghostty, Kitty, WezTerm, and iTerm2, and the same PNG rides
along on the tool result for multimodal models.
- `browser_doctor()` — connection health with the fix that clears it.

The bundled `SKILL.md` teaches the workflow: one tab per task, accessibility
tree before pixels, verify each action, stop for passwords and purchases.

## Which browsers work

browser-harness speaks the Chrome DevTools Protocol, so it needs a
Chromium-family browser: **Chrome, Chrome Canary, Chromium, Brave, Edge, Arc,
and Helium** are detected and drivable.

**Firefox and Safari cannot be driven.** Firefox implements WebDriver BiDi and
Safari implements the Apple WebKit inspector protocol; neither exposes the CDP
endpoint the harness requires. `/browser status` lists them as *present but not
drivable* rather than pretending they are usable. Options:

- install a Chromium-family browser (`/browser install`), or
- point at any CDP endpoint elsewhere: `/browser connect http://host:9222` for a
Chromium started with `--remote-debugging-port=9222`, or a hosted browser.

macOS shows a per-connection "Allow remote debugging?" sheet, and Chrome's
`chrome://inspect/#remote-debugging` toggle must be ticked once. `browser-harness
mac-approve` clears the sheet; `browser_doctor()` tells you when either is the
blocker.

## Privacy

Nothing here phones home beyond browser-harness's own optional telemetry
(`browser-harness telemetry disable`). Code Puppy stores only two settings — the
consent flag and an optional endpoint — in its config directory. Page content,
screenshots, and any recordings stay on your machine under the harness's state
directory.

## Test

```bash
uv run pytest tests/test_browser_harness_*.py -q --no-cov
```
87 changes: 87 additions & 0 deletions code_puppy_core_plugins/browser_harness/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
name: browser-harness
description: Use for any web task that needs the user's real browser - logged-in
sessions, their cookies, clicks, uploads, downloads, forms, or local web apps.
Public read-only pages need no browser.
version: "1.0"
author: code-puppy
tags:
- web
- browser-automation
- cdp
---

# Browser Harness

Drive Mike's actual browser through `browser-harness`, a CDP harness that keeps
one tab attached across calls. Adapted from
[browser-use/browser-harness](https://github.com/browser-use/browser-harness)
(MIT); the upstream `SKILL.md` (`browser-harness skill`) is the canonical,
version-matched reference and goes deeper than this file.

## Pick the right browser tool

- Plain HTTP gets it (public page, API, docs): use `curl`/`web_fetch`. Do not
open a tab.
- Sandboxed scraping, crawling, or parallel extraction: delegate to the
`web-retriever` agent, which drives a throwaway Playwright browser.
- Anything needing *this* browser - existing logins, cookies, local
`localhost` apps, extensions, uploads, downloads, or "watch me do it":
use the `browser_harness` tool.

## Tools

- `browser_harness(script)` - run Python with the harness helpers pre-imported.
stdout is the only channel back, so `print()` what you need.
- `browser_screenshot()` - capture the attached tab; it renders inline for the
user. Pixels never tell you what is clickable, so confirm with `page_info()`
or `js()`.
- `browser_doctor()` - install/daemon/browser health, with the exact fix.

## Workflow

1. First navigation of a task: `new_tab(url)`. The attached tab survives across
calls, so do **not** call `new_tab()` again in every script. Check
`current_tab()` / `list_tabs()` and `switch_tab()` before opening duplicates;
never close a tab you did not create.
2. After navigation call `wait_for_load()`; after a click that triggers a
request, `wait_for_network_idle()`.
3. Find elements in the accessibility tree, not pixels:
`cdp("Accessibility.getFullAXTree")["nodes"]` carries role, name, and
`backendDOMNodeId`. Filter it in Python - it is thousands of nodes. Then box
center -> `click_at_xy(x, y)` -> verify with a targeted `js()` or
`page_info()` check.
4. Fall back to `js(...)` for DOM/extract work, and screenshots only when
layout or imagery is the question.
5. An action that does nothing usually means the attached tab is hidden: call
`activate_tab(current_tab())`, retry the same action once, then re-check.
This visibly switches tabs, so skip it if the user asked you not to touch
their foreground.
6. Write the reusable part into `$BH_AGENT_WORKSPACE/agent_helpers.py` when a
site-specific trick took real discovery; keep task code in the tool call.

## Which browser

The harness speaks CDP, so Chrome, Chromium, Brave, Edge, Arc, or Helium work.
**Firefox and Safari cannot** - they expose no CDP endpoint. If asked for
Firefox, say so and offer a Chromium-family browser or an explicit endpoint.

- `/browser status` lists what is installed, running, and reachable.
- `/browser connect http://127.0.0.1:9222` (or a `wss://` URL) pins a specific
endpoint; leave it unset to auto-discover the running Chromium browser.
- Cloud browsers: `browser_harness(script='start_remote_daemon("name")')`, then
pass `browser_name="name"` on every later call. Ask before leaving one
running, and stop it with `stop_remote_daemon("name")`.

## Ask first

Stop and ask before typing passwords, solving MFA, approving a payment or
purchase, deleting an account, sending a message, or anything that posts
content as the user. Being already signed in is not consent to act.

## When it will not connect

`browser_doctor()` (or `/browser doctor`) names the fix. The usual three:
Chrome's `chrome://inspect/#remote-debugging` toggle is off; macOS is waiting on
the "Allow remote debugging?" sheet (`browser-harness mac-approve`); or no
Chromium browser is running at all.
1 change: 1 addition & 0 deletions code_puppy_core_plugins/browser_harness/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Opt-in browser control through browser-harness."""
Loading