Skip to content

feat(browser): click without indices, scroll inner containers, send modifiers, and add opt-in page tools - #123

Open
youli42 wants to merge 2 commits into
omdsh-dev:mainfrom
youli42:feat/p0-p1-browser-capabilities
Open

youli42 wants to merge 2 commits into
omdsh-dev:mainfrom
youli42:feat/p0-p1-browser-capabilities

Conversation

@youli42

@youli42 youli42 commented Oct 18, 2026

Copy link
Copy Markdown

Problems

The browser surface addresses everything through snapshot indices and the window scroller, which leaves common pages readable but not operable:

  • Pagination / JS-only controls are unclickable. The inventory whitelist (a[href], button, inputs, a few roles) skips <a> without href, <li onclick> rows and cursor:pointer cells; the 60-item cap then pushes bottom-of-page pagination out. The model sees "Next" in the text but has no index to click.
  • Inner scroll containers cannot be scrolled. browser_scroll only moves the window, so app-shell layouts (body { overflow: hidden } plus an inner scrolling div) do not move at all.
  • Modifier combinations are dropped. browser_press builds KeyboardEvent without modifier state, so page-level shortcuts such as Control+Enter are inexpressible.
  • Page storage, page-origin requests and page JavaScript are unreachable. The extension already holds scripting and all-site host permissions, so this is a tool-contract gap, not a platform limit.
  • The form-field cap is hardcoded at 30 in the content script with no negotiation path, so dense pages require hand-patching built bundles.

Changes

  • browser_click accepts exactly one of index, selector, or text (plus role/nth); text resolution walks to the nearest actionable ancestor and reports ambiguity instead of guessing. Index addressing keeps its document-binding guard.
  • browser_snapshot lists inner scroll containers with a unique selector, scroll percentage and remaining pixels; browser_scroll moves an addressed container.
  • browser_press accepts modifiers.
  • Opt-in (advancedTools: true, default off): browser_get_storage, browser_fetch, browser_eval. Approval-gated; bulk storage listings mask credential-looking values; fetch runs from the page origin; eval runs in the page main world and polls async results. Fetch/eval never receive persistent origin trust, and every payload crosses the untrusted-content boundary.
  • maxForms is negotiated like the other two snapshot caps (protocol, plugin config, budget push, per-frame allocation).

Testing

  • pnpm run typecheck and pnpm run build pass.
  • Extension: 437 tests pass. Bridge: 151/152 — the one failure (token.spec, 0600 mode) is pre-existing on Windows and fails on a clean checkout too.
  • New tests cover selector/text addressing and ambiguity, modifiers, inner-container scrolling, storage masking, fetch construction/CORS failure, main-world evaluation and polling, approval denial, and negotiated maxForms.

Known limits, documented in the READMEs: browser-reserved shortcuts and native editing defaults still need a CDP/debugger path; browser_eval is subject to page CSP and cannot preempt an infinite loop. The new eval path was unit-tested by executing the injected function directly, not smoke-tested in a real browser.

…and opt-in page tools

P0 — operate pages the model could already read:

- browser_click now accepts exactly one of index, CSS selector, or visible
  text (plus role/nth filters). Text resolution scans direct text nodes and
  accessible names, then walks to the nearest actionable ancestor, so
  JS-only pagination, <li onclick> rows, and cursor:pointer cells become
  clickable. Ambiguity is reported with candidate positions instead of
  guessing.
- browser_snapshot discovers inner scroll containers and prints each with a
  unique selector, scroll percentage, and remaining pixels. browser_scroll
  moves the addressed container (or the nearest scrollable ancestor of an
  indexed element); a window scroll that did not move points at the region
  list. App-shell layouts are no longer unscrollable.
- browser_press accepts ctrl/meta/shift/alt modifiers and derives
  KeyboardEvent.code for common keys, so page-level shortcuts such as
  Control+Enter can be expressed.

P1 — page-crossing tools behind the existing approval machinery, opt-in via
the bridge config flag advancedTools (default false; the bundled cordis
profile maps it to DSH_ADVANCED_TOOLS=1):

- browser_get_storage reads localStorage/sessionStorage; bulk listings mask
  credential-looking values, naming a key explicitly reads it unmasked.
- browser_fetch sends method/headers/body from the page origin with the
  page's cookies (same-origin APIs work directly; cross-origin stays subject
  to CORS).
- browser_eval evaluates JavaScript in the page main world, supporting both
  expression and statement/return forms, with async results polled for 3s.
- Storage is approved as a page read; fetch and eval are actions that never
  offer persistent origin trust; every payload crosses the untrusted-content
  boundary.

Tests: new actions-advanced, snapshot-regions, and background-advanced-tools
specs cover addressing, modifiers, container scrolling, storage masking,
request construction, main-world injection, polling, approvals, and the
sharing-off gate. Docs and bilingual pairing hashes updated, plus an
architecture agent note recording the decision and its limits.
maxForms was hardcoded in the content script (30) with no path through the
handshake, so dense pages could only expose more form fields by hand-patching
the built content.js. Make it a first-class negotiated cap:

- protocol: add DEFAULT_MAX_FORMS and an optional BridgeCaps.maxForms. Absent
  stays compatible with an adjacent-release peer; present must be a positive
  integer.
- bridge: new maxForms plugin config (default 30, validated) included in the
  hello caps; cordis.patch.yml documents the three snapshot caps.
- extension: ContentBudget.maxForms flows through contentBudgetFromCaps, the
  DSH_BUDGET push, and per-frame allocation. The content script default now
  uses the shared constant instead of a literal.
- tests: caps validation, config defaults/validation, per-frame partitioning,
  and an end-to-end snapshot that only shows the 40th field because the
  negotiated form cap replaced the built-in default.

Bridge README documents the three negotiable caps; pairing hashes re-recorded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant