Browser Control M1 (read-only): stream browser_control per-action progress

[ianu82]

Streams human-readable, per-action progress for the host-registered browser_control tool (Browser Control Milestone 1, read-only). When the LLM dispatches browser_control, the session now yields a StreamTaskProgress(phase="browser_action") event before and after tool execution, carrying the tool call's progress_message (e.g. "Opening the July report") so the host UI can show live "Browsing …" status with an ETA. Anton stays host-agnostic: the tool itself is defined and executed by the host (cowork-server); anton only surfaces its progress.

Changes

• anton/core/session.py: special-case in _stream_and_handle_tools for tool calls named browser_control that carry a truthy progress_message argument — yields a pre-dispatch StreamTaskProgress(phase="browser_action", message=<progress_message>, id=<tool_call_id>) and a post-dispatch event with eta_seconds. All other tools fall through to the generic path unchanged.
• tests/test_chat_browser_progress.py: unit tests — phase carries the human message, fallback behavior without progress_message, other tools unaffected.
• Review fixes (06b5c4b, b79852d, from PR review): StreamDisplay.update_progress (anton/chat_ui.py) gained an explicit browser_action branch so the CLI marks the browser activity done with its elapsed/✔ line on completion (previously browser_action fell through as a status update and the activity never completed). The tool_use id is now threaded through update_progress (call sites in anton/chat.py, anton/commands/goal.py) so completion matches the exact activity when multiple browser actions are pending, with a name-scan fallback.
• tests/e2e/scenarios/test_browser_tool.py: two e2e scenarios against the in-process ChatSession.turn_stream with a stub LLM server and a fake browser_control ToolDef — a three-step read-only task streams human progress; a connector-preferred task never emits browser_control.

How to review

1. anton/core/session.py — the only behavioral change: the browser_control special-case in _stream_and_handle_tools. Check the pre/post StreamTaskProgress(phase="browser_action") pair and that every other tool falls through to the generic tool_start/tool_done path unchanged.
2. anton/chat_ui.py (StreamDisplay.update_progress) — the browser_action branch: eta_seconds=None = status update, eta_seconds set = mark the matching activity done; matching is by threaded tool_use id with a name-scan fallback.
3. anton/chat.py / anton/commands/goal.py — mechanical call-site updates threading the id through.
4. tests/test_chat_browser_progress.py + tests/e2e/scenarios/test_browser_tool.py — the e2e stream-shape guard pins exactly two browser_action events per action and no generic tool events for browser_control; this is the contract cowork-server's stream formatter depends on.

Demo

Desktop demo recording (mp4, ~5 min, 2.6 MB) — recorded on the final code of this stack against a real cowork-server, real Electron app, and a real Chrome window on a local fixture site. It shows: connect + one-tab approval (awaiting-approval → connected), the Browser Control connection card, real dispatched inspect → follow_link (Chrome visibly navigates to july.html) → scroll → wait through the live server command queue → Electron poller → CDP path with content-free observed digests, then Disconnect/revoke and a refused post-revoke dispatch ("The browser was stopped or handed back before this action ran."). The asset is hosted on the throwaway branch vorflux/browser-control-m1-demo-assets (cowork repo); safe to delete after the stack merges.

Testing

Automated (all run by the testing subagent on this branch):

• uv run --extra dev pytest tests/test_chat_browser_progress.py tests/e2e/scenarios/test_browser_tool.py -v — 5 passed.
• uv run --extra dev pytest tests/ --ignore=tests/e2e — 1208 passed, 17 skipped on the staging base (branch rebased from main onto staging; feature diff unchanged).
• uv run --extra dev pytest tests/e2e/ — 37 passed.
• The e2e stream-shape guard pins exactly two browser_action events per action (start with eta_seconds=None, done with elapsed) and no generic tool_start/tool_done for browser_control, protecting cowork-server's stream formatter.

Cross-repo integration (server↔Electron broker seam with a real headless Chrome and the real cowork-server) was exercised in the sibling PRs' testing; anton's contribution there is the streamed browser_action phase consumed by cowork-server's stream formatter.

Live real-LLM verification (added 2026-07-16 after ANTHROPIC_API_KEY became available): the previously-blocked connector-first + cited-answer acceptance check now PASSES with real Claude calls — a real chat turn streamed the browser_action progress phases while dispatching inspect then follow_link through the real cross-repo path, and the final answer correctly cited "July Total Revenue: $1,234,567" from the fixture; Stop during a live streaming turn completed in 13.7ms with in-flight calls refused cleanly. Full live evidence is in the sibling cowork/cowork-server PRs. Remaining caveat: no actual connector was configured in the dev env, so a live connector-vs-browser routing decision was not observable (the preference is confirmed via prompt/schema, unit test, and live reason usage).

Testing caveats (disclosed per policy):

• The end-to-end testing subagent hit six 3600s timeouts and one platform-internal error across its runs (the third during the demo-recording run, the fourth during the live-LLM round, the fifth and sixth during the Stop→resume verification round — in each case the round's work had completed and its results were recovered from the subagent's preserved report); it was resumed from its preserved transcripts each time and the suites above completed with the recorded results. See the sibling cowork/cowork-server PRs for the full integration/UI evidence and any remaining unverified cases.

[ianu82]

https://linear.app/mindsdb/issue/ENG-815/claude-parity-browser-use

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21b741b82e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve completion semantics for browser actions

When browser_control is registered on the standard streaming path, this branch replaces the generic tool_done event with a second browser_action event. The in-repo CLI consumer forwards phases to StreamDisplay.update_progress, which only marks non-scratchpad tool activities done on tool_done; browser_action falls through as a status update, so the browser activity never gets its elapsed/✔ completion line. Either keep emitting a completion phase the existing display understands or add explicit browser_action completion handling keyed on eta_seconds.

Useful? React with 👍 / 👎.

[ianu82]

Addressed in 06b5c4b + b79852d. StreamDisplay.update_progress now has an explicit browser_action branch: without eta_seconds it's a live status update (progress message on the spinner); with eta_seconds it marks the matching browser_control activity done with work_elapsed, so the ✔/elapsed completion line prints through the existing paths. Chose the display-side fix over emitting an extra tool_done to avoid a redundant progress line in cowork-server's stream formatter. Follow-up b79852d threads the tool_use id through update_progress so completion matches act.tool_id exactly when multiple browser actions are pending (name-scan fallback retained). Covered by new tests incl. the two-pending-activities case and an e2e guard pinning exactly 2 browser_action events per action with no generic tool_start/tool_done.