From 7e0b25dffaafcd7e5ce2c0f4ea37ab1783cccdf5 Mon Sep 17 00:00:00 2001 From: Bob Lee Date: Tue, 11 Aug 2026 08:41:46 -0700 Subject: [PATCH] fix(computer-use): bound AX payloads, report true image dimensions, cap osascript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four follow-ups left open by #2224, each a case where a result told the agent something that was either far too large or quietly untrue. **`get_app_state` was unbounded.** `describe_screen` got a 60 KB cap; the explicit query did not. Measured unbounded output on real Electron apps was 220-390 KB from a single call — roughly 100k tokens spent on one look at one app. Capped at 120 KB (higher, because asking for an app's tree is an explicit request, but still a ceiling), reusing the same clip that lands on a char boundary and announces itself. Applied in `snap_state_json`, so `app_click` / `app_type_text` / `app_scroll` / `app_key_chord` / `app_wait_for` are covered too — they all carry the same post-action tree and shared the same risk. **`analyze_image` reported the resized dimensions as the file's.** Large screenshots get downscaled to fit the provider (repeated 0.75x passes, floor 64px), and `ProcessedImage` kept only the final size — the source dimensions were computed and dropped. So a caller mapping anything the vision model said back to the screen was off by an unknown factor, with nothing in the result hinting at it. `ProcessedImage` now carries `original_width` / `original_height` with `scale()` and `was_resized()`; `analyze_image` and `view_image` report both frames. `analyze_image` also states plainly what its numbers are: any position in the prose is a vision model's estimate in the resized frame, not a measurement and not a click target — use `locate` / `move_to_text` / `describe_screen`, which return real coordinates. Building a coordinate contract on estimated numbers would be worse than saying they are estimates. **`open_app` could block for two minutes.** `activate` sends an AppleEvent and waits for the app to answer; a hung app does not, and macOS's default AppleEvent timeout is 120s, held on a blocking thread with the agent unaware. `Command::output()` has no timeout, so osascript now runs under a polled 10s deadline and is killed past it. A timeout reports as a failed launch the agent can act on, not an opaque io error. **`interaction_state.displays` rode on every result.** On a single-screen machine it repeats what `active_display_id` already says. Sent only when more than one display is attached; `list_displays` and `describe_screen` still report the full list on demand. --- .../src/computer_use/desktop_host/mod.rs | 133 +++++++++++++++--- .../image_analysis/image_processing.rs | 112 +++++++++++++++ .../implementations/analyze_image_tool.rs | 34 ++++- .../implementations/computer_use_actions.rs | 10 +- .../implementations/computer_use_tool.rs | 72 ++++++++-- .../tools/implementations/view_image_tool.rs | 8 ++ .../tool-contracts/src/computer_use.rs | 5 + 7 files changed, 342 insertions(+), 32 deletions(-) diff --git a/src/apps/desktop/src/computer_use/desktop_host/mod.rs b/src/apps/desktop/src/computer_use/desktop_host/mod.rs index d2a3fa13e..c3fcd69a6 100644 --- a/src/apps/desktop/src/computer_use/desktop_host/mod.rs +++ b/src/apps/desktop/src/computer_use/desktop_host/mod.rs @@ -48,6 +48,55 @@ const OPEN_APP_WINDOW_WAIT_MS: u64 = 8_000; #[cfg(target_os = "macos")] const OPEN_APP_POLL_INTERVAL_MS: u64 = 150; +/// How long an `open_app` AppleScript may run before it is killed. +/// +/// `activate` sends an AppleEvent to the target app and waits for it to answer. +/// A hung or busy app simply does not answer, and macOS's default AppleEvent +/// timeout is **120 seconds** — during which `open_app` occupies a blocking +/// thread and the agent has no idea anything is wrong. An app that has not +/// acknowledged activation in a few seconds is not going to. +#[cfg(target_os = "macos")] +const OSASCRIPT_TIMEOUT_MS: u64 = 10_000; + +/// Run `osascript -e