fix(computer-use): bound AX payloads, report true image dimensions, cap osascript - #2225
Merged
Conversation
…ap osascript Four follow-ups left open by #2224, each a case where a result told the agent something that was either far too large or quietly untrue. **`get_app_state` was unbounded.** `describe_screen` got a 60 KB cap; the explicit query did not. Measured unbounded output on real Electron apps was 220-390 KB from a single call — roughly 100k tokens spent on one look at one app. Capped at 120 KB (higher, because asking for an app's tree is an explicit request, but still a ceiling), reusing the same clip that lands on a char boundary and announces itself. Applied in `snap_state_json`, so `app_click` / `app_type_text` / `app_scroll` / `app_key_chord` / `app_wait_for` are covered too — they all carry the same post-action tree and shared the same risk. **`analyze_image` reported the resized dimensions as the file's.** Large screenshots get downscaled to fit the provider (repeated 0.75x passes, floor 64px), and `ProcessedImage` kept only the final size — the source dimensions were computed and dropped. So a caller mapping anything the vision model said back to the screen was off by an unknown factor, with nothing in the result hinting at it. `ProcessedImage` now carries `original_width` / `original_height` with `scale()` and `was_resized()`; `analyze_image` and `view_image` report both frames. `analyze_image` also states plainly what its numbers are: any position in the prose is a vision model's estimate in the resized frame, not a measurement and not a click target — use `locate` / `move_to_text` / `describe_screen`, which return real coordinates. Building a coordinate contract on estimated numbers would be worse than saying they are estimates. **`open_app` could block for two minutes.** `activate` sends an AppleEvent and waits for the app to answer; a hung app does not, and macOS's default AppleEvent timeout is 120s, held on a blocking thread with the agent unaware. `Command::output()` has no timeout, so osascript now runs under a polled 10s deadline and is killed past it. A timeout reports as a failed launch the agent can act on, not an opaque io error. **`interaction_state.displays` rode on every result.** On a single-screen machine it repeats what `active_display_id` already says. Sent only when more than one display is attached; `list_displays` and `describe_screen` still report the full list on demand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接 #2224 的四个遗留项。共同点:结果要么大得离谱,要么悄悄告诉了模型不实的数字。
1.
get_app_state完全没有上限#2224 给
describe_screen加了 60 KB 截断,但显式查询没加。实测两台真实 Electron 应用:单次调用 ~100k token,一眼就吃掉大半个上下文窗口。
现在上限 120 KB——比
describe_screen高,因为这是模型显式要整棵树,但仍然是个天花板。复用同一套按字符边界截断 + 显式声明的逻辑。关键是加在
snap_state_json而不是get_app_state调用点:app_click/app_type_text/app_scroll/app_key_chord/app_wait_for的结果都带同一棵事后树,本来就共享同一个风险。2.
analyze_image把缩放后的尺寸当成原图尺寸报optimize_image_with_size_limit会把大图缩小(反复 0.75× 直到过关,下限 64px),但ProcessedImage只留最终尺寸——原图尺寸在image_processing.rs:204算完就丢了。于是模型拿到的
width/height根本不是它传进去那个文件的尺寸,而结果里没有任何东西提示这一点。日志里模型反复算错 bbox,这是原因之一(另一半是 Retina 2x)。ProcessedImage现在带original_width/original_height,配scale()和was_resized();analyze_image和view_image两个坐标系都报。另外
analyze_image现在直说它的数字是什么:analysis里任何位置都是视觉模型在缩放后画面里的估计,不是测量值,也不是点击目标——要操作屏幕请用locate/move_to_text/describe_screen,那些返回真实坐标。3.
open_app可能阻塞两分钟activate向目标应用发 AppleEvent 并等回复。应用卡死就不回复,而 macOS 默认 AppleEvent 超时是 120 秒——期间占着一个 blocking 线程,模型完全不知道出了什么事。Command::output()没有超时,所以改成轮询try_wait的 10 秒预算,超时直接 kill。超时会作为"启动失败"报给模型(附带可执行的下一步),而不是抛一个看不懂的 io error。测试用
delay 30配 700ms 预算验证真的会被杀掉,整个套件跑完 0.71s。4. 单显示器上
displays是纯重复interaction_state挂在每一个 ComputerUse 结果上(均值 624 字节),其中displays数组在单屏机器上说的话active_display_id已经说完了。改成只在多显示器时输出;list_displays和describe_screen仍然随时可查。顺带发现的深度数据
重跑深度剖面,换了台应用,结论一致(这也验证了 #2224 里 depth 20 的选择能泛化):
验证
新增测试都确认过"没有修复时会失败",不是摆设:
scale()真的能把原图宽映射回发送宽delay 30+ 700ms 预算,断言TimedOut且 5 秒内返回const _: () = assert!(...)编译期检查,颠倒顺序直接构建失败