fix(computer-use): restore text-only observation and unblock the Enter path - #2224
Merged
Conversation
…r path A single Feishu "send one message" run took 73 tool round-trips. Tracing the log back, most of them came from four defects that compound. **`describe_screen` was blind on macOS.** `macos_foreground_application` built its AppleScript with a `try … end try` *block* in expression position, which AppleScript rejects at compile time (-2741). The command always exited non-zero, so the function always returned `None` — and `describe_screen` derives its target app from that value. Every call reported `foreground_application: null` and `ax_tree_text: null`. The agent reasonably concluded its own tool output was being truncated, stopped trusting results, and fell back to `screencapture` + image-analysis as a substitute for eyes: 10 extra model round-trips that returned prose instead of coordinates and misread the screen repeatedly. Replaced with an in-process `NSWorkspace.frontmostApplication` read, and did the same for `macos_ax_ui::frontmost_pid`, which shelled out to `osascript` on every call — three spawns per `describe_screen`, each ~120ms, each able to block on a System Events AppleEvent timeout. **Enter was permanently refused in text-only mode.** The stale-capture guard is cleared only by a successful capture, but a text-only `screenshot` short-circuits before capturing. So the guard latched: its own error said "call `screenshot` first", calling it changed nothing, and every `click`/Enter stayed blocked. The observed escape was the agent bypassing the tool with raw `osascript … keystroke return`, which skips every check the guard exists to enforce. `describe_screen` (the text-only equivalent of looking) and the text-only `screenshot` stub now waive it. `paste` also clears it unconditionally instead of only when `submit:true` — paste-then-Enter is the common shape and the pointer never moved. **Results were mostly duplicate.** Every `app_state` carried both `tree_text` and `app_state_nodes`, the same nodes re-serialised as verbose JSON. `render_tree_text` already emits every addressable field, nothing consumed the array, and it was 82 KB of a 107 KB result. Dropped, `node_count` kept. **`get_app_state` returned mostly closed menus.** Observing a windowless app produced 188 nodes, 180 of them menu items at zero-size off-screen frames — unclickable until the menu opens. Closed `AXMenu` subtrees are no longer walked; the container stays visible and a note points at `get_app_shortcuts`. Also: `open_app` reported `success: true` for an app running with no window, which is how this run lost ~15 calls rediscovering that `activate` does not reopen an Electron window. It now resolves the bundle id (the launch name, executable name and bundle id are routinely three different strings), polls for a window, retries via `open -b`, and reports `window_count` / `windowless`. Tests: AppleScript templates are now compile-checked with `osacompile`, which catches exactly the class of bug above without executing anything. Drive-by: `embedded_relay_host` tests reserved an ephemeral port, dropped the listener, then assumed it was still free. Harmless on an idle machine and ~80% failing once the suite spawns subprocesses. Port acquisition and the release assertions now retry.
The macOS branch gained bundle_id / process_name / window_count / launch_path; the other two construct the same struct and would not compile without them. Both leave the identity fields None rather than guessing: neither `start` nor `xdg-open` reports what it launched, so there is no pid to resolve identity or a window count from. `window_count: Some(0)` would tell the model the app is definitely windowless when it was simply never measured.
…names Only "Safari" was covered, which never exercises applescript_quote. Compiling the escaped forms of a quote, a backslash and CJK is what proves the escaping matches AppleScript's actual string-literal syntax rather than a plausible guess about it.
The two lines this prompt was missing are the ones that would have ended the observed failure early. When describe_screen returned nulls the agent concluded its own output was truncated and improvised screencapture plus image analysis as a substitute for eyes. State plainly that an empty ax_tree_text is a result with a reason attached (ax_tree_status / ax_tree_note), that re-calling returns the same thing, and that building eyes out of screencapture costs a model round-trip per glance and returns prose instead of coordinates.
…tron apps
Fixing `foreground_application` gave `describe_screen` an app to look at.
It was still walking only 8 levels into the focused window, which is fine
for a native Cocoa app and far too shallow for the Electron / WebView
clients agents are most often asked to drive.
Measured against a real Electron window (focused window only):
depth 8: 17 nodes, 7 actionable, 1 KB
depth 12: 25 nodes, 15 actionable, 2 KB
depth 16: 50 nodes, 40 actionable, 5 KB
depth 20: 207 nodes, 197 actionable, 27 KB
depth 24: 233 nodes, 223 actionable, 31 KB
depth 32: 1289 nodes, 1279 actionable, 206 KB
Seven actionable elements is not enough to find a search field or a send
button, so the tree read as "this app has no AX tree" and pushed the
agent onto OCR and screenshot guessing. The actionable layer appears
around 20; past it the payload grows far faster than the number of things
worth clicking.
Depth is a poor proxy for size, though — a document or a long list can
multiply that 27 KB — so the returned tree is also capped at 60 KB, cut on
a line boundary. The clip announces itself and says a control that is
missing from the view may still exist: an agent that reads a truncated
tree as the whole UI concludes the control is not there and gives up.
The `#[ignore]`d dump test now prints this depth profile, so the constant
can be retuned against evidence rather than intuition.
…e index The 60 KB cap I added in the previous commit sliced the tree at a byte offset. `&str[..n]` panics when `n` falls inside a multi-byte character, so a CJK app tree — the kind most likely to be large enough to hit the cap in the first place — would panic the whole tool call roughly two times in three. Walk back to a char boundary before slicing. The first tests I wrote for this passed against the broken version: repeating a fixed line, and repeating a bare 3-byte character, both happen to land exactly on 60_000. Shifting the content by one and two bytes is what exposes it, so the test now covers all three alignments and was confirmed to fail without the fix.
bobleer
added a commit
that referenced
this pull request
Aug 11, 2026
…ap osascript Four follow-ups left open by #2224, each a case where a result told the agent something that was either far too large or quietly untrue. **`get_app_state` was unbounded.** `describe_screen` got a 60 KB cap; the explicit query did not. Measured unbounded output on real Electron apps was 220-390 KB from a single call — roughly 100k tokens spent on one look at one app. Capped at 120 KB (higher, because asking for an app's tree is an explicit request, but still a ceiling), reusing the same clip that lands on a char boundary and announces itself. Applied in `snap_state_json`, so `app_click` / `app_type_text` / `app_scroll` / `app_key_chord` / `app_wait_for` are covered too — they all carry the same post-action tree and shared the same risk. **`analyze_image` reported the resized dimensions as the file's.** Large screenshots get downscaled to fit the provider (repeated 0.75x passes, floor 64px), and `ProcessedImage` kept only the final size — the source dimensions were computed and dropped. So a caller mapping anything the vision model said back to the screen was off by an unknown factor, with nothing in the result hinting at it. `ProcessedImage` now carries `original_width` / `original_height` with `scale()` and `was_resized()`; `analyze_image` and `view_image` report both frames. `analyze_image` also states plainly what its numbers are: any position in the prose is a vision model's estimate in the resized frame, not a measurement and not a click target — use `locate` / `move_to_text` / `describe_screen`, which return real coordinates. Building a coordinate contract on estimated numbers would be worse than saying they are estimates. **`open_app` could block for two minutes.** `activate` sends an AppleEvent and waits for the app to answer; a hung app does not, and macOS's default AppleEvent timeout is 120s, held on a blocking thread with the agent unaware. `Command::output()` has no timeout, so osascript now runs under a polled 10s deadline and is killed past it. A timeout reports as a failed launch the agent can act on, not an opaque io error. **`interaction_state.displays` rode on every result.** On a single-screen machine it repeats what `active_display_id` already says. Sent only when more than one display is attached; `list_displays` and `describe_screen` still report the full list on demand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
排查
logs/cua-202608110456.txt——一次「给飞书联系人发一条消息」的完整调用。任务本身约 8 步,实际跑了 73 次工具往返,其中 10 次是模型自己用screencapture+ 图像分析凑合出来的「眼睛」。顺着日志回溯,绝大部分开销来自四个互相放大的缺陷。
1. macOS 上
describe_screen一直是瞎的macos_foreground_application拼的 AppleScript 把try … end try语句块放在了表达式位置,AppleScript 编译期直接拒绝(-2741):命令永远非零退出 → 函数永远返回
None。而describe_screen的目标 app 就是从这个值推出来的,于是每次都返回foreground_application: null、ax_tree_text: null。模型的反应完全合理——它认为自己的工具输出被截断了:
从此不再信任工具结果,改用
screencapture+ 图像分析当眼睛:每看一眼多一次模型往返,返回的是描述而不是可点击坐标,而且多次看错(把前台应用认成别的 App、把输入框占位符读成无关文字、给出越界的 bbox)。改成进程内
NSWorkspace.frontmostApplication。顺带把macos_ax_ui::frontmost_pid也换掉——它每次调用 spawn 一个osascript,而describe_screen一次要走三遍,每次 ~120ms,还可能卡在 System Events 的 AppleEvent 超时上。2. text-only 模式下 Enter 被永久拒绝
stale-capture 守卫只有截图成功才会清除,但 text-only 的
screenshot在截图之前就短路返回了。于是守卫锁死:它自己的报错说「先调screenshot」,调了没有任何变化,之后所有click/ Enterkey_chord全部被拒。日志里模型最后是这样脱困的:
绕开工具直接发按键——守卫想拦的所有东西一并跳过了。
现在
describe_screen(text-only 下就等价于「看一眼」)和 text-only 的screenshot桩都会主动 waive 这个守卫。paste也改成无条件清除,而不是只在submit: true时——「先 paste 再单独按 Enter」是常见写法,而 paste 根本没动过指针。3. 结果里大部分是重复内容
每个
app_state同时带tree_text和app_state_nodes,后者是同一批节点再用啰嗦 JSON 序列化一遍。render_tree_text已经输出了全部可寻址字段(idx / role / title / value / id / desc / help / url / frame / 各种 flag,父子关系用缩进表达),而且没有任何消费方。实测:一次
get_app_state结果 107 KB,其中 82 KB(78%)是这份重复。已删除,保留node_count。4.
get_app_state返回的几乎全是折叠菜单观察一个无窗口的 App 得到 188 个节点,其中 180 个是
frame=(0,982,0x0)的菜单项——菜单没展开前根本点不到。现在不再下钻未打开的AXMenu,容器节点保留,并附一行说明指向get_app_shortcuts。附带:
open_app对「零窗口」无感日志里飞书
success: true+ 有效 pid,但 0 个窗口,模型花了约 15 次调用才自己摸索出要用open -b com.electron.lark——因为activate不会让已在运行的 Electron 应用重开窗口。现在
open_app会解析 bundle id(启动名Lark、可执行名Feishu、bundle idcom.electron.lark经常是三个不同字符串)、轮询等窗口、零窗口时用open -b兜底,并返回window_count/windowless/next_step。测试
osacompile做编译检查——不执行任何东西,但正好能拦住上面第 1 类 bug。附一个「守卫的守卫」用例,确保这个检查真的会拒绝坏语法。screenshot/describe_screen必须 waive 守卫;describe_screen空树必须给出ax_tree_status/ax_tree_note而不是裸 null;折叠菜单剪枝的判定;AppleScript 参数转义(App 名来自模型,不能让引号逃逸出字面量)。顺手修了
embedded_relay_host的测试:它预留一个临时端口、释放监听、然后假设端口还空着。机器空闲时看不出问题,一旦测试套件开始 spawn 子进程就有 ~80% 失败率。现在端口获取和「已释放」断言都会重试。验证
macos_foreground_application的实机冒烟用例(--ignored)在本机通过——修复前它必然返回None。