Skip to content

fix(computer-use): restore text-only observation and unblock the Enter path - #2224

Merged
bobleer merged 6 commits into
mainfrom
bob/bitfun-computer-use-perf-03f09b
Aug 11, 2026
Merged

fix(computer-use): restore text-only observation and unblock the Enter path#2224
bobleer merged 6 commits into
mainfrom
bob/bitfun-computer-use-perf-03f09b

Conversation

@bobleer

@bobleer bobleer commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

背景

排查 logs/cua-202608110456.txt——一次「给飞书联系人发一条消息」的完整调用。任务本身约 8 步,实际跑了 73 次工具往返,其中 10 次是模型自己用 screencapture + 图像分析凑合出来的「眼睛」。

顺着日志回溯,绝大部分开销来自四个互相放大的缺陷。

1. macOS 上 describe_screen 一直是瞎的

macos_foreground_application 拼的 AppleScript 把 try … end try语句块放在了表达式位置,AppleScript 编译期直接拒绝(-2741):

$ osascript -e 'tell application "System Events"
  set p to first process whose frontmost is true
  return (unix id of p as text) & "|" & (name of p) & "|" & (try (bundle identifier of p as text) on error "" end try)
end tell'
143:146: syntax error: Expected expression, ")", etc. but found "try". (-2741)

命令永远非零退出 → 函数永远返回 None。而 describe_screen 的目标 app 就是从这个值推出来的,于是每次都返回 foreground_application: nullax_tree_text: null

模型的反应完全合理——它认为自己的工具输出被截断了

describe_screen 的输出被截断了,没有显示 ax_tree_text 内容。

从此不再信任工具结果,改用 screencapture + 图像分析当眼睛:每看一眼多一次模型往返,返回的是描述而不是可点击坐标,而且多次看错(把前台应用认成别的 App、把输入框占位符读成无关文字、给出越界的 bbox)。

改成进程内 NSWorkspace.frontmostApplication。顺带把 macos_ax_ui::frontmost_pid 也换掉——它每次调用 spawn 一个 osascript,而 describe_screen 一次要走三遍,每次 ~120ms,还可能卡在 System Events 的 AppleEvent 超时上。

2. text-only 模式下 Enter 被永久拒绝

stale-capture 守卫只有截图成功才会清除,但 text-only 的 screenshot 在截图之前就短路返回了。于是守卫锁死:它自己的报错说「先调 screenshot」,调了没有任何变化,之后所有 click / Enter key_chord 全部被拒。

日志里模型最后是这样脱困的:

osascript -e 'tell application "System Events" to tell process "Feishu" to keystroke return'

绕开工具直接发按键——守卫想拦的所有东西一并跳过了。

现在 describe_screen(text-only 下就等价于「看一眼」)和 text-only 的 screenshot 桩都会主动 waive 这个守卫。paste 也改成无条件清除,而不是只在 submit: true 时——「先 paste 再单独按 Enter」是常见写法,而 paste 根本没动过指针。

3. 结果里大部分是重复内容

每个 app_state 同时带 tree_textapp_state_nodes,后者是同一批节点再用啰嗦 JSON 序列化一遍。render_tree_text 已经输出了全部可寻址字段(idx / role / title / value / id / desc / help / url / frame / 各种 flag,父子关系用缩进表达),而且没有任何消费方

实测:一次 get_app_state 结果 107 KB,其中 82 KB(78%)是这份重复。已删除,保留 node_count

4. get_app_state 返回的几乎全是折叠菜单

观察一个无窗口的 App 得到 188 个节点,其中 180 个是 frame=(0,982,0x0) 的菜单项——菜单没展开前根本点不到。现在不再下钻未打开的 AXMenu,容器节点保留,并附一行说明指向 get_app_shortcuts

附带:open_app 对「零窗口」无感

日志里飞书 success: true + 有效 pid,但 0 个窗口,模型花了约 15 次调用才自己摸索出要用 open -b com.electron.lark——因为 activate 不会让已在运行的 Electron 应用重开窗口。

现在 open_app 会解析 bundle id(启动名 Lark、可执行名 Feishu、bundle id com.electron.lark 经常是三个不同字符串)、轮询等窗口、零窗口时用 open -b 兜底,并返回 window_count / windowless / next_step

测试

  • AppleScript 模板现在用 osacompile 做编译检查——不执行任何东西,但正好能拦住上面第 1 类 bug。附一个「守卫的守卫」用例,确保这个检查真的会拒绝坏语法。
  • 新增:text-only screenshot / describe_screen 必须 waive 守卫;describe_screen 空树必须给出 ax_tree_status / ax_tree_note 而不是裸 null;折叠菜单剪枝的判定;AppleScript 参数转义(App 名来自模型,不能让引号逃逸出字面量)。

顺手修了 embedded_relay_host 的测试:它预留一个临时端口、释放监听、然后假设端口还空着。机器空闲时看不出问题,一旦测试套件开始 spawn 子进程就有 ~80% 失败率。现在端口获取和「已释放」断言都会重试。

验证

cargo test -p bitfun-desktop --lib     309 passed, 0 failed   (连跑 12 次全绿)
cargo test -p bitfun-core   --lib     1952 passed, 0 failed
cargo test -p bitfun-agent-runtime     all green
cargo check --workspace --all-targets  no errors
cargo clippy (computer_use)            189 warnings (main 基线 190)

macos_foreground_application 的实机冒烟用例(--ignored)在本机通过——修复前它必然返回 None

…r path

A single Feishu "send one message" run took 73 tool round-trips. Tracing
the log back, most of them came from four defects that compound.

**`describe_screen` was blind on macOS.** `macos_foreground_application`
built its AppleScript with a `try … end try` *block* in expression
position, which AppleScript rejects at compile time (-2741). The command
always exited non-zero, so the function always returned `None` — and
`describe_screen` derives its target app from that value. Every call
reported `foreground_application: null` and `ax_tree_text: null`. The
agent reasonably concluded its own tool output was being truncated,
stopped trusting results, and fell back to `screencapture` +
image-analysis as a substitute for eyes: 10 extra model round-trips that
returned prose instead of coordinates and misread the screen repeatedly.

Replaced with an in-process `NSWorkspace.frontmostApplication` read, and
did the same for `macos_ax_ui::frontmost_pid`, which shelled out to
`osascript` on every call — three spawns per `describe_screen`, each
~120ms, each able to block on a System Events AppleEvent timeout.

**Enter was permanently refused in text-only mode.** The stale-capture
guard is cleared only by a successful capture, but a text-only
`screenshot` short-circuits before capturing. So the guard latched: its
own error said "call `screenshot` first", calling it changed nothing, and
every `click`/Enter stayed blocked. The observed escape was the agent
bypassing the tool with raw `osascript … keystroke return`, which skips
every check the guard exists to enforce. `describe_screen` (the text-only
equivalent of looking) and the text-only `screenshot` stub now waive it.
`paste` also clears it unconditionally instead of only when `submit:true`
— paste-then-Enter is the common shape and the pointer never moved.

**Results were mostly duplicate.** Every `app_state` carried both
`tree_text` and `app_state_nodes`, the same nodes re-serialised as
verbose JSON. `render_tree_text` already emits every addressable field,
nothing consumed the array, and it was 82 KB of a 107 KB result. Dropped,
`node_count` kept.

**`get_app_state` returned mostly closed menus.** Observing a windowless
app produced 188 nodes, 180 of them menu items at zero-size off-screen
frames — unclickable until the menu opens. Closed `AXMenu` subtrees are
no longer walked; the container stays visible and a note points at
`get_app_shortcuts`.

Also: `open_app` reported `success: true` for an app running with no
window, which is how this run lost ~15 calls rediscovering that
`activate` does not reopen an Electron window. It now resolves the bundle
id (the launch name, executable name and bundle id are routinely three
different strings), polls for a window, retries via `open -b`, and
reports `window_count` / `windowless`.

Tests: AppleScript templates are now compile-checked with `osacompile`,
which catches exactly the class of bug above without executing anything.

Drive-by: `embedded_relay_host` tests reserved an ephemeral port, dropped
the listener, then assumed it was still free. Harmless on an idle machine
and ~80% failing once the suite spawns subprocesses. Port acquisition and
the release assertions now retry.
The macOS branch gained bundle_id / process_name / window_count /
launch_path; the other two construct the same struct and would not
compile without them.

Both leave the identity fields None rather than guessing: neither
`start` nor `xdg-open` reports what it launched, so there is no pid to
resolve identity or a window count from. `window_count: Some(0)` would
tell the model the app is definitely windowless when it was simply
never measured.
…names

Only "Safari" was covered, which never exercises applescript_quote.
Compiling the escaped forms of a quote, a backslash and CJK is what
proves the escaping matches AppleScript's actual string-literal syntax
rather than a plausible guess about it.
The two lines this prompt was missing are the ones that would have ended
the observed failure early. When describe_screen returned nulls the agent
concluded its own output was truncated and improvised screencapture plus
image analysis as a substitute for eyes.

State plainly that an empty ax_tree_text is a result with a reason
attached (ax_tree_status / ax_tree_note), that re-calling returns the
same thing, and that building eyes out of screencapture costs a model
round-trip per glance and returns prose instead of coordinates.
…tron apps

Fixing `foreground_application` gave `describe_screen` an app to look at.
It was still walking only 8 levels into the focused window, which is fine
for a native Cocoa app and far too shallow for the Electron / WebView
clients agents are most often asked to drive.

Measured against a real Electron window (focused window only):

    depth  8:    17 nodes,    7 actionable,   1 KB
    depth 12:    25 nodes,   15 actionable,   2 KB
    depth 16:    50 nodes,   40 actionable,   5 KB
    depth 20:   207 nodes,  197 actionable,  27 KB
    depth 24:   233 nodes,  223 actionable,  31 KB
    depth 32:  1289 nodes, 1279 actionable, 206 KB

Seven actionable elements is not enough to find a search field or a send
button, so the tree read as "this app has no AX tree" and pushed the
agent onto OCR and screenshot guessing. The actionable layer appears
around 20; past it the payload grows far faster than the number of things
worth clicking.

Depth is a poor proxy for size, though — a document or a long list can
multiply that 27 KB — so the returned tree is also capped at 60 KB, cut on
a line boundary. The clip announces itself and says a control that is
missing from the view may still exist: an agent that reads a truncated
tree as the whole UI concludes the control is not there and gives up.

The `#[ignore]`d dump test now prints this depth profile, so the constant
can be retuned against evidence rather than intuition.
…e index

The 60 KB cap I added in the previous commit sliced the tree at a byte
offset. `&str[..n]` panics when `n` falls inside a multi-byte character,
so a CJK app tree — the kind most likely to be large enough to hit the
cap in the first place — would panic the whole tool call roughly two
times in three.

Walk back to a char boundary before slicing.

The first tests I wrote for this passed against the broken version:
repeating a fixed line, and repeating a bare 3-byte character, both
happen to land exactly on 60_000. Shifting the content by one and two
bytes is what exposes it, so the test now covers all three alignments and
was confirmed to fail without the fix.
@bobleer
bobleer merged commit ffe2aab into main Aug 11, 2026
7 checks passed
@bobleer
bobleer deleted the bob/bitfun-computer-use-perf-03f09b branch August 11, 2026 14:55
bobleer added a commit that referenced this pull request Aug 11, 2026
…ap osascript

Four follow-ups left open by #2224, each a case where a result told the
agent something that was either far too large or quietly untrue.

**`get_app_state` was unbounded.** `describe_screen` got a 60 KB cap; the
explicit query did not. Measured unbounded output on real Electron apps
was 220-390 KB from a single call — roughly 100k tokens spent on one look
at one app. Capped at 120 KB (higher, because asking for an app's tree is
an explicit request, but still a ceiling), reusing the same clip that
lands on a char boundary and announces itself. Applied in
`snap_state_json`, so `app_click` / `app_type_text` / `app_scroll` /
`app_key_chord` / `app_wait_for` are covered too — they all carry the
same post-action tree and shared the same risk.

**`analyze_image` reported the resized dimensions as the file's.** Large
screenshots get downscaled to fit the provider (repeated 0.75x passes,
floor 64px), and `ProcessedImage` kept only the final size — the source
dimensions were computed and dropped. So a caller mapping anything the
vision model said back to the screen was off by an unknown factor, with
nothing in the result hinting at it. `ProcessedImage` now carries
`original_width` / `original_height` with `scale()` and `was_resized()`;
`analyze_image` and `view_image` report both frames.

`analyze_image` also states plainly what its numbers are: any position in
the prose is a vision model's estimate in the resized frame, not a
measurement and not a click target — use `locate` / `move_to_text` /
`describe_screen`, which return real coordinates. Building a coordinate
contract on estimated numbers would be worse than saying they are
estimates.

**`open_app` could block for two minutes.** `activate` sends an AppleEvent
and waits for the app to answer; a hung app does not, and macOS's default
AppleEvent timeout is 120s, held on a blocking thread with the agent
unaware. `Command::output()` has no timeout, so osascript now runs under
a polled 10s deadline and is killed past it. A timeout reports as a
failed launch the agent can act on, not an opaque io error.

**`interaction_state.displays` rode on every result.** On a single-screen
machine it repeats what `active_display_id` already says. Sent only when
more than one display is attached; `list_displays` and `describe_screen`
still report the full list on demand.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant