Skip to content

fix(computer-use): bound AX payloads, report true image dimensions, cap osascript - #2225

Merged
bobleer merged 1 commit into
mainfrom
bob/cua-followups
Aug 11, 2026
Merged

fix(computer-use): bound AX payloads, report true image dimensions, cap osascript#2225
bobleer merged 1 commit into
mainfrom
bob/cua-followups

Conversation

@bobleer

@bobleer bobleer commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

#2224 的四个遗留项。共同点:结果要么大得离谱,要么悄悄告诉了模型不实的数字。

1. get_app_state 完全没有上限

#2224describe_screen 加了 60 KB 截断,但显式查询没加。实测两台真实 Electron 应用:

app A: full pruned  2422 nodes, 390 KB
app B: full pruned  1518 nodes, 221 KB

单次调用 ~100k token,一眼就吃掉大半个上下文窗口。

现在上限 120 KB——比 describe_screen 高,因为这是模型显式要整棵树,但仍然是个天花板。复用同一套按字符边界截断 + 显式声明的逻辑。

关键是加在 snap_state_json 而不是 get_app_state 调用点:app_click / app_type_text / app_scroll / app_key_chord / app_wait_for 的结果都带同一棵事后树,本来就共享同一个风险。

2. analyze_image 把缩放后的尺寸当成原图尺寸报

optimize_image_with_size_limit 会把大图缩小(反复 0.75× 直到过关,下限 64px),但 ProcessedImage 只留最终尺寸——原图尺寸在 image_processing.rs:204 算完就丢了。

于是模型拿到的 width/height 根本不是它传进去那个文件的尺寸,而结果里没有任何东西提示这一点。日志里模型反复算错 bbox,这是原因之一(另一半是 Retina 2x)。

ProcessedImage 现在带 original_width / original_height,配 scale()was_resized()analyze_imageview_image 两个坐标系都报。

另外 analyze_image 现在直说它的数字是什么:analysis 里任何位置都是视觉模型在缩放后画面里的估计,不是测量值,也不是点击目标——要操作屏幕请用 locate / move_to_text / describe_screen,那些返回真实坐标。

关于"坐标契约":我本来打算给 bbox 定义一套坐标系映射。查下来结论是不该做——那些数字是视觉模型的散文输出,是猜的不是量的。在猜测值上盖一层精确的坐标契约,比老实说明它们是猜测更糟。所以这里做的是标明它是什么,而不是假装它可靠。

3. open_app 可能阻塞两分钟

activate 向目标应用发 AppleEvent 并等回复。应用卡死就不回复,而 macOS 默认 AppleEvent 超时是 120 秒——期间占着一个 blocking 线程,模型完全不知道出了什么事。

Command::output() 没有超时,所以改成轮询 try_wait 的 10 秒预算,超时直接 kill。超时会作为"启动失败"报给模型(附带可执行的下一步),而不是抛一个看不懂的 io error。

测试用 delay 30 配 700ms 预算验证真的会被杀掉,整个套件跑完 0.71s。

4. 单显示器上 displays 是纯重复

interaction_state 挂在每一个 ComputerUse 结果上(均值 624 字节),其中 displays 数组在单屏机器上说的话 active_display_id 已经说完了。改成只在多显示器时输出;list_displaysdescribe_screen 仍然随时可查。

顺带发现的深度数据

重跑深度剖面,换了台应用,结论一致(这也验证了 #2224 里 depth 20 的选择能泛化):

depth  8:  14 nodes,   6 actionable,    846 bytes
depth 20: 181 nodes, 173 actionable,  23210 bytes
depth 32: 762 nodes, 754 actionable, 110458 bytes

验证

cargo test -p bitfun-desktop --lib      311 passed, 0 failed
cargo test -p bitfun-core   --lib      1959 passed, 0 failed
cargo test -p bitfun-agent-runtime      all green
cargo check --workspace --all-targets   no errors
cargo clippy                            回到 main 基线(无新增告警)

新增测试都确认过"没有修复时会失败",不是摆设:

  • 3024×1964(正是日志里那个尺寸)过 anthropic 限制必然走缩放路径,断言原图尺寸仍在、且 scale() 真的能把原图宽映射回发送宽
  • delay 30 + 700ms 预算,断言 TimedOut 且 5 秒内返回
  • 两个上限都真的把 body 压到 cap 以内
  • 两个常量的大小关系改用 const _: () = assert!(...) 编译期检查,颠倒顺序直接构建失败

…ap osascript

Four follow-ups left open by #2224, each a case where a result told the
agent something that was either far too large or quietly untrue.

**`get_app_state` was unbounded.** `describe_screen` got a 60 KB cap; the
explicit query did not. Measured unbounded output on real Electron apps
was 220-390 KB from a single call — roughly 100k tokens spent on one look
at one app. Capped at 120 KB (higher, because asking for an app's tree is
an explicit request, but still a ceiling), reusing the same clip that
lands on a char boundary and announces itself. Applied in
`snap_state_json`, so `app_click` / `app_type_text` / `app_scroll` /
`app_key_chord` / `app_wait_for` are covered too — they all carry the
same post-action tree and shared the same risk.

**`analyze_image` reported the resized dimensions as the file's.** Large
screenshots get downscaled to fit the provider (repeated 0.75x passes,
floor 64px), and `ProcessedImage` kept only the final size — the source
dimensions were computed and dropped. So a caller mapping anything the
vision model said back to the screen was off by an unknown factor, with
nothing in the result hinting at it. `ProcessedImage` now carries
`original_width` / `original_height` with `scale()` and `was_resized()`;
`analyze_image` and `view_image` report both frames.

`analyze_image` also states plainly what its numbers are: any position in
the prose is a vision model's estimate in the resized frame, not a
measurement and not a click target — use `locate` / `move_to_text` /
`describe_screen`, which return real coordinates. Building a coordinate
contract on estimated numbers would be worse than saying they are
estimates.

**`open_app` could block for two minutes.** `activate` sends an AppleEvent
and waits for the app to answer; a hung app does not, and macOS's default
AppleEvent timeout is 120s, held on a blocking thread with the agent
unaware. `Command::output()` has no timeout, so osascript now runs under
a polled 10s deadline and is killed past it. A timeout reports as a
failed launch the agent can act on, not an opaque io error.

**`interaction_state.displays` rode on every result.** On a single-screen
machine it repeats what `active_display_id` already says. Sent only when
more than one display is attached; `list_displays` and `describe_screen`
still report the full list on demand.
@bobleer
bobleer merged commit 2aa7bff into main Aug 11, 2026
7 checks passed
@bobleer
bobleer deleted the bob/cua-followups branch August 11, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant