feat(macos): complete background operation: sky_key, covered/other-Space windows, agent display

[hyprcat]

Hi! I'm Hyprcat, one of the maintainers of mac-cua, an MCP server for macOS desktop automation built on the same idea as this project: drive apps through accessibility and screenshots without stealing the user's keyboard, focus or pointer. The work here is genuinely exciting and exactly the direction I'm working towards, so I'd like to contribute as much as I can. This PR makes background operation complete: sky_click already handled background clicks; this adds background keyboard, keeps covered and other-Space windows readable, and gives the agent a way to work on windows the app has already hidden.

Everything follows the sky_click conventions: explicit opt-in, never part of auto, no silent fallback, fail closed when a private symbol is missing, and nothing is ever sent to the real foreground app. Windows and Linux expose the new enums and return a stable unsupported error before snapshot lookup.

What's in it

key_method=sky_key on type_text / press_key (macOS). Chromium only inserts text while its NSWindow is the AppKit key window, so sky_key reuses sky_click's target-only synthetic-active record, posts yabai's make-key-window record pair (which makes the target set the window key internally, with no front-process switch, raise or Space change), delivers the existing postToPid key events, and releases the state. Command chords (cmd+a, cmd+c, cmd+v, ...) are pressed through the target's own menu bar via AX, because AppKit only dispatches NSMenu key equivalents for the active app.

Covered and other-Space windows in get_app_state. Apps learn that a window is covered only through WindowServer occlusion notifications; Chromium/Electron/WebKit react by hiding the page, which drops the web accessibility subtree. get_app_state now resolves off-screen windows instead of falling into the activate-and-raise recovery, pins the visibility of each window it snapshots while unoccluded (SLSPackagesEnableWindowOcclusionNotifications, restored to its prior state on normal server/app shutdown or process exit), re-walks Chromium-family trees while their lazy web tree builds, and says so in the tree when a window was already hidden. sky_click accepts covered and other-Space windows too.

window_placement=agent_display on get_app_state (macOS). For a window that is already hidden when the agent first sees it, there is no notification trick; macOS's own answer is a virtual display. agent_display creates an invisible 1920x1080 CGVirtualDisplay owned by the runtime and moves the window's frame onto it with the AX position attribute; the app renders it, exposes its full tree and accepts sky_click / sky_key. restore puts it back; normal server/app shutdown and process exit both attempt restoration; failed live-window restores remain retryable. The private classes live behind a small ObjC shim resolved with NSClassFromString.

Faster snapshots and input. Screenshots use SLSHWCaptureWindowList first (15 to 45 ms, retina, and it captures windows on inactive fullscreen Spaces where ScreenCaptureKit returns -3811), with SCK as fallback. The AX root window is bound to its CGWindowID via _AXUIElementGetWindow so tree, capture and element frames refer to one window. Every fixed gap in the input recipes is an environment-tunable knob. The macOS 27 benchmark reached 82 ms sky_click and 30 ms sky_key returns, while the final defaults retain small compatibility margins validated on macOS 26.6.2. OPEN_COMPUTER_USE_DEBUG_TIMING=1 prints per-phase timings.

Why it's designed this way

Visibility is a WindowServer concept the app only receives notifications about; input is WindowServer event delivery to a process; reading is AX plus capture by window id. Solving at that level keeps it app-agnostic. The reference notes record what was verified on macOS 27 and, just as importantly, what does not work: Cua's SLSEventAuthenticationMessage envelope alone, window-field stamping, SLPSSetFrontProcessWithOptions (makes the user's app resign active), CGSSetConnectionProperty "SetFrontmost", AX window focus, and moving other apps' windows between Spaces from a third-party process (gated on macOS 26+). Guessed private signatures are avoided; the ones used were read from disassembly and are documented.

Compatibility follow-up

A macOS 26.6.2 rerun found occasional dropped keys with zero sky_key settle/release (18/20, then 49/50). The branch now defaults sky_key settle/release to 10/10 ms (50/50), and preserves the established auto-input defaults of 20 ms between type_text chunks and 100 ms after press_key. Occlusion cleanup restores each window’s prior notification state, and agent-display cleanup keeps failed live-window restores retryable. Normal MCP/app shutdown runs cleanup; uncatchable forced termination remains a documented boundary.

Verification

• swift test: 185 tests, 0 failures; 7 opt-in live regressions skipped by default.
• Live, on macOS 27: covered isolated Chrome receives typing, chords and cmd+a with the foreground fixture keeping active/key/first-responder state and zero resign counts (SkyKeyboardLiveTests); SkyClickLiveTests still green; a visible snapshot pins a window so covering it keeps the web tree and screenshot (OcclusionKeepAliveLiveTests); a Chrome window on a real second Desktop is snapshotted, clicked and typed into from the first Desktop with the active Space unchanged (CrossSpaceLiveTests); a Chrome window hidden before first contact is parked, driven, restored, and the display removed (AgentDisplayLiveTests).
• BackgroundInputBenchmarkLiveTests: 50/50 clicks and 50/50 keys on covered Chrome, exact-once, frontmost unchanged.
• AppMatrixLiveTests: every running GUI app with a window (native, Electron, WebKit, GL) snapshotted read-only without activation; background typing verified in System Settings and Slack, on the desktop and parked on the agent display, with windows restored to their original positions.
• go vet / go test for both runtimes, the smoke suite, make check-docs.

Benchmarks

Apple M5 / macOS 27.0 (26A5425a) / Chrome 152, target = isolated Chrome --app window fully covered by the fixture app, machine idle. Every number comes from a test in this PR; the full report with methodology and reproduce commands is docs/references/background-input-benchmarks.md.

Background click and key latency, 50 cycles, benchmarked macOS 27 timing configuration ("returns" = dispatcher returned; "observed" = page reflected it, including Chromium's own processing)

Action	Success	Returns p50	Returns p95	Observed p50	Observed p95	Before this PR (returns p50)
sky_click, exact-once	50/50	82 ms	87 ms	260 ms	271 ms	323 ms
sky_key type_text	50/50	30 ms	35 ms	237 ms	244 ms	526 ms

Frontmost app unchanged in every cycle.

How the defaults were chosen (30 cycles per row; FOCUS = gap after a synthetic focus record, SCALE = multiplier on the sky_click recipe gaps)

FOCUS	SCALE	sky_click	sky_key
0 ms	0	0/30	5/30
20 ms	0	0/30	7/30
20 ms	0.05	30/30	30/30
0 ms	0.1	30/30	30/30
0 ms	1	30/30	30/30
40 ms	1 (previous defaults)	30/30	30/30

The macOS 27 sweep showed that same-queue events can run at 0 ms on that system, but macOS 26.6.2 did not make that a safe cross-version default. Final defaults are type_text chunk 20 ms, press_key tail 100 ms, sky_key key-window settle/release 10/10 ms, focus-record gap 10 ms, and sky_click recipe scale 0.2. All six gaps remain environment-tunable.

Snapshot phases (real apps, OPEN_COMPUTER_USE_DEBUG_TIMING=1)

Phase	Measured
Window capture, SLSHWCaptureWindowList (primary)	15 to 33 ms, retina; works on inactive fullscreen Spaces
Window capture, ScreenCaptureKit (fallback)	87 to 155 ms; -3811 on inactive fullscreen Spaces
AX tree walk	26 to 81 ms for 300 to 600 nodes

Agent display

Step	Measured
Create virtual display	333 to 353 ms
Ready (bounds + Space registered)	407 to 572 ms
Park a window (display already up)	144 to 230 ms
Restore a window	260 ms (live test), up to 1.1 s in the survey
Display removed after last restore	every run

Multi-app survey (27 running GUI apps, read-only snapshot each; input only into a currently empty text field, marker removed afterwards)

App	Engine	Desktop: tree / screenshot / input	Parked on agent display: tree / screenshot / input / restored
System Settings	native	98 nodes / yes / typed + cleared	101 / yes / typed + cleared 108 ms / yes
Slack	Electron	261 / yes / typed + cleared	261 / yes / typed + cleared 132 ms / yes
Google Chrome	Chromium	610 / yes / no empty field	610 / yes / no empty field / yes
Safari	WebKit	394 / yes / popover field outside window, rejected by design	394 / yes / no empty field / yes
VS Code	Electron	310 / yes (fullscreen Space, HW capture) / no empty field	skipped: fullscreen window
Blender	GL	5 / yes (fullscreen Space, HW capture) / no empty field	skipped: fullscreen window
Zed, TextEdit, KiCad, Alma	native	tree + screenshot / no empty field	tree + screenshot / no empty field / yes
Notes, Mail, Excel, Activity Monitor, Terminal, Helium, Figma, Canva, Linear, Devin	various	cgWindowNotFound: running without a window; read-only policy does not activate them

Frontmost app and pointer unchanged for both survey runs. Physical keystrokes sent while a target held synthetic key state went to the real frontmost app, not to the target.

Known boundaries

• Hidden apps (cmd+h) fail closed for sky_key / sky_click; nothing unhides on the user's behalf.
• agent_display is the one mode that changes user-visible state: the display sits to the right in the arrangement (a pointer can slide into it) and the parked window is absent from the desktop until restored. It is opt-in only.
• Fullscreen-Space windows can be read and captured but not parked.

Docs: docs/references/macos-skylight-background-keyboard.md, docs/references/macos-window-visibility-and-spaces.md, plus architecture, security, reliability, skill usage, release notes, history entries and archived execution plans.