feat(ios): control physical iOS devices over CoreDevice (experimental)#369
Draft
latekvo wants to merge 25 commits into
Draft
feat(ios): control physical iOS devices over CoreDevice (experimental)#369latekvo wants to merge 25 commits into
latekvo wants to merge 25 commits into
Conversation
latekvo
force-pushed
the
feat/physical-ios-device-support
branch
4 times, most recently
from
a1d97e7 to
5a91c67
Compare
Drive a real iPhone (iOS 27+) app-free via Apple's CoreDevice remote-control services through pymobiledevice3 — no on-device app, no WebDriverAgent. Gated behind the `physical-ios-devices` flag. - Discovery: classify physical UDIDs (8hex-16hex) as ios/device; list connected iPhones via `xcrun devicectl`, flag-gated. - CoreDevice backend (blueprints/core-device.ts): resolves the tunneld RSD endpoint, ensures the DDI is mounted, shells to pymobiledevice3 for screenshot, tap (dwell-drag — a zero-dwell tap is ignored by iOS), swipe, and buttons. - Branded privilege escalation: the tunnel needs root once to create its network interface. Instead of manual sudo, Argent auto-starts it via a signed macOS host helper (argent-device-auth) that shows the standard auth modal branded as "Argent" with the Argent icon (Authorization Services + dlsym of AuthorizationExecuteWithPrivileges). Falls back to the osascript admin prompt, then manual-sudo instructions (headless). HOME is pinned so tunneld finds its pairing records under the privileged-exec environment. - Wire screenshot/gesture-tap/gesture-swipe/button to the backend; launch-app via devicectl; boot-device starts the tunnel (branded prompt). Unsupported tools on physical iOS are cleanly rejected (simulator-server / ax-service / native-devtools guards; open-url/reinstall/restart handlers). - Helper resolvers + Argent icon in @argent/native-devtools-ios; download-native-binaries.sh fetches the signed helper (optional until published); bundle-tools copies it for distribution. Helper source + signing live in argent-private. - Tests for classification, devicectl parsing, coord mapping, and privileged command building. Overrides: ARGENT_PYMOBILEDEVICE3, ARGENT_PMD3_TUNNELD_PORT, ARGENT_DEVICE_AUTH_HELPER, ARGENT_DEVICE_ICON.
latekvo
force-pushed
the
feat/physical-ios-device-support
branch
from
5a91c67 to
6710eef
Compare
…y target list The preview / Argent Lens window streams frames over simulator-server, which refuses physical iOS devices (kind === "device", driven over CoreDevice). With the physical-ios-devices flag on, a connected iPhone leaked into GET /preview/simulators because the iOS branch matched any platform === "ios" without checking kind — surfacing a target the UI can't drive (and emitting an entry with no runtime under a type that asserted runtime: string). - Exclude physical iOS (kind === "device") from the preview list, mirroring the existing Chromium exclusion. - Consume list-devices' own exported ListDevicesResult type instead of a hand-rolled copy that had already drifted from it (the soundness hole tsc couldn't catch because invokeTool<T> takes a caller-supplied generic). - Extract the mapping into a pure, exported devicesToPreviewEntries() and add regression tests asserting physical iOS + Chromium are excluded while the simulator and Android emulator are kept.
…app behavior list-devices description and the simulator-server guard message listed screenshot/gesture-tap/gesture-swipe/button but omitted launch-app, which is implemented for physical iOS via devicectl. The README limitations list omitted restart-app, which (like open-url/reinstall-app) returns a clear "not supported" error on physical iOS.
…ice-support # Conflicts: # packages/tool-server/src/preview.ts # packages/tool-server/src/tools/open-url/platforms/ios.ts
…lity matrix Review follow-ups on the physical-iOS CoreDevice feature: - discovery: parsePhysicalIosDevices surfaced every host iOS simulator as a phantom physical device (devicectl lists simulators with transportType "sameMachine"); gate on the physical ECID UDID shape, which also aligns discovery with classifyDevice's routing. Verified against real devicectl JSON and a connected iPhone (returns only the device, not the host's simulators). - run-sequence: was eagerly resolving simulator-server for a physical iPhone, failing the whole call before step 1; hold no eager service so per-step CoreDevice routing works. - describe: swallowed the simulator-only guards and returned an empty tree with a misleading "reboot the simulator" hint; reject with a clear error instead. - swipe: command timeout was a fixed 15s, killing long swipes mid-drag; scale the timeout with the requested duration and clamp degenerate durations (extracted swipeDragParams, unit-tested). - native-profiler: guard the session blueprint so a physical iPhone isn't driven via simulator-only simctl (which mislabeled it an unresponsive "simulator"). - capability matrix: tools whose iOS backend is simulator-only (keyboard, paste, rotate, gesture-pinch/rotate/custom, screenshot-diff, native-devtools-*, native-profiler-start) now declare apple.device:false, so physical iOS is rejected with a clean 400 at the capability gate rather than a 500 from a deep service guard. open-url/reinstall-app/restart-app throw UnsupportedOperationError. - launch-app: wrap the devicectl launch so "app not installed/signed" surfaces a clean message instead of a raw subprocess blob. - core-device: check the feature flag before the pymobiledevice3 setup probe; echo a custom tunneld port in the manual-fallback hint. - docs: flags description and README now match the supported tool set and the auto-start prompt reality. Verified end-to-end on a physical iPhone (iOS 27): discovery, tunnel auto-escalation, screenshot (1179x2556), and swipe.
…e entry The previous commit guarded the shared native-profiler-session blueprint against physical iOS, but that blueprint also backs the device-agnostic analysis tools (profiler-stack-query / profiler-combined-report / -analyze), which profiler-query-android-capability.test.ts intentionally keeps supported on Apple device. The guard made those analysis tools 500 on a physical-iOS device_id. Scope the fix to where the actual bug is: only native-profiler-start enumerates running processes via simulator-only `simctl` (mislabeling a real iPhone as an unresponsive "simulator"), and it already declares apple.device:false, so the capability gate rejects it cleanly (400) before execute() runs. Drop the redundant session-blueprint guard; the analysis tools keep their cross-device capability. Also formats physical-ios-followups.test.ts.
launch-app drives a physical iPhone via `xcrun devicectl` directly rather than the CoreDevice service, so its handler never hit the flag check that gates every other physical-iOS operation — with `physical-ios-devices` off, launch-app on a physical UDID still launched the app. Extract the gate into a shared assertPhysicalIosEnabled() (reused by the CoreDevice factory and tunnel start) and call it from the launch-app physical-iOS branch before shelling devicectl. Add coverage: the launch-app flag gate, assertPhysicalIosEnabled, and the gesture-swipe physical-iOS routing (coords + duration forwarding).
… as unsupported The README limitations section listed only the interaction/lifecycle tools that reject on physical iOS; align it with the actual behavior by also noting that describe (accessibility), the native-* inspection tools, native-profiler-*, and screenshot-diff are simulator-only.
…; cover pmd3 argv On iOS 18-26 Apple gates host-driven input (the CoreDevice "remote control" services behind tap/swipe/button) to iOS 27+, so the device is discovered and advertised as tap-capable, then fails mid-session with a raw `CoreDeviceError 9021`. Detect that specific failure in `conciseError` (matched against pymobiledevice3's stderr/stdout only, never the argv-echoing execFile message, so a HID coordinate that contains "9021" can't false-positive) and surface an actionable message that points at iOS 27 and notes screenshots still work. `screen-capture` is not gated, so the screenshot path is untouched. Also note the physical-iOS/CoreDevice routing in the gesture-tap and gesture-swipe descriptions, and add a focused test that drives the CoreDevice factory through a mocked execFile boundary to pin the exact pmd3 argv for tap, button, screenshot, and swipe (previously untested — every other physical-iOS test mocks the CoreDeviceApi wholesale) plus the 9021 translation.
…ice-support # Conflicts: # packages/tool-server/src/preview.ts
…quivalent button's services() resolved coreDeviceRef() unconditionally for any physical-iOS button press, before execute() validates whether the button is even supported. Since the registry resolves services() before execute(), pressing appSwitch/actionButton on a physical iPhone triggered CoreDevice's tunnel-setup side effects (possibly a macOS admin prompt) just to reject the button afterward. Skip resolving the service when the button has no CoreDevice HID mapping.
…essage accuracy core-device.ts threw plain Error at all 12 sites, and ax-service.ts / simulator-server.ts's own new physical-iOS rejection branches did too - unlike every sibling blueprint, which uses FailureError with a dedicated error_code. Every physical-iOS failure (auth declined, no tunnel, missing pymobiledevice3, iOS<27 gate, flag disabled) was landing in telemetry as generic REGISTRY_TOOL_FAILURE_UNCLASSIFIED, indistinguishable from any other crash. Add CORE_DEVICE_* / AX_PHYSICAL_DEVICE_UNSUPPORTED / SIMULATOR_SERVER_PHYSICAL_DEVICE_UNSUPPORTED failure codes and convert every throw site. Also, three smaller correctness fixes found in the same pass: - resolveTunnel's `tunnel-port == null` check let a port of 0 through as "present", inconsistent with the sibling tunnel-address check which treats any falsy value as missing. - The iOS<27 gate message claimed "tap/swipe/button" all require iOS 27+, but this is hardware-verified false: screenshot and hardware buttons work below iOS 27, only touch (tap/swipe) is actually gated. - conciseError's line-matching heuristic could pick a misleading rich-boxed Python source-context line (e.g. "if rsd is not None:") instead of the real exception (e.g. "OSError: No route to host") from a stale-tunnel traceback, and searched execFile's synthesized "Command failed: <argv>" message with equal priority to the real stderr/stdout output.
… too native-profiler-start already rejects physical iOS (apple.device:false) since live capture uses simctl process enumeration that mislabels a real iPhone as a simulator. stop/analyze were left at device:true, so - since a session can never exist for physical iOS - they always fell through to a generic "no active session" error instead of the same clean, actionable rejection. Also fixes the README's "native-profiler-*" claim, which the stop/analyze gap made inaccurate, and two other accuracy issues found in the same pass: the iOS<27 limitation overstated to include hardware buttons, and a note that launch-app (devicectl) doesn't need the CoreDevice tunnel other tools require.
# Conflicts: # packages/argent/scripts/bundle-tools.cjs # packages/native-devtools-ios/src/index.ts # packages/tool-server/src/blueprints/native-devtools.ts # packages/tool-server/src/blueprints/simulator-server.ts # packages/tool-server/src/tools/describe/platforms/ios/index.ts # packages/tool-server/src/tools/devices/boot-device.ts # packages/tool-server/src/tools/devices/list-devices.ts # packages/tool-server/src/tools/gesture-custom/index.ts # packages/tool-server/src/tools/gesture-pinch/index.ts # packages/tool-server/src/tools/gesture-rotate/index.ts # packages/tool-server/src/tools/keyboard/index.ts # packages/tool-server/src/tools/launch-app/index.ts # packages/tool-server/src/tools/launch-app/platforms/ios.ts # packages/tool-server/src/tools/native-devtools/native-describe-screen.ts # packages/tool-server/src/tools/native-devtools/native-devtools-status.ts # packages/tool-server/src/tools/native-devtools/native-find-views.ts # packages/tool-server/src/tools/native-devtools/native-full-hierarchy.ts # packages/tool-server/src/tools/native-devtools/native-network-logs.ts # packages/tool-server/src/tools/native-devtools/native-user-interactable-view-at-point.ts # packages/tool-server/src/tools/native-devtools/native-view-at-point.ts # packages/tool-server/src/tools/paste/index.ts # packages/tool-server/src/tools/restart-app/index.ts # packages/tool-server/src/tools/restart-app/platforms/ios.ts # packages/tool-server/src/tools/rotate/index.ts # packages/tool-server/src/tools/run-sequence/index.ts # packages/tool-server/src/tools/screenshot/index.ts # packages/tool-server/src/utils/device-info.ts # packages/tool-server/src/utils/ios-devices.ts # scripts/download-native-binaries.sh
# Conflicts: # packages/tool-server/src/preview.ts
The app-free CoreDevice accessibility path was investigated on a physical
iPhone (iOS 27). The on-device accessibility tree is served by CoreDevice's
axAuditDaemon, but Apple gates it: the DTX service
`com.apple.accessibility.axAuditDaemon.remoteserver.shim.remote` requires the
`com.apple.mobile.lockdown.remote.trusted` entitlement — over the developer
(untrusted) CoreDevice tunnel pymobiledevice3 forms, the daemon accepts the
socket but terminates it on the first request (every audit selector, and even
the standard DTX capability handshake). The RemoteXPC replacement
`…axAuditDaemon.remoteAXService` requires `AppleInternal`, unreachable by any
third party. DTX transport itself works over the same tunnel
(`developer dvt proclist` succeeds), so this is Apple's auth wall, not a
transport gap. Screenshot (`canViewDeviceDisplay`) remains the only app-free
screen capture.
Replace the imprecise describe rejection ("no on-device accessibility/describe
path") and the ax-service backstop's "not supported yet" with accurate,
evidence-based messages, and split describe out of the README "not yet" list.
Drive a physical iPhone through one long-lived pymobiledevice3 process per device instead of spawning the CLI per interaction. Each `pymobiledevice3` invocation cost ~0.8s (~0.5s just `import pymobiledevice3`); the sidecar connects the RSD tunnel and opens the HID/screenshot services once, so a warm tap/button/swipe is now ~150-320ms (5x) — verified live on an iPhone 15 (iOS 27) through the tool-server. The agent is a small Python program, base64-embedded in coredevice-agent.ts (survives esbuild, no build-pipeline wiring), materialized to a temp file and run with pmd3's own venv interpreter; it speaks newline-delimited JSON over stdio. core-device.ts now maps its responses (incl. the iOS-27 9021 gate) to FailureErrors and tears the process down on dispose. Also make `describe` work on a physical iPhone: it returns the SpringBoard home-screen layout (app icons + dock) via CoreDevice's springboardservices — the only app-free structured screen data on a real device. Icon frames are derived from the home-screen grid (row-major first-fit so multi-cell widgets don't shift the icons after them) and are approximate; the hint says to confirm with screenshot and that in-app content isn't reachable. Verified the describe→tap loop live (tapped Settings via describe's frame; it opened). In-app accessibility stays Apple-gated: the axAuditDaemon DTX service drops the connection on the first message over the developer tunnel, and its RemoteXPC replacement needs the AppleInternal entitlement. Replaces the per-call argv test with sidecar/adapter/agent-protocol coverage.
Member
Author
|
Describe doesn't work well yet. This is the primary blocker as of now |
Replaces the SpringBoard home-screen-only describe with the device's live on-screen accessibility tree, read app-free over CoreDevice via the iOS-26+ axAudit service. Works in ANY app and on the home screen (the same VoiceOver walk): labels, values, traits (mapped to roles) and reading order are exact. The unlock is the RSDCheckin handshake iOS 26 added to RSD DTX services. pmd3's DtxServiceProvider opens RSD services with a raw connect and skips RSDCheckin, so iOS 26/27 dropped the accessibility daemon on the first byte (which had read as an "Apple wall"); the sidecar monkeypatches the open to go through start_lockdown_service, which performs the checkin. Verified live on iPhone/iOS 27: describe Settings -> read the "Other…" button's frame -> gesture-tap it -> the Join-Wi-Fi form opened. Home screen lists every app + widget. Frames: Apple exposes no per-element geometry on hardware, so exact rects come only from the accessibility audit (a subset of elements, correlated to the tree by element id); the rest are interpolated from reading-order neighbours - enough to tap a list row, with a hint to confirm precise taps via screenshot. Pixel-exact in-app frames would need an on-device XCUITest runner (code-signing). - sidecar: RSDCheckin fix + an `axtree` op (iter_elements walk + run_audit rects + screen size), replacing `homescreen`. - new ios-coredevice-ax-adapter (caption->role/label parse, rect normalize, frame interpolation); removes the springboard adapter. Source: coredevice-ax.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Physical iOS device support (experimental)
Adds app-free control of a physical iPhone (iOS 27+) over Apple's CoreDevice
"remote control" services (the path Xcode's device window uses) via
pymobiledevice3— no app installed on thedevice, no WebDriverAgent, no code signing. Gated behind a new
physical-ios-devicesflag.Mirrors the existing physical-Android split: a real iPhone is a separate backend
(
blueprints/core-device.ts), selected bydevice.kind === "device", the same wayandroid_devicediffers from the emulator controller.Supported on physical iOS
list-devices(shows the iPhone askind: "device"),screenshot,gesture-tap,gesture-swipe,button,launch-app(viadevicectl), anddescribe(the SpringBoardhome-screen grid — see below). Every other tool returns a clear "not supported on physical iOS"
error rather than mis-routing.
Requirements (⚠️ needs sudo, once)
device returns
CoreDeviceError 9021and only screenshots work.pymobiledevice3installed; iPhone connected, unlocked, trusted, Developer Mode on.sudo pymobiledevice3 remote tunneld— this needs rootonce to create the tunnel interface; every Argent command then runs unprivileged.
argent enable physical-ios-devices.The sudo/tunnel requirement is surfaced in the runtime error, the README, the flag
description, and the
list-devicesdescription.Performance: persistent CoreDevice sidecar
Interactions no longer spawn the
pymobiledevice3CLI per call (~0.8s each, ~0.5s of which isjust
import pymobiledevice3). A single long-lived helper per device connects the RSD tunnel andopens the HID / screenshot services once, then serves newline-delimited JSON on stdio. Warm
tap/button/swipedrop from ~800ms to ~150-320ms (5x), measured live through the tool-serveron an iPhone 15 (iOS 27). The helper is a small Python program base64-embedded in
coredevice-agent.ts(survives esbuild, no build-pipeline wiring), materialized to a temp file andrun with pmd3's own venv interpreter;
core-device.tsmaps its responses (incl. the 9021 gate) toFailureErrors and disposes the process with the service.describe— the real on-screen accessibility tree (app-free, any app)describeon a physical iPhone returns the device's live accessibility tree — the frontmostapp's elements (or the home screen) — read app-free via the iOS-26+ axAudit service over CoreDevice.
Labels, values, traits (mapped to roles) and reading order are exact.
The unlock: iOS 26 re-plumbed the accessibility daemon onto RSD and it now needs the RSDCheckin
handshake before it accepts DTX framing. pmd3's
DtxServiceProvideropens RSD services with a rawconnect and skips RSDCheckin, so iOS 26/27 dropped it on the first byte — which had read as an
"Apple wall". The fix is to open the service through
start_lockdown_service(which performsRSDCheckin); the sidecar does that. (
installation_proxyworked over the same tunnel because italready used that path — the entitlement was never the problem.)
Verified live (iPhone 15 / iOS 27):
describeSettings → read the "Other…" button's frame →gesture-tapits centre → the Join-Wi-Fi form opened. On the home screen it lists every app + widget.Frames: Apple exposes no per-element geometry on hardware, so exact rects come only from the
accessibility audit (a subset of elements, correlated to the tree by element id); the rest are
interpolated from reading-order neighbours — enough to tap a list row, with a hint to confirm precise
taps via
screenshot. Pixel-exact in-app frames/taps would need an on-device XCUITest runner(code-signing), which is a possible follow-up.
Not supported yet (clean rejections)
keyboard/typing, pinch & rotate (multi-touch),
open-url,reinstall-app,restart-app,native-*(simulator-only mechanisms).Verification
tsc --build,typecheck:tests,typecheck:scripts,prettier --check .all clean;1207 tool-server + 38 configuration-core unit tests pass (new suite covers
classification,
devicectlparsing, coord mapping).list-devices(kind:device),screenshot,button(woke device),gesture-swipe(Notification Center),
gesture-tap(opened Messages).list-devices→ singlekind:"device"(no phantom sims); warmgesture-tap155ms,button187ms,gesture-swipe323ms (vs ~810ms per CLI call);describe→ home-screentree, and tapping "Settings" via its
describeframe opened Settings. The in-app AX wall wasreproduced against the real
axAuditDaemon(see thedescribesection).pymobiledevice3 → install hint; no tunnel → the
sudo … tunnelderror above.Notes
Interactions run through a persistent
pymobiledevice3sidecar (one process per device); anative re-implementation in the Rust simulator-server is still a possible follow-up. Env overrides:
ARGENT_PYMOBILEDEVICE3,ARGENT_PMD3_TUNNELD_PORT.