Repository navigation
Conversation
Integrate Xcode's native MCP interaction sessions and simctl with the shared device driver, standalone CLI, and embedded SDK.\n\nPreserve Android defaults, native accessibility hit points, and cancellation-safe device leases. Add read-only setup checks, usage documentation, and hermetic regression coverage.
Capture via xcrun simctl io recordVideo anchored to the first-frame marker, seal segments on demand for live analyzer clips, roll on rotation/duration/crash, and finalize VFR .mov segments to CFR MP4 with a version-2 manifest so iOS recordings flow through the same DataEngine, extraction, and replay pipeline as Android scrcpy. Video tools now auto-detect per platform (simctl+ffmpeg on iOS) and the CLI/SDK no longer disable recording for iOS tasks.
simctl io screenshot does not stream to stdout on Xcode 27, so the dimension probe always failed and segments finalized on the 1080x1920 fallback canvas. Parse the framebuffer IOSurface port size instead - it is the exact surface recordVideo encodes, excludes external scene displays, and swaps on rotation.
Extend the native Xcode 27 iOS support to every Artemis surface so iOS reaches feature parity with Android where the platform permits: - Shared simctl discovery (artemis/drivers/ios/discovery.py) and an IosDevicePool producing platform-tagged DeviceStatus entries, with explicit-UDID validation, Booted/Shutdown admission, and ambiguous "booted" rejection. - iOS locks scoped under "ios" (ios__<UDID>) so a simulator UDID can never collide with an Android serial; queue items carry platform, ios_workspace, and the pinned UDID, and workers spawn with --platform ios --ios-workspace without leaking ADB_DEVICE_SERIAL. - Web API accepts platform="ios", validates UDIDs, skips the Android screen-lock probe, rejects locked_app_package, and lists simulators alongside Android devices in /api/devices. - Live stream follows the active iOS lock owner and captures frames via simctl io screenshot; /api/stream/device-state reports platform. - Replay preserves mobile_platform from session device_info, rebuilds iOS contexts with DevicePlatform.IOS, honors UDID overrides, and lists simulators as replay targets. - MCP mobile_run_task accepts a platform argument (daemon + standalone paths), mobile_get_device_state observes iOS through UnifiedMobileController + XcodeSimulatorDriver, and mobile_diagnose adjusts verdicts/probes for iOS targets. - An ios_simulators readiness probe reports Xcode/simulator/MCP availability without becoming a blocker for Android-only hosts. - artemis run and artemis batch forward platform, device_serial, and ios_workspace through the daemon. Android remains the default platform and its behavior is unchanged. Android-only operations (shell, logcat, UIAutomator, AVD launch, accessibility helper) fail explicitly for iOS.
The run command still carried a pre-queue guard that rejected iOS workers and forced every iOS invocation into standalone mode, so daemon-submitted iOS tasks died at worker startup with "The daemon and device queue support Android only". Drop the stale checks; iOS now daemon-routes like Android (--standalone still forces local), and the worker path reaches execute_task with platform/ios_workspace intact. Verified live: /api/run platform=ios now spawns the iOS worker and reaches agent init, stopping only at the missing GOOGLE_API_KEY.
The console frontend was Android-only even though the backend already listed iOS simulators and accepted platform=ios runs: - connectedDevices merges ios_simulators probe metadata (udid/name/ state/runtime) into the device list, tagged platform=ios - Device chips render phone_iphone + an iOS badge; the hero card shows the selected simulator with its runtime - /api/system/devices/select accepts platform; iOS validates the UDID via the iOS device pool instead of retargeting the ADB probe - iOS selection is tracked client-side and satisfies the run button device gate; runTask sends platform=ios + device_serial when a simulator is selected - Sessions/queue items carry platform for an iOS tag in the session list Android selection/readiness/submission paths are unchanged.
The connected-device panel only rendered when the ADB probe passed, so on hosts without an Android device the iOS simulators could never be selected — and selecting one was what made the panel appear: - Show an "iOS Simulators" chip panel inside the no-device guide state whenever the ios_simulators probe reports simulators, independent of ADB state - Selecting a sim flips the device step to ready and shows the hero card - Retitle step 3 "Device & Emulator Connection" (Android via ADB or iOS Simulator via Xcode 27+)
Verified the live console with headless Chrome: the diagnostics tab now
lists all simulators and clicking one selects it via
POST /api/system/devices/select {serial, platform:"ios"}. But the
launcher composer gave no hint which device a task would run on:
- Add a target-device-chip to the composer action bar showing the
selected device with its platform tag; clicking it jumps to the
setup guide to change the target
- Make the prerequisites warning platform-neutral ("device" not
"Android device")
The device guide card mixed Android-only controls (Restart ADB, connection methods, AVD management) with the iOS simulator picker, and the merged chip list made the run target ambiguous. - Add platform tabs at the top of the card; auto-select the tab that has devices (or the platform of the currently selected simulator) - Scope Restart ADB, Change Connection, Android hero/switcher, connection methods, AVD list, and locked/unauthorized states to the Android tab - Give iOS its own panel: selected-simulator hero, simulator chip picker, and an empty state explaining the Xcode 27+ requirement - Selecting a chip also switches the active tab so the chosen target stays visible; ready banner text is platform-neutral
Each click on a simulator POSTed to /devices/select, which re-ran "xcrun simctl list devices" inline to validate the UDID — 4+ seconds on a busy host while the readiness probe spawned back-to-back simctl calls of its own. The chip only marked itself selected after the response, so clicks appeared to do nothing. - Cache successful simctl enumerations for 10s in drivers/ios/ discovery.py (shared by the readiness probe, device pool, and HTTP endpoints); failures are never cached, force_refresh bypasses, and the driver clears the cache after booting a simulator - Apply iOS selection optimistically in the console since it is client-side state; revert only when the server rejects the pick Select endpoint: ~4.3s -> ~0.1s. Chip shows Active instantly.
TS6 requires rootDir to be explicit whenever outDir is set; the compiler was inferring ./src anyway, so this only silences the diagnostic without changing emit layout.
- Runtime now renders as "iOS 27.0" instead of "iOS 27 0" - Hero card reports the real simulator state: green "Booted" for running sims, amber "Shutdown — boots on run" otherwise (previously always claimed "Connected"), with a matching status dot instead of the hardcoded green pulse - Drop the redundant per-chip "iOS" badge (the tab and panel already scope the list) and the cramped "(Shutdown)" suffix; state is now a colored dot — green Booted, grey Shutdown, amber Booting — with a tooltip for the raw simctl state - iPad simulators use the tablet_mac icon in chips and the hero card
The feature section and roadmap entry predated the web/daemon/MCP integration commits and still described iOS as standalone-only; only physical devices remain planned.
Extend the iOS driver family to paired physical devices. Device routing keys off the requested UDID: serials resolving to a CoreDevice physical entry get PhysicalIosDriver, while simulators and unknown serials keep the existing XcodeSimulatorDriver path. - discovery.py: devicectl list devices JSON parsing (current and deprecated property shapes) with the same fail-closed caching contract as simctl - PhysicalIosDriver subclasses the simulator driver to inherit the Xcode DeviceInteraction session (screenshots, hierarchy, taps, swipes, text), overriding only lifecycle: devicectl install/launch/terminate/openURL/apps and pairing/connectivity checks with actionable errors - connect() is decomposed into seams (_require_ios_host, _prepare_device, _start_interaction_session, _validate_session_device) so the physical driver asserts deviceIsSimulator=false without duplicating session setup - PhysicalIosRecorder polls devicectl screenshots into timestamped frames and assembles MP4 segments with the ffconcat demuxer; rotation and seal boundaries roll segments, matching the simulator manifest contract - IosDevicePool enumerates physical devices for the console/queue, validates explicit serials (paired + connected), and never auto-selects hardware - CLI help, ios.md, and both READMEs updated for physical requirements and recording limitations Physical automation requires Xcode 27+, a paired/trusted device with Developer Mode enabled, and a device-signed .app/.ipa for installs.
- Agent.init picks the driver via ios_driver_class so embedded-SDK iOS configs resolve physical UDIDs to PhysicalIosDriver - device stream service captures physical frames via devicectl screenshot (simctl io screenshot does not exist on hardware) - replay device picker lists paired physical devices for iOS retargeting - smoke-test repair hints cover pairing/trust/Developer Mode and offline physical devices
Xcode's DeviceInteraction* MCP tools accept simulators only — verified live against a paired iPhone (the eligible-device list contains simulators exclusively), so physical observation and input move to WebDriverAgent: - wda.py: stdlib HTTP client (no new dependency) covering /status, /session, /source?format=json, /screenshot, /window/size, W3C pointer actions for tap/long-press/swipe, /wda/keys for text, /wda/homescreen and /wda/pressButton for keys, /wda/activeAppInfo for the foreground bundle. - Endpoint resolution probes ARTEMIS_IOS_WDA_URL, ARTEMIS_IOS_WDA_HOST, the CoreDevice tunnel address from devicectl info details, and 127.0.0.1:8100 for iproxy/pymobiledevice3 forwards. - connect() locates an installed *WebDriverAgent* runner, launches it via devicectl, and optionally hosts a build-for-testing .xctestrun through xcodebuild test-without-building (ARTEMIS_IOS_WDA_XCTESTRUN); disconnect tears both down. - Hierarchy flattens the WDA JSON tree into the same ui_elements shape (text/resource_id/class/parsed_bounds/hit_point) so controllers, element lookup, and tap_element are unchanged. - devicectl JSON for info subcommands now goes through a scratch file — '--json-output -' stdout is polluted by the human-readable table on hardware — and terminate resolves PIDs by matching the app's install URL prefix against running executables. - docs/ios.md and READMEs describe the WDA prerequisite, signing options, endpoint overrides, and the UI-Automation passcode gate. 47 physical unit tests pass; the simulator driver and Android paths are untouched.
…ing hardening - ios_device_pool: never auto-select physical hardware; validators match by UDID or name and fail open when either enumeration is indeterminate - sdk/agent: pick the iOS driver class in a worker thread so simctl/ devicectl subprocesses never block the event loop during init - physical_driver: tunnel-IP reads every observed devicectl JSON shape; exclusive runner/xctestrun hosting; broad connect cleanup; --kill terminate with stale-PID fallback; normalize file:// executables; IPA bundle id read from Payload Info.plist; scale sanity guard - wda: map http.client.HTTPException, propagate transport errors from press_button, IPv6-safe URL normalization, bracket only v6 candidates - physical_recording: monotonic frame indices across segment rolls, shielded stop, broad poll-loop failure accounting, watchdog notices a dead poller, ffmpeg timeout, atomic .part cleanup - discovery: null-safe property reads, iPadOS classification - diagnose/probe: physical serials reachable via mobile_diagnose; pool auto-pick stays simulator-only; platform-aware smoke labels - Sweep stale 'simulator' wording across MCP tools, routers, schemas, daemon client, setup script, and docs; document XCTESTRUN + UI Automation consent; tests: generic fixtures, WDA env pinning, deterministic recorder seams
…cels - Mock _require_ios_host on the driver fixture so _resolve_device tests never spawn real xcodebuild or fail on non-macOS CI - Synthetic UDID/name/URL fixtures (no real-looking hardware identifiers) - Await the cancelled poll task on recording start failure - Base the session poll/watchdog fields as instance defaults
- list_apps: simctl emits OpenStep plists that plistlib cannot read; route through plutil -convert json (plistlib fast-path kept) - connect(): reconnect when the MCP bridge retired but a stale session key remained; null-safe deviceUUID; widen connect-cleanup catch - device_lock: annotate_active_owner preserves lock_scope and resolves scoped lock files; cleanup_stale_locks glob matches scoped names - readiness: SKIPPED probes no longer force a degraded verdict on non-macOS hosts - recording: kill+reap display-probe on timeout/cancel; bound crash-loop respawns (spawn success no longer resets the counter; only a segment surviving a healthy interval does); ffmpeg/probe timeouts; bound concurrent conversions; guard stale seal through_time - bridge: scoped child env (no API keys into mcpbridge), start() lock, close() never masks caller errors - adb_server: iOS controller no longer aliases into the Android global - cli: batch --ios-workspace path validation; run no longer misclassifies iOS prerequisite errors as missing API keys - stream: reap cancelled simctl child; drop stale frames on target switch; annotate active_tasks with platform; queue items surface platform in the UI mapper - env scrub pops ARTEMIS_DEVICE_ID for iOS workers (daemon + MCP) - device_smoke: platform-aware error wording for iOS - docs: fix broken SDK builder example; mobile_run_task documents platform/ios_workspace; platform_guidance no longer claims video analysis is unavailable - diagnose validates platform values; visualization resolves 0-1000 coords on small screenshots - pyright-core covers the new iOS runtime/actuator modules
- Operator/schema key enum gains POWER, VOLUME_UP, VOLUME_DOWN — the iOS driver and platform guidance already support them; both platforms map them natively (regenerated action-surface fixtures) - action_names translation now keeps power/volume keys bare so iOS drivers receive supported names - Diagnoser drops the logcat-only analyze_logs tool on iOS - mobile_run_task app_path doc covers iOS .app directories - Regression test pins the iOS key set to the operator gate
Combines feat/ios-physical (devicectl discovery, PhysicalIosDriver with WDA backend, physical recording) with the simulator branch so a single PR delivers full iOS support: simulators + paired physical devices. Conflict resolution keeps both sides' fixes: plutil listapps parsing, dead-bridge reconnect, scoped lock metadata, SKIPPED-neutral readiness, MCP env scrubbing, and physical driver/recording internals.
…elpers - Shared physical_ios_ready() predicate across pool, probe, and driver paths (paired + connected-or-absent tunnelState) - Sync and async validate_explicit_serial now share matching/state rules (UDID or device name) - normalize_device_platform/device_pool_for/target_for_platform in adb_endpoint; adopted at CLI, admin, MCP surfaces - BOOTED_SIMULATOR_ID and DEFAULT_MAX_DURATION_SECONDS replace magic literals across iOS code - Shared helpers: reap_process, devicectl_screenshot, pixel_element, device_matches_identifier - MCP worker env now reuses IosTarget.apply_to_environment; IOS_LOCK_SCOPE replaces 'ios' literals - WDA-aware smoke hints and hierarchy_backend='wda' for physical devices - Removed dead code: duplicate replay /api/devices route, ReplayManager.list_devices/_init_device, fetchDevices JS, no-op driver overrides, unused imports - for_ios_device() is the canonical SDK method; for_ios_simulator kept as alias; IosDeviceProbe with stable probe id and class alias - Docstring/wording sweep: 'iOS device' instead of simulator-only phrasing where hardware is supported
Lease and close one-shot iOS observations; drain cancelled native and WDA requests; reap incomplete simulator recording starts and bound recovery loops. Preserve modern CoreDevice schema, unambiguous targets, per-platform queue identity, and physical-device selection without changing Android driver code. Add remote SDK iOS platform/workspace forwarding with capabilities preflight to reject legacy hosts before submission. Keep Android payloads unchanged. Add regressions and resolve protected-core typing diagnostics without changing quality thresholds. Verification: 2900 deterministic tests passed, 11 skipped; protected-core pyright reports 0 errors; Ruff and quality ratchets pass. Live simulator acceptance is blocked by Xcode agent consent; physical/Android hardware and the compatible frontend Node runtime are unavailable. No production-readiness claim.
Live acceptance on iPhone 15 Pro surfaced two defects that blocked every
normally-named physical device:
- WDA device_info reports the product family name ("iPhone"), not the
personalized devicectl name ("Jane's iPhone"). The identity guard now
rejects only a conflicting *specific* name instead of any inequality.
- A driver-launched WDA runner auto-creates a session on startup; the
foreign-session refusal then deadlocked connect(). open_session gains
adopt_existing, enabled only when the driver owns the runner process;
discovered/user-provisioned endpoints keep the refusal.
Verified live: WDA session, 319-element hierarchy, tap/swipe/home, WDA and
devicectl screenshots, pid-verified terminate, MP4 recording, clean
disconnect with zero orphaned processes.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
8 tasks done
Three compatibility bugs surfaced while driving a real iOS task through the admin console against a strict OpenAI-compatible local endpoint: - strip_json_comments stripped "//" inside string literals, corrupting URL values such as "api_base": "http://127.0.0.1:8080/v1". Replaced the regexes with a string/escape-aware scanner that removes comments only outside string literals and preserves newlines for stable error positions. - _resolve_endpoint dropped api_base/api_key/max_tokens/timeout because the LLM schema had no fields for them, so custom providers silently fell back to the environment OpenAI base URL. Added the fields and pass them through to ModelEndpoint (fallback configs resolve independently). - Screenshot bytes are PNG on both platforms yet every OpenAI-style payload labeled them data:image/jpeg; strict validators reject the mislabeled request. Added artemis.utils.image_mime sniffing helpers (PNG/JPEG magic, PNG project-default fallback) and applied them to all 21 data-URI/Part/ImageContent sites. Covered by new tests for the JSONC scanner, endpoint resolution, and mime detection; deterministic suite passes (2925 tests).
…e sessions
Observed during live iOS acceptance on a strict OpenAI-compatible endpoint:
- Local model servers reuse tool_call ids across turns (call_0 every
response); strict validators reject the ambiguous second pair with 400.
RobustChatModelWrapper now renames repeated ids with their answering
ToolMessages on OpenAI-wire providers only.
- Xcode drops idle interaction sessions mid-task ("Session not found");
since that error means the command never ran, the driver re-establishes
the session and retries once.
- A crashed runner orphans its interaction session; StartSession's
"already in use" error names the blocking session, so the driver ends
it and retries once.
- iOS input_text no longer hard-fails on clear_exist=true: focusing a
field like Safari's address bar select-alls its content, so the text
is typed without clearing and the result states that honestly.
simctl recordVideo's host lock is shared across simulators, so a recorder orphaned by a killed task or console made every later spawn exit before the first-frame marker with an opaque "closed stderr" error. Sweep stale recorders before each spawn, retry once on "already in progress" after a cross-device sweep, and surface simctl's own error text in the startup failure. Also tell the model iOS has no Back key and that typed URLs must be submitted with enter or the keyboard's Go button.
banggaoo
force-pushed
the
feat/ios-support
branch
from
October 9, 2026 09:27
c078965 to
92e0a71
Compare
The 120s ffmpeg budget was too small for multi-minute full-resolution simulator captures: finalization of a ~4.5min, 1206x2622 recording timed out during session teardown under load, leaving only the raw .mov. Scale the encode timeout with the segment's wall span (floor 180s, 2x span).
Two reliability fixes from live iOS runs: the step_capsule memory lens hardcoded the raw Google path, so deployments on custom providers (e.g. a local OpenAI-compatible backend) built a keyless client and failed every summary attempt — it now rides the configured summarizer endpoint when that endpoint is not Google. And the Flash loop had no stall guard: a model repeating the identical tool-less response (deterministic at temperature zero) burned turns forever. Three identical responses in a row now end the run with a failed report naming the stall, and a repeat earns a sharper nudge than a first miss.
adb_endpoint.py grew into the home for cross-platform primitives when iOS landed: AdbTarget/IosTarget, normalize_device_platform, SUPPORTED_PLATFORMS, and the pool/target fan-out all lived in a file named for Android's debug bridge, which hid the platform layer from anyone reading the tree. Move the cross-platform vocabulary to artemis/runtime/device_target.py (targets, platform normalization, device_pool_for/target_for_platform, IOS_LOCK_SCOPE) and leave adb_endpoint.py purely ADB (AdbEndpoint, AdbSession, adb_command, endpoint resolution). The dependency is one-way — device_target imports the endpoint primitives, adb_endpoint knows nothing about platforms — so no import cycles. runtime/__init__ keeps re-exporting the public surface, and the ~20 import sites move to the honest path. Also stop serializing the caller's ADB endpoint into iOS queue items: the snapshot now emits adb_endpoint: null instead of a meaningless Android endpoint blob (consumers already ignored it, but the API surface lied). And unflake test_startup_cancellation_reaps_child: the pre-spawn stale- recorder sweep added a suspension point before the child exists, so a fixed 20ms sleep raced the spawn. Cancel after the child is created.
Live runs showed the identical-thought guard is too narrow: a stalled model can alternate two paraphrased phrasings forever (physical-device run burned 23 turns cycling 'current observation is...' / 'I have spent many turns...' without a single tool call). Add MAX_SILENT_TURN_STREAK=8 as the backstop: any run of 8 consecutive tool-less turns fails honestly, while the stricter 3-identical check still terminates pure deterministic loops earlier.
Author
E2E evidence — iOS task completing through the Device Hub console
Screenshot: Device Hub console mid-run on the
Additional verification on
|
Author
Code architecture — module mapDesign goal: iOS is a peer platform behind the existing observation/action contract — no parallel pipeline, no Android changes.
|
Simulator (xcode_driver.py) |
Physical (physical_driver.py) |
|
|---|---|---|
| Input (tap/swipe/type/keys) | xcrun mcpbridge stdio MCP → Xcode DeviceInteraction (bridge.py) |
WebDriverAgent HTTP client (wda.py, stdlib-only) |
| Discovery | simctl list devices (JSON) |
devicectl / CoreDevice (incl. IPv6 tunnel addr) |
| Screenshot / hierarchy | simctl io screenshot + MCP XML |
WDA /screenshot, /source |
| App lifecycle | simctl launch/terminate; listapps via plutil (OpenStep plist) |
devicectl process launch/terminate, install |
| Recording | simctl io recordVideo, segmented → MP4 (timeout scales with span) |
devicectl polled capture (lower fps) |
| WDA transport | n/a | Direct xctrunner launch, or xcodebuild test-without-building via ARTEMIS_IOS_WDA_XCTESTRUN; endpoint override ARTEMIS_IOS_WDA_URL |
Session safety: physical_driver refuses to replace an active WDA session it doesn't own — POST /session would kill it.
artemis/runtime/ — platform routing split
device_target.py(new):IosTarget/AdbTarget,IOS_LOCK_SCOPE,normalize_device_platform,device_pool_for,target_for_platform. Both targets implementlock_scope/lock_key/apply_to_environment/to_dictfor the shared queue,DeviceExecutionLock, and worker-env machinery.adb_endpoint.py(restored): Android-onlyAdbEndpoint/AdbSession/adb_command. Dependency is one-directional (device_target→adb_endpoint), no cycles.- iOS locks are namespaced
ios/<UDID>;IosTarget.apply_to_environmentscrubsADB_DEVICE_SERIAL/ARTEMIS_DEVICE_ID; queue payloads emitadb_endpoint: nullfor iOS tasks. artemis.runtime.__init__re-exports everything — existing import sites unchanged.
Flow-through surfaces
--platform ios --device-serial <UDID>: CLI run/batch → SDK (embedded + remote; remote refuses legacy hosts for iOS payloads) → daemon task queue → worker → admin console (platform tabs, /api/status, queue payloads) → MCP tools → replay/streaming.
Safety rails
- ADB/
press_key(BACK)/logcat/app-lock reject iOS explicitly — errors, not silent no-ops. - Physical devices require explicit UDID — pool auto-select stays simulator-only.
XcodeBridgeowns onemcpbridgesubprocess per sim driver; cancellation-safe request draining.
A crashed mcpbridge surfaced as "MCP connection failed: unhandled errors
in a TaskGroup", which missed the transient filter, and a single failed
session re-acquire ("session identifier is currently in use or was
recently used") left the driver permanently disconnected.
Broaden the transient match to include MCP connection failures, restart
the bridge when disconnected, and retry session establishment with a
fresh label and backoff (bounded to 3 attempts) inside
_restart_interaction_session. _start_interaction_session also retries
once with a new label when Xcode reports the identifier recently used.
Verified live: killed mcpbridge mid-task on the simulator; the driver
detected the drop, respawned the bridge, acquired a fresh interaction
session, and the task completed with a PASS trace.
banggaoo
force-pushed
the
feat/ios-support
branch
from
October 10, 2026 03:50
6999edb to
f3a158e
Compare
A bare /session request binds WebDriverAgent to an ephemeral pid.0 application that dies immediately, surfacing as "stale element reference" HTTP 404s on the first command. Binding to com.apple.Preferences anchors the session to a real, always-installed app; com.apple.springboard cannot be activated as an app target on current iOS. Verified on an iPhone 15 Pro: session commands, launch, and scroll all succeed.
devicectl's "info apps" defaults to developer-installed apps only, so find_package rejected com.apple.mobilesafari and friends with "Installed iOS app not found" even though the apps exist and launch fine. Pass --include-default-apps so system apps are resolvable for launch and stop, matching what simctl listapps exposes on simulators.
banggaoo
force-pushed
the
feat/ios-support
branch
from
October 10, 2026 06:51
d1bf83a to
4933cd5
Compare
A WDA session dies when its bound app or the runner is replaced mid-task; every subsequent session-scoped call then fails with 'invalid session id' while device-level screenshots keep working, so tasks loop forever. Rebind transparently on session-loss errors: probe the foreground app for the new anchor (falling back to Preferences on the home screen), drop the zombie session first so the replacement is not refused, rewrite the old session id in the request path, and retry exactly once under a lock so concurrent requests share one rebound session. Timed-out writes are never replayed (the input may have landed); timed-out reads rebind and retry once. Session create/delete are excluded from recovery.
banggaoo
force-pushed
the
feat/ios-support
branch
from
October 10, 2026 07:00
802094f to
48d083b
Compare
Captures behavior verified on real hardware: sessions must be bundleId-anchored (bare sessions bind to a dying pid.0), springboard is not an activatable anchor, dead sessions rebind to the foreground app under a lock with timed-out writes never replayed, UI Automation authorization (Code=41) survives only until a respring, system apps resolve via devicectl's default-app listing, and WDA's serial request queue explains intermittent stalls.
Live verification on a physical iPhone showed WDA answers a killed anchor app with 'invalid element state: The application under test ... is not running, possibly crashed' rather than an invalid-session error, so the dead-session markers never matched and the task looped. Match the message body; the generic 'invalid element state' code also covers legitimately unhittable elements and must not trigger rebinds. Verified end-to-end: Settings terminated mid-task, the session rebound to a fresh anchor, the action retried once, and the task completed with a PASS trace.
Author
Update — post-submission hardening, verified on real hardwareVerified head is now
Physical runs: iPhone 15 Pro (iOS 27) via |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: add native iOS support for simulators and physical devices
What Changes
Add one opt-in iOS feature covering both simulators and paired physical
iPhones/iPads across the CLI, embedded and remote SDKs, daemon, MCP, and console.
simctl.polled recording use
devicectl.stream targets, and recorded-session replay preserve the selected target.
workspace explicitly, and refuses legacy hosts before submitting iOS tasks.
Developer Mode, WDA provisioning, and supported/unsupported operations.
Android Compatibility
Android remains the default platform. The Android driver implementation is not
replaced. Shared routing changes are platform-gated, and Android task/lock/
observation/video regressions are included in verification.
Hardening
The submission also addresses observed integration risks: cancellation-safe
native/HTTP child ownership, leased diagnostic sessions, scope-aware queue
deduplication, actual CoreDevice JSON compatibility, unambiguous identifiers,
and physical-device selection in the console.
Verification
Verified head:
b199e0eonfeat/ios-support. Results below are from thatlineage (acceptance at code commit
19e04cc;65c3412is docs-only andb199e0efixes provider-compat bugs — JSONC comment stripping, customendpoint pass-through, and screenshot MIME labels — found while running a
real iOS task through the console; deterministic suite now 2925 passed).
Platform Limits
Requires macOS and Xcode 27+. Xcode binds agent approval to the client
binary's code signature: unsigned/adhoc interpreters (uv/Homebrew Python)
receive ~24-hour grants requiring periodic re-approval, while signed agents
can hold persistent grants — documented in
docs/ios.md. Hardware needs auser-provisioned, reachable WDA runner and the user's device consent; WDA's
HTTP server cannot start while the device is locked. Physical recording is
lower-frame-rate screenshot polling; no iOS audio recording is added. Android
shell/logcat/app locking, unsupported keys, and cloud execution are not
advertised as iOS features.
Closes #174