iOS: dismissing a com.apple.SafariViewService-hosted sheet works via replay/manual but not via test; index and repeat each break differently

[TheWygoda]

Summary

Driving a Maestro-compatible flow through agent-device test on the iOS Simulator, a step that must dismiss a system-hosted sheet — specifically iOS's "Save Password?" prompt after a form submission, served by com.apple.SafariViewService per agent-device's own diagnostic hints — consistently fails to reach the screen behind it, even though the identical logical action works correctly via agent-device replay and via manual step-by-step press commands in the same session. Two other approaches tried to make the dismiss-tap robust (index disambiguation, wrapping in a repeat block) each broke in their own distinct way against this same system surface.

Version: agent-device 0.21.1.
Environment: iOS Simulator (iPhone 17 Pro Max, iOS 26.5), macOS 25.6 (Darwin), Expo dev-client app (debug, Metro-connected), driven via the --maestro Maestro-compatibility engine.

This looks closely related to #2438 and its tracked follow-ups (#2450, #2451, #2485, #2489) — same underlying com.apple.SafariViewService cross-process surface, different concrete symptom (a plain confirmation sheet, not an ASWebAuthenticationSession). Posting as a separate report since the specific failure modes below (a hard index failure, and repeat silently not dispatching) don't look identical to what those describe, but they may share a root cause or fix — happy to have this folded into one of them if a maintainer sees the overlap.

Reproduction

Flow (Maestro-compatible YAML), abbreviated to the relevant steps:

appId: com.example.app
---
- launchApp
- tapOn: { id: "email-field" }
- inputText: someone@example.com
- tapOn: { id: "password-field" }
- inputText: some-password
- tapOn: "Sign in"
# iOS's system "Save Password?" sheet appears here, a few seconds after
# the tap, once the sign-in network round trip resolves.
- tapOn: { text: "Not Now", optional: true }
- assertVisible: "Today"

What works

• The exact same steps, run via agent-device replay <file> --maestro --udid <udid>, dismiss the sheet correctly and reach the screen behind it, repeatedly, across multiple runs.
• Driving the same tap manually, step by step, via agent-device press 'text="Not Now"' --udid <udid> --settle (or by exact @ref), also dismisses it correctly every time it was tried.
• agent-device snapshot -i while the sheet is up correctly reports it as a distinct surface:

  A system web sign-in sheet is presented over the app, so this snapshot shows
  that sheet (hosted out of the app process). Its controls are real and
  interactive; complete or dismiss the sheet to return to app content.
  Simulator AX snapshot inapplicable (system-surface-presented); used XCTest
  to read the system surface presented over the app.
  @e1 [application] "Safari"
  @e2 [window] "Save Password?"
  ...
  @e9 [button] "Not Now"
  @e10 [button] "Save"
  

What doesn't: agent-device test

Running the identical flow via agent-device test <file>.yml --maestro --udid <udid> --metro-port <port> -e ... consistently fails to reach the screen behind the sheet — assertVisible: "Today" times out with COMMAND_FAILED, and the final failure snapshot still shows the sign-in screen underneath (i.e. the sheet was up and never dismissed). This reproduced across roughly a dozen runs, with three different versions of the dismiss step (see below), and with --record-video both on and off (ruled out as a confound).

Per-step timing from one such run's replay-timing.ndjson (durationMs, ok):

tapOn "Sign in": ok, 924ms
waitForAnimationToEnd: ok, 523ms
tapOn { text: "Not Now", optional: true }: ok, 7103ms   <- note the duration
assertVisible "Today": FAILED, COMMAND_FAILED, 17088ms

The 7-second duration on a tapOn ... optional: true that (per manual testing) should resolve in under a second if the sheet is present, or fail fast if it isn't, suggests something is being retried or polled internally without either succeeding or surfacing an error — the step reports ok: true but the sheet was not actually dismissed.

Three specific things tried, each breaking differently

1. extendedWaitUntil before the tap — reasonable given the sheet's real, observed arrival-time variance (anywhere from under a second to several seconds after the Sign in tap). But it hard-fails the whole flow if the sheet doesn't appear within the timeout, which is also a real, observed outcome on this same simulator+credential pair after enough repeated runs — so this is wrong by construction for a step that must tolerate absence, not just a robustness gap.

2. index to disambiguate an AMBIGUOUS_MATCH — text="Not Now" resolves to 3 accessibility nodes for the one button (a scroll-area and a cell both inherit the button's own accessible label), which agent-device's native press command correctly refuses with AMBIGUOUS_MATCH. Adding index: 2 (targeting what a raw snapshot dump showed as the innermost/actual UIButton) to the Maestro-compatible tapOn breaks differently and harder: it fails outright with

Error (REPLAY_DIVERGENCE): Replay failed at step N (tapOn "Not Now"): regular iOS
snapshot presentation requires a valid viewport

wrapped in optional: true, this error is swallowed — the tap never fires, no exception propagates, and the step reports success.

3. Wrapping the plain, un-indexed, optional tap in repeat —

- repeat:
    times: 5
    commands:
      - tapOn: { text: "Not Now", optional: true }
      - waitForAnimationToEnd: { timeout: 1500 }

builds and runs without error (the repeat step itself reports ok: true), but the sheet is never actually dismissed — confirmed by the subsequent assertVisible failing every time this was tried, and by the fact that this is the only structural difference from the version that does work via replay (same selector, same optional: true, just wrapped).

What I ruled out

• Not the app. The exact same tap, dismissed by hand or via replay, reaches the correct screen every time; the underlying sign-in and navigation are correct.
• Not credentials or rate limiting. A direct auth API call with the same email+password succeeded throughout, including in the same window as failing test runs.
• Not --record-video. Removing it from the test invocation did not change the outcome.
• Not the ambiguous-match itself, in the working cases. The plain, un-indexed tapOn: { text: "Not Now", optional: true } — the same selector that's ambiguous to the native press command — resolves and dispatches correctly under both replay and manual press. Only index, repeat, and test as a whole broke it.

What would help

1. Whatever test does differently from replay for the same Maestro-compatible step against a com.apple.SafariViewService-hosted surface — worth a look given fix(ios): distinct lineage for in-place system-surface captures (verify/settle) #2450/fix(ios): system-surface disclosure parity across selector routes #2451/perf(ios): detect a presented system surface from the bridge via AX hit test, replacing the host-side process probe #2485 are already touching exactly this boundary.
2. index failing outright against a system-surface capture, rather than resolving or failing closed with a clearer message — right now the error text ("regular iOS snapshot presentation requires a valid viewport") gives no hint that the cause is the system-surface boundary specifically, and optional: true swallowing it silently makes it worse to diagnose.
3. repeat's per-iteration settle/dispatch behavior against a system surface — possibly related to fix(ios): distinct lineage for in-place system-surface captures (verify/settle) #2450's "distinct lineage for in-place system-surface captures" if repeat's own settle comparison between iterations is what's going wrong.

Happy to share the full flow file, replay-timing.ndjson, and failure artifacts if useful.

🤖 Generated with Claude Code