feat(runner): tap, swipe and fill actions for mobile runners (AGE-4855)

.changeset/runner-act-mobile-actions.md

@@ -0,0 +1,5 @@
+---
+"@qawolf/cli": minor
+---
+
+`qawolf runner act` has mobile actions of its own: `tap` (at `--x`/`--y` or on a `--selector`), `swipe` (`--from x,y --to x,y`, optional `--duration-ms`), and `fill` (replaces the value of the `--selector` field with `--text`). `click` and `drag` still tap and swipe on a mobile runner, but are deprecated there. A browser runner refuses the mobile actions with `action-not-supported-on-browser`. Pins `@qawolf/api-contracts` 0.73.0.

package.json

@@ -71,7 +71,7 @@
     "@clack/prompts": "1.5.1",
     "@napi-rs/keyring": "1.3.0",
     "@oxc-node/core": "0.1.0",
-    "@qawolf/api-contracts": "0.70.0",
+    "@qawolf/api-contracts": "0.73.0",
     "@qawolf/emails": "1.1.1",
     "@qawolf/flow-targets": "1.0.0",
     "@qawolf/flows": "0.1.4",

skills/qawolf-cli/SKILL.md

@@ -168,7 +168,7 @@ that `url`; never guess a route and never send a repository link in its place.
 | `qawolf install clear` | local | Remove the managed runtime cache (all installed runtime versions) |
 | `qawolf investigation get` | read | Read what the QA Wolf AI investigating a run has concluded: one finding per cause of its failures, with the question waiting for a person and any answer or dispute the investigation hasn't acted on yet. |
 | `qawolf issue addFlows` | write | Add flows to a coverage request owned by the caller's team. Flows already covered stay covered. Bug and maintenance reports link to flows through the runs that reproduce them; use run.diagnose to record one. |
-| `qawolf issue create` | write | Create a bug or coverage request issue for the caller's team. Maintenance issues cannot be created through the public API. |
+| `qawolf issue create` | write | Create a bug report, maintenance report, or coverage request issue for the caller's team. A user can create a maintenance report only when they can see the workspace's maintenance reports; a team or organization API key credits it to the workspace's automation user. Link its failed flows afterwards with run.diagnose. |
 | `qawolf issue find` | read | List the team's bug reports, maintenance reports, or coverage requests, newest first. |
 | `qawolf issue get` | read | Get an issue by id. |
 | `qawolf issue removeFlows` | write | Remove flows from a coverage request owned by the caller's team. Flows the request does not cover are left alone. Bug and maintenance reports link to flows through the runs that reproduce them, so their flows cannot be set directly. |
@@ -182,8 +182,7 @@ that `url`; never guess a route and never send a repository link in its place.
 | `qawolf run get` | read | Get a run's status, per-flow results, links, and how many of its bugs are blocking. |
 | `qawolf run reattempt` | write | Request new attempts for a run's flows, in the same run. A flow is eligible once its result is failed or canceled and QA Wolf's automatic retries have finished. A fully investigated run no longer accepts reattempts. Attempts run with the latest flow code. Poll run.get for results. |
 | `qawolf run stop` | write | Stop a run, including its queued flows and automatic retries. Stopping is asynchronous and can update run-status messages and commit statuses in connected integrations. Repeated requests are safe, and finished runs keep their results. A run that is still being created returns not found; retry once run.get returns the run. If run.get returns a different runId, use that ID. Poll run.get for results. |
-| `qawolf run triage` | read | Prototype. Ask Jev, a TypeSafe System One model, whether a flow failure in a run is a bug in the product or a test that needs maintenance. Compares the failing attempt with the flow's last passing run in the same environment. Records nothing. |
-| `qawolf runner act` | write | Perform one raw action on a runner's screen: click, double_click, scroll, move, drag, keypress, navigate or type. Use - to read a whole action as JSON from stdin. On a mobile runner only click (button left), drag and type have a touchscreen equivalent; the rest answer action-not-supported-on-mobile |
+| `qawolf runner act` | write | Perform one raw action on a runner's screen. A browser runner takes click, double_click, scroll, move, drag, keypress, navigate and type; a mobile runner takes tap, swipe, fill and type. A browser runner answers mobile actions with action-not-supported-on-browser; a mobile runner answers the other browser actions with action-not-supported-on-mobile. Use - to read a whole action as JSON from stdin |
 | `qawolf runner actions` | write | Perform a sequence of up to ten raw actions on a runner's screen in one request, as a JSON array of the same actions `runner act` takes. Use - to read the array from stdin. Actions run back to back, so batch only steps whose targets are on the screen you last saw, and end the sequence at the step that changes the page. Each result carries an effect: performed, not-performed, or unknown when the runner stopped answering and the action may have landed |
 | `qawolf runner events` | read | Print a runner's journal, one entry per line. QA Wolf writes console, recorder, run-events, run-logs, run-status |
 | `qawolf runner exec` | write | Evaluate a snippet against a runner's live page. Use - to read the snippet from stdin |

skills/qawolf-cli/references/run-results.md

@@ -183,6 +183,7 @@ Every documented field of the `run.get` response. `[]` marks an array, so
 - `completedAt` — When the run finished executing. Absent while queued or running, and also absent for a terminal run that never completed execution (e.g. every flow was canceled or skipped).
 - `git` — The branch and commit under test. The fields are present when a deploy notification started the run, and absent for runs started another way, for example manually or with run.create.
 - `git.commitUrl` — Link to the commit on the code host.
+- `needsReview` — True once every flow has finished its attempts and at least one failed flow still has no diagnosis, including one whose investigation carried over to a later run. `status` stays `running` until that failure is diagnosed, so a caller that is not waiting for a diagnosis can stop polling here and read the run's failed flows.
 - `runId` — The run this response describes. Treat it as canonical: it can differ from the id you asked for. A deploy notification returns a run id before the run exists, and if a second notification for the same commit is folded into an earlier run, that id resolves to the earlier run instead.
 - `status` — One of: queued, running, passed, failed, canceled
 - `blockingBugCount` — How many bugs this run found are blocking: still open, and priority urgent, high, or unprioritized — unprioritized counts because nobody has ruled it out yet. This is what `failed` is derived from, so it explains a failure rather than adding a second verdict: gate on `status`, then read this to say how many bugs are holding the build. It counts a bug filed against a later run by an investigation that carried over from this one, and it drops a bug once that bug is resolved.

skills/qawolf-cli/references/runner.md

@@ -126,7 +126,7 @@ Retry on the exit code, not on the message text:
   launch the id or name one that is running. The message says which runner was
   meant and whether `--runner`, `QAWOLF_RUNNER_ID` or this directory's stored
   default chose it — read that line before you pick an id to launch.
-- `2` will not clear on its own. Nothing has run on this runner yet, so run a flow; or the runner has no browser at all, so launch with `--name playwright` instead; or the action has no touchscreen equivalent on a mobile runner, so reach for one that has. A `runner actions` sequence also exits `2` before it sends anything, for an argument that is not a JSON array, an array of more than ten actions, and `--screenshot` flags that contradict each other. The message says which.
+- `2` will not clear on its own. Nothing has run on this runner yet, so run a flow; or the runner has no browser at all, so launch with `--name playwright` instead; or the action belongs to the other runner family, so send `tap`, `swipe` or `fill` to a mobile runner and the browser actions to a browser runner. A `runner actions` sequence also exits `2` before it sends anything, for an argument that is not a JSON array, an array of more than ten actions, and `--screenshot` flags that contradict each other. The message says which.
 
 ## Seeing and acting: the loop is yours
 
@@ -167,17 +167,18 @@ Coordinates are pixels on the same screenshot you just read. The runner serves o
 
 `act`, `actions`, `run` and `exec` are the commands whose lost answer may still have taken effect. On a `4` from `act` or `actions`, take a screenshot before repeating a click. `exec`'s message says the snippet could not be evaluated, but a lost answer looks the same from outside, so treat a `4` from a snippet that changes something as "may have run" rather than "did not run".
 
-A mobile runner has a touchscreen, not a mouse, so only three of the eight
-actions have a touchscreen equivalent and go through: `click` with
-`button: "left"` taps, `drag` swipes, and `type` types into whatever the last
-tap focused. The rest — `double_click`, `scroll`, `move`, `keypress`,
-`navigate` — answer `action-not-supported-on-mobile` rather than doing
-something approximate. `navigate` is the one to watch for, since it works on a
-browser runner without a run first but has no meaning on mobile at all.
+A mobile runner has a touchscreen, not a mouse, so it has actions of its own:
+
+- `tap` touches a point (`--x`, `--y`) or the element `--selector` names. Check the selector first with `inspect elements --selector`, and pass `--strategy ios-predicate` or `shadow` when it is not XPath.
+- `swipe --from x,y --to x,y` moves in a straight line between two points. `--duration-ms` sets how long it takes, up to 10000: a slow swipe scrolls, a fast one flings.
+- `fill --selector ... --text ...` replaces the value of that field, and `--text ""` clears it. To add to what a field already holds, `tap` it and then `type`.
+- `type` types into whatever the last tap focused, the same as on a browser.
+
+The browser actions `double_click`, `scroll`, `move`, `keypress` and `navigate` answer `action-not-supported-on-mobile` rather than doing something approximate. `navigate` is the one to watch for, since it works on a browser runner without a run first but has no meaning on mobile at all. `click` with `button: "left"` and `drag` still tap and swipe on mobile, but they are deprecated there, so send `tap` and `swipe`. A browser runner answers `tap`, `swipe` and `fill` with `action-not-supported-on-browser`.
 
 ### Several steps in one request: `runner actions`
 
-`qawolf runner actions '<json array>'` performs up to ten of those same actions back to back in one request, for the steps you already know: click the field, type into it, press Enter. One round trip instead of three, with no delay to guess at between them, and `-` reads the array from stdin the way `act -` reads one action.
+`qawolf runner actions '<json array>'` performs up to ten of the browser actions back to back in one request (not `tap`, `swipe` or `fill`), for the steps you already know: click the field, type into it, press Enter. One round trip instead of three, with no delay to guess at between them, and `-` reads the array from stdin the way `act -` reads one action.
 
 ```sh
 qawolf runner actions '[{"type":"click","button":"left","x":480,"y":260},{"type":"type","text":"me@example.com"},{"type":"keypress","keys":["Enter"]}]' --screenshot after-login.jpg

src/commands/__snapshots__/help.test.ts.snap

@@ -293,7 +293,7 @@ Commands:
   run [options] <flowFile>                 Run a flow on an interactive runner, shipping the flow and what it imports
   events [options] <stream>                Print a runner's journal, one entry per line. QA Wolf writes console, recorder, run-events, run-logs, run-status
   screenshot [options]                     Save a JPEG of an interactive runner's screen to a file, or write it to stdout with --out -
-  act [options] <action>                   Perform one raw action on a runner's screen: click, double_click, scroll, move, drag, keypress, navigate or type. Use - to read a whole action as JSON from stdin. On a mobile runner only click (button left), drag and type have a touchscreen equivalent; the rest answer action-not-supported-on-mobile
+  act [options] <action>                   Perform one raw action on a runner's screen. A browser runner takes click, double_click, scroll, move, drag, keypress, navigate and type; a mobile runner takes tap, swipe, fill and type. A browser runner answers mobile actions with action-not-supported-on-browser; a mobile runner answers the other browser actions with action-not-supported-on-mobile. Use - to read a whole action as JSON from stdin
   actions [options] <sequence>             Perform a sequence of up to ten raw actions on a runner's screen in one request, as a JSON array of the same actions \`runner act\` takes. Use - to read the array from stdin. Actions run back to back, so batch only steps whose targets are on the screen you last saw, and end the sequence at the step that changes the page. Each result carries an effect: performed, not-performed, or unknown when the runner stopped answering and the action may have landed
   exec [options] <file>                    Evaluate a snippet against a runner's live page. Use - to read the snippet from stdin
   inspect                                  Read one thing off a runner's live page (browser) or Appium session (mobile)
@@ -553,31 +553,40 @@ Examples:
 exports[`--help output qawolf runner act 1`] = `
 "Usage: qawolf runner act [options] <action>
 
-Perform one raw action on a runner's screen: click, double_click, scroll, move,
-drag, keypress, navigate or type. Use - to read a whole action as JSON from
-stdin. On a mobile runner only click (button left), drag and type have a
-touchscreen equivalent; the rest answer action-not-supported-on-mobile
+Perform one raw action on a runner's screen. A browser runner takes click,
+double_click, scroll, move, drag, keypress, navigate and type; a mobile runner
+takes tap, swipe, fill and type. A browser runner answers mobile actions with
+action-not-supported-on-browser; a mobile runner answers the other browser
+actions with action-not-supported-on-mobile. Use - to read a whole action as
+JSON from stdin
 
 Options:
-  --button <button>    click: left, right, wheel, back or forward (mobile: left
-                       only)
-  --keys <keys...>     keypress: modifiers and the key, e.g. Control a
-  --path <json>        drag: JSON array of points to drag through (mobile: only
-                       the first and last are used)
-  --runner <id>        Runner to target. Defaults to QAWOLF_RUNNER_ID, then this
-                       directory's stored runner
-  --screenshot <path>  Also save a JPEG of the screen, taken after the action,
-                       to this file, in place of a separate screenshot. An
-                       action that did not take effect answers with one too. -
-                       writes it to stdout and moves the confirmation, JSON
-                       included, to stderr
-  --scroll-x <delta>   scroll: horizontal wheel delta
-  --scroll-y <delta>   scroll: vertical wheel delta
-  --text <text>        type: the text to type
-  --url <url>          navigate: the http or https URL to go to
-  --x <pixels>         pointer x, in screenshot pixels
-  --y <pixels>         pointer y, in screenshot pixels
-  -h, --help           display help for command
+  --button <button>      click: left, right, wheel, back or forward
+  --duration-ms <ms>     swipe: how long it takes, up to 10000. Slow scrolls,
+                         fast flings
+  --from <x,y>           swipe: the point it starts at
+  --keys <keys...>       keypress: modifiers and the key, e.g. Control a
+  --path <json>          drag: JSON array of points to drag through
+  --runner <id>          Runner to target. Defaults to QAWOLF_RUNNER_ID, then
+                         this directory's stored runner
+  --screenshot <path>    Also save a JPEG of the screen, taken after the action,
+                         to this file, in place of a separate screenshot. An
+                         action that did not take effect answers with one too. -
+                         writes it to stdout and moves the confirmation, JSON
+                         included, to stderr
+  --scroll-x <delta>     scroll: horizontal wheel delta
+  --scroll-y <delta>     scroll: vertical wheel delta
+  --selector <selector>  tap or fill: the element to act on, as a screen object
+                         would find it. tap takes it in place of --x and --y
+  --strategy <strategy>  how --selector is resolved: xpath (default),
+                         ios-predicate or shadow
+  --text <text>          type: the text to type into what has focus. fill: the
+                         field's new value
+  --to <x,y>             swipe: the point it ends at
+  --url <url>            navigate: the http or https URL to go to
+  --x <pixels>           click, tap and the like: x, in screenshot pixels
+  --y <pixels>           click, tap and the like: y, in screenshot pixels
+  -h, --help             display help for command
 
 Examples:
   $ qawolf runner act click --button left --x 480 --y 260
@@ -588,6 +597,12 @@ Examples:
   $ qawolf runner act click --button left --x 480 --y 260 --screenshot step-4.jpg
   $ echo '{"type":"click","button":"left","x":1,"y":2}' | qawolf runner act -
   $ echo '{"type":"click","button":"left","x":1,"y":2}' | qawolf runner act - --screenshot - > step-5.jpg
+
+Mobile:
+  $ qawolf runner act tap --x 540 --y 1200
+  $ qawolf runner act tap --selector '//*[@content-desc="Continue"]'
+  $ qawolf runner act fill --selector 'name == "Postal code"' --strategy ios-predicate --text 94107
+  $ qawolf runner act swipe --from 540,1600 --to 540,600 --duration-ms 1500
 "
 `;
 

src/commands/runner/actExamples.ts

@@ -0,0 +1,16 @@
+export const actExamples = `
+Examples:
+  $ qawolf runner act click --button left --x 480 --y 260
+  $ qawolf runner act type --text "hello@example.com"
+  $ qawolf runner act keypress --keys Control a
+  $ qawolf runner act navigate --url https://example.com
+  $ qawolf runner act drag --path '[{"x":10,"y":20},{"x":80,"y":90}]'
+  $ qawolf runner act click --button left --x 480 --y 260 --screenshot step-4.jpg
+  $ echo '{"type":"click","button":"left","x":1,"y":2}' | qawolf runner act -
+  $ echo '{"type":"click","button":"left","x":1,"y":2}' | qawolf runner act - --screenshot - > step-5.jpg
+
+Mobile:
+  $ qawolf runner act tap --x 540 --y 1200
+  $ qawolf runner act tap --selector '//*[@content-desc="Continue"]'
+  $ qawolf runner act fill --selector 'name == "Postal code"' --strategy ios-predicate --text 94107
+  $ qawolf runner act swipe --from 540,1600 --to 540,600 --duration-ms 1500`;