From f4ea8dea6849bc652e588b5446e9af480a8c7657 Mon Sep 17 00:00:00 2001 From: Erzhan Torokulov Date: Wed, 30 Sep 2026 12:28:34 +0000 Subject: [PATCH 1/4] feat(runner): tap, swipe and fill actions for mobile runners in runner act (AGE-4855) --- .changeset/runner-act-mobile-actions.md | 5 + package.json | 2 +- skills/qawolf-cli/SKILL.md | 3 +- skills/qawolf-cli/references/runner.md | 19 ++-- src/commands/__snapshots__/help.test.ts.snap | 62 +++++++---- src/commands/runner/actExamples.ts | 16 +++ src/commands/runner/interact.register.ts | 57 ++++++---- .../interactiveRunner/browserAction.test.ts | 104 +++++++++++++++++- src/core/interactiveRunner/browserAction.ts | 70 ++++++++---- src/core/interactiveRunner/flagValues.ts | 28 +++++ .../messages/interactiveRunner/interact.ts | 4 +- .../performAction.outcomes.test.ts | 35 +++++- .../performAction.screenshot.failures.test.ts | 5 + .../performAction.screenshot.refused.test.ts | 5 + .../performAction.screenshot.test.ts | 5 + .../interactiveRunner/performAction.test.ts | 11 +- .../interactiveRunner/performAction.ts | 6 +- .../interactiveRunner/performActionFailure.ts | 6 + .../performActionScreenshot.ts | 4 +- src/domains/interactiveRunner/readAction.ts | 14 +-- src/runnerSdk/types.ts | 4 +- 21 files changed, 357 insertions(+), 108 deletions(-) create mode 100644 .changeset/runner-act-mobile-actions.md create mode 100644 src/commands/runner/actExamples.ts create mode 100644 src/core/interactiveRunner/flagValues.ts diff --git a/.changeset/runner-act-mobile-actions.md b/.changeset/runner-act-mobile-actions.md new file mode 100644 index 000000000..cf160ebb3 --- /dev/null +++ b/.changeset/runner-act-mobile-actions.md @@ -0,0 +1,5 @@ +--- +"@qawolf/cli": minor +--- + +`qawolf runner act` has mobile actions of its own: `tap` (at `--x`/`--y` or on a `--selector`), `swipe` (`--from x,y --to x,y`, optional `--duration-ms`), and `fill` (replaces the value of the `--selector` field with `--text`). `click` and `drag` still tap and swipe on a mobile runner, but are deprecated there. A browser runner refuses the mobile actions with `action-not-supported-on-browser`. Pins `@qawolf/api-contracts` 0.72.0. diff --git a/package.json b/package.json index 839c3ecb6..9ac05ff3b 100644 --- a/package.json +++ b/package.json @@ -71,7 +71,7 @@ "@clack/prompts": "1.5.1", "@napi-rs/keyring": "1.3.0", "@oxc-node/core": "0.1.0", - "@qawolf/api-contracts": "0.70.0", + "@qawolf/api-contracts": "0.72.0", "@qawolf/emails": "1.1.1", "@qawolf/flow-targets": "1.0.0", "@qawolf/flows": "0.1.4", diff --git a/skills/qawolf-cli/SKILL.md b/skills/qawolf-cli/SKILL.md index f7be762c3..160751376 100644 --- a/skills/qawolf-cli/SKILL.md +++ b/skills/qawolf-cli/SKILL.md @@ -182,8 +182,7 @@ that `url`; never guess a route and never send a repository link in its place. | `qawolf run get` | read | Get a run's status, per-flow results, links, and how many of its bugs are blocking. | | `qawolf run reattempt` | write | Request new attempts for a run's flows, in the same run. A flow is eligible once its result is failed or canceled and QA Wolf's automatic retries have finished. A fully investigated run no longer accepts reattempts. Attempts run with the latest flow code. Poll run.get for results. | | `qawolf run stop` | write | Stop a run, including its queued flows and automatic retries. Stopping is asynchronous and can update run-status messages and commit statuses in connected integrations. Repeated requests are safe, and finished runs keep their results. A run that is still being created returns not found; retry once run.get returns the run. If run.get returns a different runId, use that ID. Poll run.get for results. | -| `qawolf run triage` | read | Prototype. Ask Jev, a TypeSafe System One model, whether a flow failure in a run is a bug in the product or a test that needs maintenance. Compares the failing attempt with the flow's last passing run in the same environment. Records nothing. | -| `qawolf runner act` | write | Perform one raw action on a runner's screen: click, double_click, scroll, move, drag, keypress, navigate or type. Use - to read a whole action as JSON from stdin. On a mobile runner only click (button left), drag and type have a touchscreen equivalent; the rest answer action-not-supported-on-mobile | +| `qawolf runner act` | write | Perform one raw action on a runner's screen. A browser runner takes click, double_click, scroll, move, drag, keypress, navigate and type; a mobile runner takes tap, swipe, fill and type. Each family answers the other's actions with action-not-supported-on-mobile or action-not-supported-on-browser. Use - to read a whole action as JSON from stdin | | `qawolf runner actions` | write | Perform a sequence of up to ten raw actions on a runner's screen in one request, as a JSON array of the same actions `runner act` takes. Use - to read the array from stdin. Actions run back to back, so batch only steps whose targets are on the screen you last saw, and end the sequence at the step that changes the page. Each result carries an effect: performed, not-performed, or unknown when the runner stopped answering and the action may have landed | | `qawolf runner events` | read | Print a runner's journal, one entry per line. QA Wolf writes console, recorder, run-events, run-logs, run-status | | `qawolf runner exec` | write | Evaluate a snippet against a runner's live page. Use - to read the snippet from stdin | diff --git a/skills/qawolf-cli/references/runner.md b/skills/qawolf-cli/references/runner.md index 50bbb1718..d0d6776f4 100644 --- a/skills/qawolf-cli/references/runner.md +++ b/skills/qawolf-cli/references/runner.md @@ -126,7 +126,7 @@ Retry on the exit code, not on the message text: launch the id or name one that is running. The message says which runner was meant and whether `--runner`, `QAWOLF_RUNNER_ID` or this directory's stored default chose it — read that line before you pick an id to launch. -- `2` will not clear on its own. Nothing has run on this runner yet, so run a flow; or the runner has no browser at all, so launch with `--name playwright` instead; or the action has no touchscreen equivalent on a mobile runner, so reach for one that has. A `runner actions` sequence also exits `2` before it sends anything, for an argument that is not a JSON array, an array of more than ten actions, and `--screenshot` flags that contradict each other. The message says which. +- `2` will not clear on its own. Nothing has run on this runner yet, so run a flow; or the runner has no browser at all, so launch with `--name playwright` instead; or the action belongs to the other runner family, so send `tap`, `swipe` or `fill` to a mobile runner and the browser actions to a browser runner. A `runner actions` sequence also exits `2` before it sends anything, for an argument that is not a JSON array, an array of more than ten actions, and `--screenshot` flags that contradict each other. The message says which. ## Seeing and acting: the loop is yours @@ -167,17 +167,18 @@ Coordinates are pixels on the same screenshot you just read. The runner serves o `act`, `actions`, `run` and `exec` are the commands whose lost answer may still have taken effect. On a `4` from `act` or `actions`, take a screenshot before repeating a click. `exec`'s message says the snippet could not be evaluated, but a lost answer looks the same from outside, so treat a `4` from a snippet that changes something as "may have run" rather than "did not run". -A mobile runner has a touchscreen, not a mouse, so only three of the eight -actions have a touchscreen equivalent and go through: `click` with -`button: "left"` taps, `drag` swipes, and `type` types into whatever the last -tap focused. The rest — `double_click`, `scroll`, `move`, `keypress`, -`navigate` — answer `action-not-supported-on-mobile` rather than doing -something approximate. `navigate` is the one to watch for, since it works on a -browser runner without a run first but has no meaning on mobile at all. +A mobile runner has a touchscreen, not a mouse, so it has actions of its own: + +- `tap` touches a point (`--x`, `--y`) or the element `--selector` names. Check the selector first with `inspect elements --selector`, and pass `--strategy ios-predicate` or `shadow` when it is not XPath. +- `swipe --from x,y --to x,y` moves in a straight line between two points. `--duration-ms` sets how long it takes, up to 10000: a slow swipe scrolls, a fast one flings. +- `fill --selector ... --text ...` replaces the value of that field. To add to what a field already holds, `tap` it and then `type`. +- `type` types into whatever the last tap focused, the same as on a browser. + +The browser actions `double_click`, `scroll`, `move`, `keypress` and `navigate` answer `action-not-supported-on-mobile` rather than doing something approximate. `navigate` is the one to watch for, since it works on a browser runner without a run first but has no meaning on mobile at all. `click` with `button: "left"` and `drag` still tap and swipe on mobile, but they are deprecated there, so send `tap` and `swipe`. A browser runner answers `tap`, `swipe` and `fill` with `action-not-supported-on-browser`. ### Several steps in one request: `runner actions` -`qawolf runner actions ''` performs up to ten of those same actions back to back in one request, for the steps you already know: click the field, type into it, press Enter. One round trip instead of three, with no delay to guess at between them, and `-` reads the array from stdin the way `act -` reads one action. +`qawolf runner actions ''` performs up to ten of the browser actions back to back in one request (not `tap`, `swipe` or `fill`), for the steps you already know: click the field, type into it, press Enter. One round trip instead of three, with no delay to guess at between them, and `-` reads the array from stdin the way `act -` reads one action. ```sh qawolf runner actions '[{"type":"click","button":"left","x":480,"y":260},{"type":"type","text":"me@example.com"},{"type":"keypress","keys":["Enter"]}]' --screenshot after-login.jpg diff --git a/src/commands/__snapshots__/help.test.ts.snap b/src/commands/__snapshots__/help.test.ts.snap index b048cfab5..53c306615 100644 --- a/src/commands/__snapshots__/help.test.ts.snap +++ b/src/commands/__snapshots__/help.test.ts.snap @@ -293,7 +293,7 @@ Commands: run [options] Run a flow on an interactive runner, shipping the flow and what it imports events [options] Print a runner's journal, one entry per line. QA Wolf writes console, recorder, run-events, run-logs, run-status screenshot [options] Save a JPEG of an interactive runner's screen to a file, or write it to stdout with --out - - act [options] Perform one raw action on a runner's screen: click, double_click, scroll, move, drag, keypress, navigate or type. Use - to read a whole action as JSON from stdin. On a mobile runner only click (button left), drag and type have a touchscreen equivalent; the rest answer action-not-supported-on-mobile + act [options] Perform one raw action on a runner's screen. A browser runner takes click, double_click, scroll, move, drag, keypress, navigate and type; a mobile runner takes tap, swipe, fill and type. Each family answers the other's actions with action-not-supported-on-mobile or action-not-supported-on-browser. Use - to read a whole action as JSON from stdin actions [options] Perform a sequence of up to ten raw actions on a runner's screen in one request, as a JSON array of the same actions \`runner act\` takes. Use - to read the array from stdin. Actions run back to back, so batch only steps whose targets are on the screen you last saw, and end the sequence at the step that changes the page. Each result carries an effect: performed, not-performed, or unknown when the runner stopped answering and the action may have landed exec [options] Evaluate a snippet against a runner's live page. Use - to read the snippet from stdin inspect Read one thing off a runner's live page (browser) or Appium session (mobile) @@ -553,31 +553,39 @@ Examples: exports[`--help output qawolf runner act 1`] = ` "Usage: qawolf runner act [options] -Perform one raw action on a runner's screen: click, double_click, scroll, move, -drag, keypress, navigate or type. Use - to read a whole action as JSON from -stdin. On a mobile runner only click (button left), drag and type have a -touchscreen equivalent; the rest answer action-not-supported-on-mobile +Perform one raw action on a runner's screen. A browser runner takes click, +double_click, scroll, move, drag, keypress, navigate and type; a mobile runner +takes tap, swipe, fill and type. Each family answers the other's actions with +action-not-supported-on-mobile or action-not-supported-on-browser. Use - to read +a whole action as JSON from stdin Options: - --button