Hold modifier keys on agent tap and double tap

COMPATIBILITY.md

@@ -114,7 +114,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen.
 - Parallel workers and serial suites
 - The full locator action set (`dragTo`, `swipe`, `hover`, `setInputFiles`, `secondaryTap`, `longPress`, pointer points). `secondaryTap` and `longPress` have no counterpart in the document engine
 - The screen-level `screen.tapAt`, `screen.swipe`, and `screen.scrollUntilVisible`
-- Per-action options other than `timeout` (and `delay` for `pressSequentially`): click `modifiers`, tap `position`, and the `{ label, value, index }` form of `selectOption`
+- Per-action options other than `timeout` (and `delay` for `pressSequentially`): click `modifiers` on a locator (the agent's `tap` and `double_tap` hold them, above), tap `position`, and the `{ label, value, index }` form of `selectOption`
 - Route patterns that ignore a record id. This port compares the URL path exactly and ignores the query and fragment
 - Diff-only observations. Each model turn receives a full text snapshot
 - Telemetry
@@ -275,14 +275,15 @@ Not ported:
 - As upstream, the judge prompt says a value shown in more than one place holds only when every place agrees, unless the statement names the place or is about some item among several. The wording follows upstream, with "instruction" as "statement" and the `"fails"` verdict as "false", because the port's judge answers through `done`
 - `ExtractAsync<T>`: the type argument is the schema, in place of a Standard Schema. The judge gets the JSON schema of `T` (`JsonSchemaExporter`), and the answer must deserialize into `T` with required members and nullable annotations respected. A failure gets one repair round with the validation error, then `MODEL_OUTPUT_INVALID` (`extracted data failed schema validation: ...`). A judge that says the data is not shown ends `ASSERTION_INCONCLUSIVE` (`nothing to extract: ...`)
 - `tools` (`AgentOptions.Tools`, `AgentTools` on the default agent) are project tools the model is offered beside the agent's own. As upstream's `defineTool`, an `AgentTool` declares `mutates` and `platforms`: a tool is offered only on the platforms it names, and a mutating tool takes an action slot. Its output is redacted and cut at 16384 characters. A name the agent already uses (`observe`, `done`, `navigate`, `back`, `scroll`, `scroll_to`, or a verb) is refused with `INVALID_CONFIG`. Differences: the tool is an `AgentTool` built with its schema as a `JsonObject` and a delegate, since the port has no `defineTool`, AI SDK tool, or `toModelOutput`; a tool that wants upstream's `getToolContext` takes the second constructor, whose delegate gets an `AgentToolContext` (`ObserveAsync`, `AttachScreenshotAsync`), and the first constructor keeps working; and upstream's reserved names for tools it adds elsewhere (`complete_step`, `locate`, the recording tools, `report_finding`) have no counterpart. Upstream records a mutating tool as a gap in the trace, and so does the port: a recording holds an action of kind `tool` with the tool name, a replay runs the actions before it and then hands the step to the model, and the step is never replayed whole. Older versions of this port cannot read that action and run the step live. As upstream, a recording that stops at such a gap (reason `gap`) is not stale: a strict cache run replays what it can and runs the rest of the step live. The recording keeps the tool's name only, never its arguments
+- `tap` and `double_tap` take an optional `modifiers` list (`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) on an engine that declares `EngineCapabilities.TapModifiers`, as upstream's `tap`, `double_tap`, and `right_click` do on an engine that declares `tapModifiers`. The web engine declares it. The device engines and the document engine do not, and refuse the field with `UNSUPPORTED_CAPABILITY`. The held keys are recorded as `modifiers` of the `tap` or `doubleTap` action and replayed from the cache. An entry whose `modifiers` is empty, repeats a key, or sits on another action is a miss (`invalid-entry`). The tools do not close their schemas, so a tool called with a field its engine does not declare is refused by the action, not by the schema. There is no `right_click`, because no engine here has a secondary tap.
 - `act` also offers a `double_tap` tool, which upstream does not have. Without it, the agent cannot open a control that reacts only to a double-click, such as a TodoMVC label that opens its editor. It runs the locator `doubleTap`, and the replay cache records it as `doubleTap`. Upstream and older versions of this port cannot replay that entry, so they run the step live
 - `act` params are checked as upstream: JSON-safe values, at most 32 levels deep, no cycle, and at most 64 KiB once a `Secret` is projected to its name and purpose and a `Values.Unique` to its value. The instruction is at most 8192 UTF-8 bytes. Each limit throws `INVALID_ARGUMENT` with the upstream message before any model call. A value reached twice through different paths is not a cycle
 - `Vision` on `AssertAsync`, `WaitForAsync`, and `ExtractAsync` is a `VisionMode` (`Off`, `On`, `Only`) in place of upstream's `false`, `true`, and `'only'`, and an undefined value throws `INVALID_ARGUMENT`. `On` sends a masked screenshot beside the tree, and `Only` sends the screenshot alone, with a note that no tree is attached on purpose. The screenshot is sized for the model (long side at most 768 pixels) and its note names the regions masked for security. A secret filled in the attempt (by a locator or by the agent), an engine that captures no pixels, or masking the engine cannot prove (fewer masked regions than secure nodes in the tree, or no count at all) takes the image away and the judgment goes on with the tree; `Only` has no tree to go on with, so it fails with `POLICY_DENIED` that names `PIXEL_TAINTED`, `MASKING_UNPROVEN`, or `UNSUPPORTED_CAPABILITY`, before any model call. The judge's system text says an attached screenshot is data, as upstream's policy does. `WaitForAsync` takes a screenshot with each look and counts a changed picture as a changed screen. An inconclusive judgment made from the tree alone ends with the hint to pass `Vision = VisionMode.On`, firm once a judgment of the attempt has received pixels and conditional before ("if the engine captures pixels"), and none when pixels cannot help: a secret was filled, the engine declares no `Screenshot` capability, or an earlier request for pixels was degraded. The hint's words name the C# option
 - `AssertOptions.Screenshot` (on by default) keeps the redacted viewport after the verdict, a passing one too, under `.e2e/results/<test>/attempt-<n>/screenshots/<n>-assert.png`, and `AgentException.Screenshot` names it on a failed judgment as a path relative to the project root, where upstream names it relative to the artifacts directory. A secret filled in the attempt, or an engine that captures no pixels, keeps none. Taking it never fails the judgment. The port has no step record, so it has no `visionInput`, `visionDegraded`, `visionOnly`, or `pixelBytes` on a step, no `vision.pixels` or `assert.screenshot` policy events, and no observation revision in a pixels-only request
 
 - `executor` (`AgentOptions.Executor`, `AgentExecutor` on `E2ESessionOptions` and the fixtures) replaces the built-in agent for `act` and `assert`. The socket is `IStepExecutor`: `Name`, and `Version`, `Cache`, `Model`, and `Judge` with defaults, so a hand-rolled executor writes `Name` and `RunStepAsync(context, cancellationToken)`. The token is upstream's `signal`; the context's `Attempt.CancellationToken` is upstream's `attempt.signal`. `StepExecutorContext` is a class a test can build and override: `Step`, `Attempt`, `Target`, `ReplayedPrefix`, `Model`, `ProviderOptions`, `Ledger`, `AgentContext`, `Budgets`, `Actions`, `PixelsTainted`, `ObserveAsync`, `AttachTranscript`, and `AttachScreenshotAsync`. An executor is code, so `agents.<name>.executor` in `e2e.config.json` stays `INVALID_CONFIG` and names `AgentOptions.Executor`. As upstream, `System` and `Tools` are refused beside an executor, and the `Model` and `Judge` an executor brought are the agent's, for the judgments too. An entry that names another is `INVALID_CONFIG` (`agents.default.model (gpt-5.4) and the executor's own model (gpt-5.4-mini) differ; configure the model in one place`). Two models are the same when they are one instance or share a `Name`, where upstream compares the provider and the id
 - `ToolLoop.CreateToolLoopExecutor` and `DefaultAgent.CreateBuiltInAgent` are public. Differences: `ToolLoopExecutorOptions.Tools` returns a list of `ToolLoopTool` (a `ModelTool` and a delegate that answers a `ToolLoopResult`), where upstream's returns an AI SDK `ToolSet`; `BuildPrompt` answers a string, where upstream's can answer a whole message history; the chassis adds `done`, where upstream adds `complete_step`; there is no `maxTurns` to remove. `ToolLoopHelpers` has `GuardAsync`, `Concluding`, and `ReportHardStop`, and the loop guards every tool call itself. The built-in agent drives the context the runtime builds, so it runs as a step's executor or inside a custom executor that passes its context on, and not on a context built by hand
-- The verbs are the ones the port's engines have: `TapAsync`, `DoubleTapAsync`, `TypeAsync`, `TypeSecretAsync`, `ClearAsync` (upstream clears with `type`), `PressAsync`, `SelectAsync`, `CheckAsync`, `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, and `BackAsync`. Not ported, because no engine here has them: long press, secondary tap, hover, drag, upload, the point verbs (`tapAt`, `hoverAt`, `hitTest`), `typeText`, `pressKey`, and `dismissKeyboard`. `Target.Verbs` follows the engine's capabilities: the scroll verbs need `Scroll`, `Back` needs `History`. A node is named by its ref (`new ExecutorTarget("e12")`, the `[ref=e12]` of a line), where upstream names `n42`. Actions and looks run in call order, mutating project tools included, and a mutating `RunToolAsync` body that calls an action or a look waits behind itself, as upstream's does. A verb checks its argument before it takes a slot (`INVALID_ARGUMENT`), but the port has no key grammar, so `PressAsync` takes any non-blank key
+- The verbs are the ones the port's engines have: `TapAsync` and `DoubleTapAsync` (both take `ClickOptions`, whose `Modifiers` need `Target.TapModifiers`, as upstream's `tap(target, { modifiers })` needs `target.tapModifiers`), `TypeAsync`, `TypeSecretAsync`, `ClearAsync` (upstream clears with `type`), `PressAsync`, `SelectAsync`, `CheckAsync`, `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, and `BackAsync`. Not ported, because no engine here has them: long press, secondary tap, hover, drag, upload, the point verbs (`tapAt`, `hoverAt`, `hitTest`), `typeText`, `pressKey`, and `dismissKeyboard`. `Target.Verbs` follows the engine's capabilities: the scroll verbs need `Scroll`, `Back` needs `History`. A node is named by its ref (`new ExecutorTarget("e12")`, the `[ref=e12]` of a line), where upstream names `n42`. Actions and looks run in call order, mutating project tools included, and a mutating `RunToolAsync` body that calls an action or a look waits behind itself, as upstream's does. A verb checks its argument before it takes a slot (`INVALID_ARGUMENT`), but the port has no key grammar, so `PressAsync` takes any non-blank key
 - `ObserveAsync` answers `Text`, `Truncated`, and `Path`, and with `ExecutorObserveOptions` `Tree` (a list of root `ExecutorNode`s, redacted, with no value or selection on a secure field, a link target absolute and cut as a line cuts it) and `Pixels` (the masked screenshot sized for the model, or `PixelsWithheld`, a `VisionDegradation`). It has no `Revision`, no `Viewport`, and no `framePath`, since the port has none. `ReplayedPrefix` carries the actions as the text the hand-off notice lists, `ReplayHandOffReason` adds the port's `ActionBudget` and `InvalidEntry`, and has no `viewport-changed`. A replay that meets `ACTION_MAY_HAVE_COMMITTED` stops as `action-uncertain` and names the action, which the built-in agent's notice repeats
 - The runtime owns the budgets and the clock. `RecordModelCall` throws `STEP_BUDGET_EXHAUSTED` past `MaxModelCalls`, and a verdict never outranks a hard stop: a passed one over a spent budget, a timeout, or a code-less failure after one ends with the runtime's code. An executor that ignores its token gets one second to unwind after the step is stopped, and then the step settles without it. The closed verdict grammar is upstream's: an empty summary, an unknown code, a passed verdict with a code, or a blocked one without a blockable code is `MODEL_OUTPUT_INVALID`, and so is a runtime code the runtime did not record. A thrown error that is no agent error becomes `ACTION_FAILED`, as upstream's `toAgentError` does, and a cancel by the caller stays an `OperationCanceledException`. The message of a failed or blocked act is the executor's summary alone, where upstream's reads `agent.act failed: <summary>`
 - `AssertAsync` on an agent with an executor goes to it as a step of kind `Assert`, with the agent's budgets and `JudgmentTimeout`. An assert is never cached, and a passing one verifies the acts before it. `Vision` is refused with `UNSUPPORTED_CAPABILITY`, since the executor decides what its model sees, and `Screenshot` keeps its evidence after the verdict, bounded by the action timeout: a capture that outlives it is dropped and the verdict stands. `WaitForAsync` and `ExtractAsync` keep the judge, which is the executor's `Judge` or `Model`. `ExecutorCache.Off` never replays or records, and takes no look before the step. A step with the cache on that cannot read its first screen (no app opened) still runs, and records nothing

skills/e2e-dotnet/references/agent.md

@@ -99,6 +99,10 @@ The tools the model can call: `observe`, `tap`, `double_tap`, `fill`,
 `back` with one that has history. There is no `hover`, `drag`, `upload`,
 `long_press`, `right_click`, `screenshot`, or point tool: a step that needs
 one is `AUTOMATION_UNSUPPORTED`. Do that part with `Screen` actions instead.
+On an engine that declares `EngineCapabilities.TapModifiers` (the web engine),
+`tap` and `double_tap` take an optional `modifiers` list (`Shift`, `Control`,
+`Alt`, `Meta`, `ControlOrMeta`) to hold keys for the click, as a Shift-click
+extends a selection. The held keys are recorded and replayed from the cache.
 A project tool adds its own name to this list (see "Project tools").
 `fill_secret` fills any editable field, a rich-text editor included, with a
 generic secret; a password secret needs a password field.
@@ -472,8 +476,8 @@ What the context holds:
 | --- | --- |
 | `Step` | The `Kind`, `Instruction`, `Params`, `Index`, and `Secrets`. A secret value the attempt knows reads `<secret:name>`, and a `Secret` in the params is its name and purpose, never its value |
 | `ObserveAsync(options)` | Redacted screen text, and with `Tree` the node tree (`ExecutorNode`, whose `Id` is the ref) and with `Pixels` a masked screenshot, or `PixelsWithheld` with the reason |
-| `Actions` | Checked and recorded verbs: `TapAsync`, `DoubleTapAsync`, `TypeAsync`, `TypeSecretAsync`, `ClearAsync`, `PressAsync`, `SelectAsync`, `CheckAsync`, `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, `BackAsync`. A node is `new ExecutorTarget(ref)` |
-| `Target` | The target's `Name`, `Platform`, and `Verbs`, the verbs its engine supports. Offer a model only these |
+| `Actions` | Checked and recorded verbs: `TapAsync` and `DoubleTapAsync` (both take `new ClickOptions { Modifiers = [KeyModifier.Shift] }` when `Target.TapModifiers`), `TypeAsync`, `TypeSecretAsync`, `ClearAsync`, `PressAsync`, `SelectAsync`, `CheckAsync`, `ScrollToAsync`, `ScrollUntilAsync`, `ScrollAsync`, `NavigateAsync`, `BackAsync`. A node is `new ExecutorTarget(ref)` |
+| `Target` | The target's `Name`, `Platform`, and `Verbs`, the verbs its engine supports, and `TapModifiers`, true when `TapAsync` and `DoubleTapAsync` take `ClickOptions.Modifiers`. Offer a model only these |
 | `PixelsTainted` | True after a secret fill. Then no screenshots are available |
 | `ReplayedPrefix` | Set when a cache replay ran part of the step first. The actions and why it stopped. Continue from the current screen without repeating them |
 | `Model`, `ProviderOptions`, `AgentContext`, `Ledger` | The agent's model and provider options, its project context, and the steps completed before this one |

skills/e2e-dotnet/references/mcp.md

@@ -84,7 +84,7 @@ config declares a `secrets` entry.
 | Tool | Does |
 | --- | --- |
 | `observe` | The whole current screen |
-| `tap`, `double_tap`, `check`, `uncheck`, `clear` `{target}` | Act on one node |
+| `tap`, `double_tap`, `check`, `uncheck`, `clear` `{target}` | Act on one node. On an engine that declares `TapModifiers` (the web engine), `tap` and `double_tap` also take an optional `modifiers` list (`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) to hold keys for the click, as a Shift-click extends a selection |
 | `type {target, text}` | Replace the text in a field |
 | `type_secret {target, secret}` | Fill a configured secret into an editable field by name; you never see the value |
 | `press {key, target?}` | Press a key on a node, or on whatever has focus |

src/E2E.Cli/Mcp/SessionCatalog.cs

@@ -61,8 +61,9 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities
         void Add(string name, CatalogTool tool) => tools.Add(new(name, tool));
 
         Add("observe", Observe(context));
-        Add("tap", Verb(context, "tap", "tap", "Tap or click one node: the gesture for a button, link, menu item, tab, checkbox, row, or field.", Schema(Target)));
-        Add("double_tap", Verb(context, "double_tap", "double_tap", "Double-click one node. Use it only when a tap does nothing, such as a label that opens an editor on a double-click.", Schema(Target)));
+        var holdable = capabilities.HasFlag(EngineCapabilities.TapModifiers) ? new (string Name, JsonNode Schema, bool Required)[] { ("modifiers", AgentTools.ModifiersSchema(), false) } : [];
+        Add("tap", Verb(context, "tap", "tap", "Tap or click one node: the gesture for a button, link, menu item, tab, checkbox, row, or field.", Schema(Target, holdable)));
+        Add("double_tap", Verb(context, "double_tap", "double_tap", "Double-click one node. Use it only when a tap does nothing, such as a label that opens an editor on a double-click.", Schema(Target, holdable)));
         Add("type", Verb(context, "type", "fill", "Replace the text in a field with text.", Schema(Target, ("text", Property("string", "The text the field should hold"), true)), ("text", "value")));
         Add("press", Verb(context, "press", "press", "Press a key such as Enter, Escape, or Tab. With a target, the key goes to that node; without one, to whatever has focus.", Schema(OptionalTarget, ("key", Property("string", "The key, such as Enter"), true))));
         Add("select", Verb(context, "select", "select", "Choose an option of a combobox or select, by its value or label.", Schema(Target, ("value", Property("string", "The option's value or label"), true))));

src/E2E/Agent/ActContext.cs

@@ -66,7 +66,7 @@ public ActContext(AttemptScope scope, ObservationFeed feed, StepAccounting budge
             CancellationToken = scope.Ended.Token,
             Memory = scope.ExecutorMemory,
         };
-        Target = new ExecutorTargetInfo { Name = scope.TargetName, Platform = scope.EnginePlatform, Verbs = VerbsOf(scope.EngineCapabilities) };
+        Target = new ExecutorTargetInfo { Name = scope.TargetName, Platform = scope.EnginePlatform, Verbs = VerbsOf(scope.EngineCapabilities), TapModifiers = scope.EngineCapabilities.HasFlag(EngineCapabilities.TapModifiers) };
         ReplayedPrefix = prefix;
         Model = agent.Model;
         ProviderOptions = agent.ProviderOptions;