Skip to content

fix(tool-server): press named key after text in keyboard tool#486

Open
j-piasecki wants to merge 2 commits into
mainfrom
@jpiasecki/keyboard-text-key-at-once
Open

fix(tool-server): press named key after text in keyboard tool#486
j-piasecki wants to merge 2 commits into
mainfrom
@jpiasecki/keyboard-text-key-at-once

Conversation

@j-piasecki

Copy link
Copy Markdown
Member

Problem

A keyboard call combining text and key pressed the named key before typing the text — the reverse of the intended "type, then submit". Every platform branch had the same ordering (copied verbatim from the original iOS implementation into each port), and nothing in the schema or description documented it. Ironically, run-sequence's own description lists "typing then pressing enter" as a canonical use case.

The observed failure: an agent ran { text: "jane@test.dev", key: "enter" } against a freshly-focused email field in a React Native dev build. Enter fired first into the still-empty field, submitting and blurring it; the text was then typed with nothing focused. RN's RCTKeyCommands matches Cmd-registered dev commands even with no modifier held when there is no first responder (_flags == flags || flags == 0 in matchesInput:flags:), so the bare d in ".dev" toggled the dev menu mid-sequence, and every subsequent tap landed on the dev-menu overlay.

Fix

  • All executing branches now type text first and press key after it:
    • simulator-server-keys.ts (iOS simulator / Android emulator)
    • keyboard/platforms/chromium.ts (Electron / CDP)
    • keyboard/platforms/vega.ts (Fire TV)
    • (the TV branch already rejects key; unchanged)
  • Unknown key names are still rejected before any text is typed — each branch resolves the named key up front instead of relying on the old key-first order for fail-fast. The Vega branch gained resolveVegaNamedKeycode in vega-input.ts so validation is separable from injection.
  • The ordering is now documented in the keyboard tool's key param description and tool description, and in run-sequence's args table. The run-sequence "type text and submit" example now shows the combined single-step form, since that is the pattern agents naturally reach for.

Verification

  • New regression suite test/keyboard-key-order.test.ts: 6 tests asserting key-after-text ordering and fail-fast on unknown keys across all three branches.
  • Full tool-server suite: 1995/1996 pass — the single failure (await-ui-element Android hidden edge case) fails identically on clean main, as do the two tsc errors from a stale @argent/telemetry build.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the keyboard tool’s combined { text, key } behavior so it types the text first and presses the named key afterward, aligning with the intended “type, then submit” workflow across simulator-server, Chromium (CDP), and Vega implementations. It also documents this ordering in tool/run-sequence help and adds regression tests to prevent reintroducing the prior key-first behavior.

Changes:

  • Reordered keyboard injection so { text, key } executes as text → key across simulator-server, Chromium, and Vega, while still failing fast on unknown key names.
  • Added resolveVegaNamedKeycode to validate Vega named keys independently from injection.
  • Documented the ordering in the keyboard tool and run-sequence, and added a regression test suite covering ordering + fail-fast behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
packages/tool-server/test/keyboard-key-order.test.ts Adds regression coverage for “text then key” ordering and fail-fast unknown-key validation across simulator-server, Chromium, and Vega.
packages/tool-server/src/utils/vega-input.ts Introduces resolveVegaNamedKeycode and refactors named-key injection to use it.
packages/tool-server/src/tools/run-sequence/index.ts Updates run-sequence documentation to state key-after-text semantics and shows the single-step “type and submit” example.
packages/tool-server/src/tools/keyboard/simulator-server-keys.ts Resolves named key up-front (fail-fast) and presses it after typing text.
packages/tool-server/src/tools/keyboard/platforms/vega.ts Validates named key before text injection and injects the key after text.
packages/tool-server/src/tools/keyboard/platforms/chromium.ts Resolves named key up-front (fail-fast) and dispatches it after typing text.
packages/tool-server/src/tools/keyboard/index.ts Documents the key-after-text behavior in schema descriptions and tool description.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants