Native UI control driver (object-level desktop automation)#214
Merged
Conversation
The accessibility layer could only list/find/click elements. Add control-pattern actions so it can read and drive native controls by name/role/app/AutomationId — far more reliable than pixel/OCR for native apps and the #1 gap vs object-aware desktop tools (pywinauto / FlaUI / WinAppDriver / UiPath), surfaced by web research of competitors and practitioner pain points. - Backend interface: get_value / set_value / invoke / toggle, with a clear AccessibilityNotAvailableError for backends that can't act. - Windows UIAutomation backend implements all four via the Value / Invoke / Toggle control patterns (comtypes), degrading gracefully on COM errors. - Facade re-exports control_get_value / control_set_value / control_invoke / control_toggle; AC_control_* executor commands; ac_control_* MCP tools; Script Builder "Native UI" entries. - Headless tests inject a fake backend (no GUI/comtypes needed) and verify the API, executor, MCP wiring and graceful degradation. - v7 reference page (EN + Traditional Chinese) + README sections.
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 88 |
| Duplication | 2 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Complete the native control driver with table reading — the agent research's other top desktop gap (data scraping). Adds read_table to the backend interface, a Windows UIAutomation Grid-pattern implementation, read_control_table in the facade, the AC_read_table executor command, the ac_read_table MCP tool, a Script Builder entry, fake-backend tests, and the v7 docs / README sections.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Found via the multi-agent web research ("再想想"): the accessibility layer
could only list / find / click elements — it could not read or set a
control's value, invoke it, or toggle it. That object-level control is the
core of every object-aware desktop tool (pywinauto / FlaUI / WinAppDriver /
UiPath) and was AutoControl's single biggest gap vs them (cross-validated by
competitor analysis and practitioner pain-point research). It's far more
reliable than pixel/OCR/VLM for native apps.
What's added (full five-layer wiring)
get_value/set_value/invoke/toggle,with a clear
AccessibilityNotAvailableErrorfor backends that can't act.Invoke / Toggle control patterns (comtypes), degrading gracefully on COM
errors; targets a control by
name/role/app_name/automation_id(the stable Windows identifier).
control_get_value/control_set_value/control_invoke/control_toggle.AC_control_get_value/AC_control_set_value/AC_control_invoke/AC_control_toggle.ac_control_*. Script Builder: "Native UI" entries.Testing
executor commands, MCP registry and graceful degradation on the null
backend. The Windows UIA path is platform code (validated like the rest of
the backend, which CI can't exercise headlessly).
ruff/radon/banditclean;import je_auto_controlstays Qt-free;no orphan Script Builder commands.