Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-19) — Unattended Reliability](#whats-new-2026-06-19--unattended-reliability)
- [What's new (2026-06-19) — Popup Watchdog](#whats-new-2026-06-19--popup-watchdog)
- [What's new (2026-06-19) — Native UI Control](#whats-new-2026-06-19--native-ui-control)
- [What's new (2026-06-19)](#whats-new-2026-06-19)
Expand Down Expand Up @@ -61,6 +62,14 @@

---

## What's new (2026-06-19) — Unattended Reliability

Three practitioner-pain fixes for unattended / login automation, all headless and full-stack. Full reference: [`docs/source/Eng/doc/new_features/v9_features_doc.rst`](docs/source/Eng/doc/new_features/v9_features_doc.rst).

- **OTP / TOTP for 2FA** — `generate_totp` / `verify_totp` (`AC_otp_to_var`, `ac_generate_otp`): mint the current 6-digit code from a base32 secret to type into a login form (reuses the remote-desktop TOTP engine).
- **Native file dialogs** — `handle_file_dialog` (`AC_handle_file_dialog`): wait for the OS Open/Save/folder dialog, type the path, confirm — in one call, with an injectable driver.
- **Locked-session guard** — `ensure_interactive_session` / `is_session_locked` (`AC_assert_session_active`): fail clearly when the workstation is locked / disconnected instead of emitting phantom clicks.

## What's new (2026-06-19) — Popup Watchdog

The #1 cause of unattended-automation failure is an unexpected dialog the script never coded for (UAC, "session expiring", Windows Update, a modal). The popup watchdog runs a concurrent guard thread that watches for registered patterns and dismisses them independently of the main flow. Surfaced by the practitioner pain-point research as the top unattended failure cause; full stack (facade, `AC_*`, MCP, Script Builder), fully headless. Full reference: [`docs/source/Eng/doc/new_features/v8_features_doc.rst`](docs/source/Eng/doc/new_features/v8_features_doc.rst).
Expand Down
9 changes: 9 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-19) — 无人值守可靠性](#本次更新-2026-06-19--无人值守可靠性)
- [本次更新 (2026-06-19) — 弹窗看门狗](#本次更新-2026-06-19--弹窗看门狗)
- [本次更新 (2026-06-19) — 原生 UI 控制](#本次更新-2026-06-19--原生-ui-控制)
- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
Expand Down Expand Up @@ -60,6 +61,14 @@

---

## 本次更新 (2026-06-19) — 无人值守可靠性

三个无人值守/登录自动化的社区痛点修复,均 headless 且走完整五层。完整参考:[`docs/source/Eng/doc/new_features/v9_features_doc.rst`](../docs/source/Eng/doc/new_features/v9_features_doc.rst)。

- **2FA 的 OTP / TOTP** — `generate_totp` / `verify_totp`(`AC_otp_to_var`、`ac_generate_otp`):从 base32 secret 生成当下 6 位码,填进登录表单(重用远程桌面 TOTP 引擎)。
- **原生文件对话框** — `handle_file_dialog`(`AC_handle_file_dialog`):等 OS 打开/保存/文件夹对话框、输入路径、确认,一次完成,driver 可注入。
- **锁定会话守卫** — `ensure_interactive_session` / `is_session_locked`(`AC_assert_session_active`):工作站锁定/断开时清楚失败,而非发出幽灵点击。

## 本次更新 (2026-06-19) — 弹窗看门狗

无人值守自动化失败的第一大主因,是脚本没写到的意外对话框(UAC、"会话过期"、Windows Update、modal)。弹窗看门狗以并行守卫线程监看注册 pattern,独立于主流程把它们关掉。由社区痛点研究指出为无人值守头号失败主因;走完整五层(facade、`AC_*`、MCP、Script Builder),完全 headless。完整参考:[`docs/source/Eng/doc/new_features/v8_features_doc.rst`](../docs/source/Eng/doc/new_features/v8_features_doc.rst)。
Expand Down
9 changes: 9 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-19) — 無人值守可靠性](#本次更新-2026-06-19--無人值守可靠性)
- [本次更新 (2026-06-19) — 彈窗看門狗](#本次更新-2026-06-19--彈窗看門狗)
- [本次更新 (2026-06-19) — 原生 UI 控制](#本次更新-2026-06-19--原生-ui-控制)
- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
Expand Down Expand Up @@ -60,6 +61,14 @@

---

## 本次更新 (2026-06-19) — 無人值守可靠性

三個無人值守/登入自動化的社群痛點修復,皆 headless 且走完整五層。完整參考:[`docs/source/Zh/doc/new_features/v9_features_doc.rst`](../docs/source/Zh/doc/new_features/v9_features_doc.rst)。

- **2FA 的 OTP / TOTP** — `generate_totp` / `verify_totp`(`AC_otp_to_var`、`ac_generate_otp`):從 base32 secret 產生當下 6 碼,填進登入表單(重用遠端桌面 TOTP 引擎)。
- **原生檔案對話框** — `handle_file_dialog`(`AC_handle_file_dialog`):等 OS 開啟/儲存/資料夾對話框、輸入路徑、確認,一次完成,driver 可注入。
- **鎖定工作階段守衛** — `ensure_interactive_session` / `is_session_locked`(`AC_assert_session_active`):工作站鎖定/斷線時清楚失敗,而非送出幽靈點擊。

## 本次更新 (2026-06-19) — 彈窗看門狗

無人值守自動化失敗的第一大主因,是腳本沒寫到的未預期對話框(UAC、「工作階段過期」、Windows Update、modal)。彈窗看門狗以並行守衛執行緒監看註冊 pattern,獨立於主流程把它們關掉。由社群痛點研究指出為無人值守頭號失敗主因;走完整五層(facade、`AC_*`、MCP、Script Builder),完全 headless。完整參考:[`docs/source/Zh/doc/new_features/v8_features_doc.rst`](../docs/source/Zh/doc/new_features/v8_features_doc.rst)。
Expand Down
70 changes: 70 additions & 0 deletions docs/source/Eng/doc/new_features/v9_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
===================================================
New Features (2026-06-19) — Unattended Reliability
===================================================

Three practitioner-pain fixes for unattended and login automation: mint
2FA codes, drive native file dialogs, and refuse to act on a locked
screen. Each ships through the full stack (facade, ``AC_*`` executor
commands, MCP tools, Script Builder) and is fully headless — external
steps are deterministic or injectable, so they unit-test without 2FA, a
real dialog, or a locked session.

.. contents::
:local:
:depth: 2


OTP / TOTP for 2FA logins
=========================

2FA blocks automated logins. Store the base32 secret (ideally in the
secrets store) and mint the current code mid-flow::

from je_auto_control import generate_totp, verify_totp

code = generate_totp(secret) # 6-digit TOTP for "now"
type_text(code)

``AC_otp_to_var`` writes the code into a flow variable for the next step::

["AC_otp_to_var", {"secret": "JBSWY3DPEHPK3PXP", "var": "otp"}]
["AC_type_keyboard", {"keycode": "${otp}"}]

Reuses the TOTP engine that backs remote-desktop auth. Executor command:
``AC_otp_to_var``; MCP tool: ``ac_generate_otp``.


Native file dialogs
===================

Recorders don't capture the OS file Open/Save/folder dialog, so everyone
hand-rolls "type the path + Enter". ``handle_file_dialog`` does it in one
call::

from je_auto_control import handle_file_dialog

handle_file_dialog("C:/reports/out.csv", action="save")

``action`` is ``open`` / ``save`` / ``folder`` (picking a default dialog
title) or pass an explicit ``window_title``; it waits for the dialog,
types the path, and presses ``confirm_key`` (default Enter). The
window-wait / type / confirm steps go through an injectable
:class:`FileDialogDriver`. Executor command: ``AC_handle_file_dialog``.


Locked-session guard
===================

Unattended runs silently fail when the workstation is locked or the RDP
session is disconnected — input no-ops or throws. Check first::

from je_auto_control import ensure_interactive_session, is_session_locked

ensure_interactive_session() # raises if locked
if is_session_locked():
...

On Windows the probe opens the input desktop (which fails when locked);
other platforms report "not locked" unless a custom probe is supplied.
Executor command: ``AC_assert_session_active`` — put it at the top of an
unattended script so it fails clearly instead of emitting phantom clicks.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v6_features_doc
doc/new_features/v7_features_doc
doc/new_features/v8_features_doc
doc/new_features/v9_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
66 changes: 66 additions & 0 deletions docs/source/Zh/doc/new_features/v9_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
========================================
新功能 (2026-06-19) — 無人值守可靠性
========================================

三個針對無人值守與登入自動化的社群痛點修復:產生 2FA 驗證碼、操作原生
檔案對話框、以及拒絕在鎖定畫面上動作。每項都走完整五層(facade、
``AC_*`` 執行器指令、MCP 工具、Script Builder),且完全 headless——外部
步驟皆為決定性或可注入,因此不需 2FA、真實對話框或鎖定工作階段即可單元
測試。

.. contents::
:local:
:depth: 2


2FA 登入的 OTP / TOTP
=====================

2FA 會擋住自動登入。把 base32 secret 存好(最好放進 secrets store),在
流程中即時產生當前驗證碼::

from je_auto_control import generate_totp, verify_totp

code = generate_totp(secret) # 當下的 6 碼 TOTP
type_text(code)

``AC_otp_to_var`` 會把驗證碼寫進流程變數供下一步使用::

["AC_otp_to_var", {"secret": "JBSWY3DPEHPK3PXP", "var": "otp"}]
["AC_type_keyboard", {"keycode": "${otp}"}]

重用支撐遠端桌面驗證的 TOTP 引擎。執行器指令:``AC_otp_to_var``;
MCP 工具:``ac_generate_otp``。


原生檔案對話框
==============

錄製器抓不到 OS 的檔案開啟/儲存/資料夾對話框,大家只好手刻「輸入路徑 +
Enter」。``handle_file_dialog`` 一次搞定::

from je_auto_control import handle_file_dialog

handle_file_dialog("C:/reports/out.csv", action="save")

``action`` 為 ``open`` / ``save`` / ``folder``(自動選預設對話框標題),
或傳明確的 ``window_title``;它會等對話框、輸入路徑、按 ``confirm_key``
(預設 Enter)。等視窗/輸入/確認三步透過可注入的 :class:`FileDialogDriver`。
執行器指令:``AC_handle_file_dialog``。


鎖定工作階段守衛
================

無人值守在工作站鎖定或 RDP 斷線時會默默失敗——輸入會 no-op 或拋例外。
先檢查::

from je_auto_control import ensure_interactive_session, is_session_locked

ensure_interactive_session() # 鎖定時拋例外
if is_session_locked():
...

Windows 上以開啟 input desktop 偵測(鎖定時會失敗);其他平台除非提供自訂
probe,否則回報「未鎖定」。執行器指令:``AC_assert_session_active``——放在
無人值守腳本最前面,讓它清楚地失敗,而非送出幽靈點擊。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v6_features_doc
doc/new_features/v7_features_doc
doc/new_features/v8_features_doc
doc/new_features/v9_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
15 changes: 15 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,18 @@
from je_auto_control.utils.hotkey.hotkey_daemon import (
HotkeyBinding, HotkeyDaemon, default_hotkey_daemon,
)
# OTP/TOTP for automated 2FA logins
from je_auto_control.utils.otp import (
TOTPError, generate_secret, generate_totp, verify_totp,
)
# Native file Open/Save/folder dialog helper
from je_auto_control.utils.file_dialog import (
FileDialogDriver, handle_file_dialog,
)
# Locked / non-interactive session guard
from je_auto_control.utils.session_guard import (
ensure_interactive_session, is_session_locked,
)
# Background popup/interrupt watchdog (unattended automation)
from je_auto_control.utils.watchdog import (
PopupWatchdog, WatchdogRule, default_popup_watchdog,
Expand Down Expand Up @@ -494,6 +506,9 @@ def start_autocontrol_gui(*args, **kwargs):
# Hotkey daemon
"HotkeyDaemon", "HotkeyBinding", "default_hotkey_daemon",
"PopupWatchdog", "WatchdogRule", "default_popup_watchdog",
"generate_totp", "verify_totp", "generate_secret", "TOTPError",
"handle_file_dialog", "FileDialogDriver",
"ensure_interactive_session", "is_session_locked",
# MCP server
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",
Expand Down
29 changes: 29 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -577,7 +577,7 @@
FieldSpec("automation_id", FieldType.STRING, optional=True),
)
specs.append(CommandSpec(
"AC_control_get_value", "Native UI", "Get Control Value",

Check failure on line 580 in je_auto_control/gui/script_builder/command_schema.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Define a constant instead of duplicating this literal "Native UI" 6 times.

See more on https://sonarcloud.io/project/issues?id=Integration-Automation_AutoControlGUI&issues=AZ7c0nRiL49KgC-owA80&open=AZ7c0nRiL49KgC-owA80&pullRequest=217
fields=fields,
description="Read a native control's value via the accessibility API.",
))
Expand Down Expand Up @@ -623,6 +623,35 @@
"AC_watchdog_stop", "Flow", "Watchdog: Stop"))
specs.append(CommandSpec(
"AC_watchdog_list", "Flow", "Watchdog: List Rules / Hits"))
specs.append(CommandSpec(
"AC_otp_to_var", "Flow", "OTP (TOTP) into Variable",
fields=(
FieldSpec("secret", FieldType.STRING),
FieldSpec("var", FieldType.STRING, default="otp"),
FieldSpec("digits", FieldType.INT, optional=True, default=6),
FieldSpec("step", FieldType.INT, optional=True, default=30),
),
description="Generate a TOTP 2FA code from a base32 secret.",
))
specs.append(CommandSpec(
"AC_handle_file_dialog", "Native UI", "Handle File Dialog",
fields=(
FieldSpec("path", FieldType.STRING),
FieldSpec("action", FieldType.ENUM,
choices=("open", "save", "folder"),
optional=True, default="open"),
FieldSpec("window_title", FieldType.STRING, optional=True),
FieldSpec("timeout_s", FieldType.FLOAT, optional=True,
default=10.0),
FieldSpec("confirm_key", FieldType.STRING, optional=True,
default="enter"),
),
description="Wait for a native file dialog, type a path, confirm.",
))
specs.append(CommandSpec(
"AC_assert_session_active", "Flow", "Assert Session Active",
description="Fail if the session is locked / non-interactive.",
))
specs.append(CommandSpec(
"AC_shell_command", "Shell", "Shell Command",
fields=(FieldSpec("shell_command", FieldType.STRING),),
Expand Down
19 changes: 19 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -2289,6 +2289,23 @@ def _watchdog_list() -> Dict[str, Any]:
return {"running": w.running, "rules": w.rule_names(), "hits": w.hits}


def _handle_file_dialog(path: str, action: str = "open",
window_title: Optional[str] = None,
timeout_s: float = 10.0,
confirm_key: str = "enter") -> Dict[str, Any]:
"""Adapter: wait for a native file dialog, type the path, confirm."""
from je_auto_control.utils.file_dialog import handle_file_dialog
return handle_file_dialog(path, action=action, window_title=window_title,
timeout_s=float(timeout_s),
confirm_key=confirm_key)


def _assert_session_active() -> Dict[str, Any]:
"""Adapter: raise unless the session is interactive (not locked)."""
from je_auto_control.utils.session_guard import ensure_interactive_session
return {"interactive": ensure_interactive_session()}


class Executor:
"""
Executor
Expand Down Expand Up @@ -2445,6 +2462,8 @@ def __init__(self):
"AC_watchdog_start": _watchdog_start,
"AC_watchdog_stop": _watchdog_stop,
"AC_watchdog_list": _watchdog_list,
"AC_handle_file_dialog": _handle_file_dialog,
"AC_assert_session_active": _assert_session_active,
"AC_a11y_record_start": _a11y_record_start,
"AC_a11y_record_stop": _a11y_record_stop,
"AC_a11y_record_events": _a11y_record_events,
Expand Down
11 changes: 11 additions & 0 deletions je_auto_control/utils/executor/flow_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,16 @@ def exec_pdf_to_var(executor: Any, args: Mapping[str, Any]) -> Dict[str, Any]:
return {"var": var_name, "length": len(text)}


def exec_otp_to_var(executor: Any, args: Mapping[str, Any]) -> Dict[str, Any]:
"""Generate a TOTP code from a base32 secret into a flow variable (2FA)."""
from je_auto_control.utils.otp import generate_totp
code = generate_totp(args["secret"], step=int(args.get("step", 30)),
digits=int(args.get("digits", 6)))
var_name = args.get("var", "otp")
executor.variables.set(var_name, code)
return {"var": var_name}


def exec_sql_to_var(executor: Any, args: Mapping[str, Any]) -> Dict[str, Any]:
"""Run a read-only SQLite query and store its result in a flow variable."""
from je_auto_control.utils.sql.sql_query import query_sqlite
Expand Down Expand Up @@ -670,6 +680,7 @@ def exec_call_macro(executor: Any, args: Mapping[str, Any]) -> Any:
"AC_shell_to_var": exec_shell_to_var,
"AC_read_file_to_var": exec_read_file_to_var,
"AC_pdf_to_var": exec_pdf_to_var,
"AC_otp_to_var": exec_otp_to_var,
"AC_sql_to_var": exec_sql_to_var,
"AC_assert_db": exec_assert_db,
"AC_http_to_var": exec_http_to_var,
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/file_dialog/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Drive native file Open/Save/folder-picker dialogs."""
from je_auto_control.utils.file_dialog.file_dialog import (
FileDialogDriver, handle_file_dialog,
)

__all__ = ["FileDialogDriver", "handle_file_dialog"]
Loading
Loading