Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-19)](#whats-new-2026-06-19)
- [What's new (2026-06-18)](#whats-new-2026-06-18)
- [What's new (2026-06-17)](#whats-new-2026-06-17)
- [What's new (2026-06)](#whats-new-2026-06)
Expand Down Expand Up @@ -58,6 +59,16 @@

---

## What's new (2026-06-19)

Two headless cores that shipped without the rest of their stack are now
first-class. Both gain a facade re-export, an `AC_*` executor command, an
MCP tool, and a Script Builder entry, with headless tests. Full reference:
[`docs/source/Eng/doc/new_features/v6_features_doc.rst`](docs/source/Eng/doc/new_features/v6_features_doc.rst).

- **Visual regression (golden images)** — `take_golden` / `compare_to_golden` (`AC_take_golden` / `AC_assert_visual`): capture a baseline screenshot and fail when the screen drifts beyond a pixel tolerance, with a highlighted diff image and mask regions. `AC_assert_visual` auto-creates the baseline on first run. PIL-only.
- **Finite-state machine** — `run_state_machine` (`AC_run_state_machine`): drive a script as a declarative `{initial, states}` spec whose `on_enter` actions run through the executor and whose transitions fire on `after` / `if_var_eq` / predicate guards, bounded by `max_steps` / `global_timeout_s`.

## What's new (2026-06-18)

Eight headless capabilities that round out scripting, integration, and CI
Expand Down
9 changes: 9 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
- [本次更新 (2026-06-18)](#本次更新-2026-06-18)
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
- [本次更新 (2026-06)](#本次更新-2026-06)
Expand Down Expand Up @@ -57,6 +58,14 @@

---

## 本次更新 (2026-06-19)

两个早已存在、却没接上其余各层的 headless 核心,现在成为一级功能。两者都新增 facade re-export、`AC_*` 执行器指令、MCP 工具与 Script Builder 项目,并有 headless 测试。完整参考:
[`docs/source/Eng/doc/new_features/v6_features_doc.rst`](../docs/source/Eng/doc/new_features/v6_features_doc.rst)。

- **视觉回归(黄金图像)** — `take_golden` / `compare_to_golden`(`AC_take_golden` / `AC_assert_visual`):捕获基准截图,画面偏离超过像素容差时判失败,并输出高亮差异图与遮罩区域。`AC_assert_visual` 首跑会自动建立基准。纯 PIL。
- **有限状态机** — `run_state_machine`(`AC_run_state_machine`):把脚本当成声明式 `{initial, states}` spec 驱动,`on_enter` 动作经执行器执行,transition 依 `after` / `if_var_eq` / predicate 触发,并以 `max_steps` / `global_timeout_s` 限制。

## 本次更新 (2026-06-18)

八项 headless 能力,补齐脚本化、集成与 CI 场景:真正的命令行界面、把录制转成代码,以及一级的 HTTP / SQL / Email / PDF / 等待步骤。每项都附带 headless API、`AC_*` 执行器指令、MCP 工具与可视化脚本构建器项目,并有 headless 测试(网络 / SMTP / PDF 后端均注入,不接触外部系统)。完整参考页:
Expand Down
9 changes: 9 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
- [本次更新 (2026-06-18)](#本次更新-2026-06-18)
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
- [本次更新 (2026-06)](#本次更新-2026-06)
Expand Down Expand Up @@ -57,6 +58,14 @@

---

## 本次更新 (2026-06-19)

兩個早已存在、卻沒接上其餘各層的 headless 核心,現在成為一級功能。兩者都新增 facade re-export、`AC_*` 執行器指令、MCP 工具與 Script Builder 項目,並有 headless 測試。完整參考:
[`docs/source/Zh/doc/new_features/v6_features_doc.rst`](../docs/source/Zh/doc/new_features/v6_features_doc.rst)。

- **視覺回歸(黃金影像)** — `take_golden` / `compare_to_golden`(`AC_take_golden` / `AC_assert_visual`):擷取基準截圖,畫面偏離超過像素容差時判失敗,並輸出標示差異圖與遮罩區域。`AC_assert_visual` 首跑會自動建立基準。純 PIL。
- **有限狀態機** — `run_state_machine`(`AC_run_state_machine`):把腳本當成宣告式 `{initial, states}` spec 驅動,`on_enter` 動作經執行器執行,transition 依 `after` / `if_var_eq` / predicate 觸發,並以 `max_steps` / `global_timeout_s` 限制。

## 本次更新 (2026-06-18)

八項 headless 能力,補齊腳本化、整合與 CI 情境:真正的命令列介面、把錄製轉成程式碼,以及一級的 HTTP / SQL / Email / PDF / 等待步驟。每項都附帶 headless API、`AC_*` 執行器指令、MCP 工具與視覺化腳本建構器項目,並有 headless 測試(網路 / SMTP / PDF 後端皆注入,不碰外部系統)。完整參考頁:
Expand Down
70 changes: 70 additions & 0 deletions docs/source/Eng/doc/new_features/v6_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
=========================================================
New Features (2026-06-19) — Visual Regression & FSM
=========================================================

Two headless cores that already existed but were never wired through the
rest of the stack are now first-class: **golden-image visual regression**
and a **declarative finite-state-machine** runner. Both ship a facade
re-export, an ``AC_*`` executor command, an MCP tool, and a Script Builder
entry, with headless tests (PIL images / specs are injected, so nothing
needs a real screen).

.. contents::
:local:
:depth: 2


Visual regression (golden images)
=================================

Capture a baseline image and later fail a run when the screen drifts from
it — the screenshot equivalent of a snapshot test::

from je_auto_control import take_golden, compare_to_golden

take_golden("goldens/login.png") # establish baseline
result = compare_to_golden("goldens/login.png", tolerance=0.5)
if not result.matched:
result.write_diff("goldens/login.diff.png")
print(result.summary)

``compare_to_golden`` returns a ``DiffResult`` (``matched``, ``diff_pct``,
``differing_pixels``, a highlighted ``diff_image``); ``tolerance`` is the
percentage of pixels allowed to differ and ``per_pixel_threshold`` ignores
small per-channel noise. ``MaskRegion`` excludes animated / volatile areas.
PIL-only — no OpenCV / SciPy dependency.

Executor commands:

* ``AC_take_golden`` — capture and save a baseline (optional ``region``).
* ``AC_assert_visual`` — compare the screen to a golden and raise on
mismatch (saving an optional ``diff_path``). On the **first run** (golden
missing) it captures the baseline and passes, unless
``create_if_missing`` is false.


Finite-state machine
====================

Drive a script as a declarative state machine — clearer than nested
loops / ifs for screen-flow automation::

from je_auto_control import run_state_machine

spec = {
"initial": "login",
"states": {
"login": {"on_enter": [["AC_click_text", {"text": "Sign in"}]],
"transitions": [{"go_to": "home", "after": 1.0}]},
"home": {"final": True},
},
}
result = run_state_machine(spec) # {final_state, steps, elapsed_s}

Each state's ``on_enter`` actions run through the executor; transitions
fire on guards (``after`` a delay, ``if_var_eq``, or a caller predicate).
``max_steps`` and ``global_timeout_s`` bound the run so it can't loop
forever.

Executor command: ``AC_run_state_machine`` (the ``spec`` dict travels in
JSON action files / the socket server / MCP unchanged).
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v3_features_doc
doc/new_features/v4_features_doc
doc/new_features/v5_features_doc
doc/new_features/v6_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
65 changes: 65 additions & 0 deletions docs/source/Zh/doc/new_features/v6_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
============================================
新功能 (2026-06-19) — 視覺回歸與狀態機
============================================

兩個早已存在、卻從未接上其餘各層的 headless 核心,現在成為一級功能:
**黃金影像視覺回歸**與**宣告式有限狀態機**執行器。兩者都提供 facade
re-export、``AC_*`` 執行器指令、MCP 工具與 Script Builder 項目,並有
headless 測試(PIL 影像 / spec 以注入方式提供,完全不需真實螢幕)。

.. contents::
:local:
:depth: 2


視覺回歸(黃金影像)
====================

擷取基準圖,之後在畫面偏離時讓該次執行失敗——等同於截圖版的 snapshot
測試::

from je_auto_control import take_golden, compare_to_golden

take_golden("goldens/login.png") # 建立基準
result = compare_to_golden("goldens/login.png", tolerance=0.5)
if not result.matched:
result.write_diff("goldens/login.diff.png")
print(result.summary)

``compare_to_golden`` 回傳 ``DiffResult``(``matched``、``diff_pct``、
``differing_pixels``、標示差異的 ``diff_image``);``tolerance`` 是允許
差異的像素百分比,``per_pixel_threshold`` 可忽略各通道的細微雜訊。
``MaskRegion`` 可排除動畫 / 易變區域。純 PIL——不需 OpenCV / SciPy。

執行器指令:

* ``AC_take_golden`` — 擷取並存基準圖(可選 ``region``)。
* ``AC_assert_visual`` — 把畫面與黃金影像比對,不符即拋例外(可存
``diff_path``)。**首跑**(基準不存在)時會擷取基準並通過,除非
``create_if_missing`` 設為 false。


有限狀態機
==========

把腳本當成宣告式狀態機來驅動——對於畫面流程自動化,比巢狀 loop / if
更清楚::

from je_auto_control import run_state_machine

spec = {
"initial": "login",
"states": {
"login": {"on_enter": [["AC_click_text", {"text": "Sign in"}]],
"transitions": [{"go_to": "home", "after": 1.0}]},
"home": {"final": True},
},
}
result = run_state_machine(spec) # {final_state, steps, elapsed_s}

每個狀態的 ``on_enter`` 動作會透過執行器執行;transition 依 guard 觸發
(延遲 ``after``、``if_var_eq`` 或呼叫端 predicate)。``max_steps`` 與
``global_timeout_s`` 限制執行,使其不會無限迴圈。

執行器指令:``AC_run_state_machine``(``spec`` dict 可原樣經 JSON 動作檔 /
socket server / MCP 傳遞)。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v3_features_doc
doc/new_features/v4_features_doc
doc/new_features/v5_features_doc
doc/new_features/v6_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
12 changes: 12 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,14 @@
wait_until_pixel_changes, wait_until_port, wait_until_process,
wait_until_region_idle, wait_until_screen_stable, wait_until_window_closed,
)
# Visual regression (golden-image comparison)
from je_auto_control.utils.visual_regression import (
DiffResult, MaskRegion, compare_to_golden, image_difference, take_golden,
)
# Declarative finite-state-machine engine for action JSON
from je_auto_control.utils.state_machine import (
StateMachine, StateMachineError, run_state_machine,
)
# Assertion DSL (verify screen state; raise on mismatch)
from je_auto_control.utils.assertion import (
AssertionResult, GroupAssertionResult, assert_all, assert_any,
Expand Down Expand Up @@ -570,6 +578,10 @@ def start_autocontrol_gui(*args, **kwargs):
"wait_until_region_idle", "wait_until_screen_stable",
"wait_until_clipboard_changes", "wait_until_window_closed",
"wait_until_file", "wait_until_port", "wait_until_process",
# Visual regression + state machine
"take_golden", "compare_to_golden", "image_difference",
"DiffResult", "MaskRegion",
"run_state_machine", "StateMachine", "StateMachineError",
# Assertion DSL
"AssertionResult", "assert_image", "assert_pixel",
"assert_text", "assert_window", "assert_clipboard", "assert_process",
Expand Down
23 changes: 23 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,29 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
"AC_shell_command", "Shell", "Shell Command",
fields=(FieldSpec("shell_command", FieldType.STRING),),
))
specs.append(CommandSpec(
"AC_take_golden", "Report", "Capture Golden Image",
fields=(FieldSpec("path", FieldType.FILE_PATH),),
description="Capture and save a baseline image for visual regression.",
))
specs.append(CommandSpec(
"AC_assert_visual", "Report", "Assert Visual (Golden)",
fields=(
FieldSpec("golden_path", FieldType.FILE_PATH),
FieldSpec("tolerance", FieldType.FLOAT, optional=True, default=0.0,
min_value=0.0),
FieldSpec("per_pixel_threshold", FieldType.INT, optional=True,
default=16, min_value=0),
FieldSpec("diff_path", FieldType.FILE_PATH, optional=True),
),
description=("Compare the screen to a golden image; first run creates "
"the baseline. Use the JSON view for a region / masks."),
))
specs.append(CommandSpec(
"AC_run_state_machine", "Flow", "Run State Machine",
description=("Run a finite-state-machine; configure the 'spec' "
"{initial, states} dict in the JSON view."),
))
specs.append(CommandSpec(
"AC_shell_to_var", "Shell", "Shell Output into Variable",
fields=(
Expand Down
46 changes: 46 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -2169,6 +2169,49 @@ def _assert_pdf_text(path: str, text: str, present: bool = True,
raise_on_fail=bool(raise_on_fail))


def _take_golden(path: str, region: Optional[List[int]] = None) -> str:
"""Adapter: capture and save a golden/baseline image."""
from je_auto_control.utils.visual_regression import take_golden
return str(take_golden(path, region=region))


def _assert_visual(golden_path: str, region: Optional[List[int]] = None,
tolerance: float = 0.0, per_pixel_threshold: int = 16,
diff_path: Optional[str] = None,
create_if_missing: bool = True,
raise_on_fail: bool = True) -> Dict[str, Any]:
"""Adapter: compare the screen to a golden image (first run creates it)."""
import os
from je_auto_control.utils.exception.exceptions import (
AutoControlAssertionException,
)
from je_auto_control.utils.visual_regression import (
compare_to_golden, take_golden,
)
if create_if_missing and not os.path.exists(
os.path.expanduser(str(golden_path))):
take_golden(golden_path, region=region)
return {"created": True, "matched": True, "golden": str(golden_path)}
result = compare_to_golden(
golden_path, region=region, tolerance=float(tolerance),
per_pixel_threshold=int(per_pixel_threshold))
if diff_path and result.diff_image is not None:
result.write_diff(diff_path)
data = {"matched": result.matched, "diff_pct": result.diff_pct,
"differing_pixels": result.differing_pixels,
"total_pixels": result.total_pixels,
"tolerance_pct": result.tolerance_pct}
if not result.matched and raise_on_fail:
raise AutoControlAssertionException(result.summary)
return data


def _run_state_machine(spec: Any) -> Dict[str, Any]:
"""Adapter: run a finite-state-machine spec through the executor."""
from je_auto_control.utils.state_machine import run_state_machine
return run_state_machine(spec)


class Executor:
"""
Executor
Expand Down Expand Up @@ -2234,6 +2277,9 @@ def __init__(self):
"AC_generate_code": _generate_code,
"AC_send_email": _send_email,
"AC_assert_pdf_text": _assert_pdf_text,
"AC_take_golden": _take_golden,
"AC_assert_visual": _assert_visual,
"AC_run_state_machine": _run_state_machine,
"AC_http_request": http_request,

# Record 錄製
Expand Down
Loading
Loading