Add reactive screen observer (appear/vanish/change)

README.md

@@ -13,6 +13,7 @@
 
 ## Table of Contents
 
+- [What's new (2026-06-19) — Reactive Observer](#whats-new-2026-06-19--reactive-observer)
 - [What's new (2026-06-19) — WCAG 2.2 Audit](#whats-new-2026-06-19--wcag-22-audit)
 - [What's new (2026-06-19) — Memory & Determinism](#whats-new-2026-06-19--memory--determinism)
 - [What's new (2026-06-19) — Office I/O](#whats-new-2026-06-19--office-io)
@@ -69,6 +70,13 @@
 
 ---
 
+## What's new (2026-06-19) — Reactive Observer
+
+A non-blocking screen observer (SikuliX `observe` model), full stack (facade, `AC_*`, MCP, Script Builder). Full reference: [`docs/source/Eng/doc/new_features/v17_features_doc.rst`](docs/source/Eng/doc/new_features/v17_features_doc.rst).
+
+- **`ScreenObserver`** (`AC_observe_add` / `AC_observe_remove` / `AC_observe_list` / `AC_observe_poll` / `AC_observe_start` / `AC_observe_stop`, `ac_observe_*`): register watches that fire on **appear** / **vanish** / **change** of an image/text/pixel and run a callback or action list — react to dialogs/progress/status while the main flow continues.
+- **Testable by design** — detection is an injectable `predicate`; transition logic is unit-tested via `poll_once()` with synthetic values. Built-in `image_predicate` / `text_predicate` / `pixel_predicate` wrap the existing locate/OCR/pixel helpers.
+
 ## What's new (2026-06-19) — WCAG 2.2 Audit
 
 The accessibility audit gains a WCAG 2.2 / EN 301 549 success-criterion layer, full stack (facade, `AC_*`, MCP, Script Builder). Full reference: [`docs/source/Eng/doc/new_features/v16_features_doc.rst`](docs/source/Eng/doc/new_features/v16_features_doc.rst).

README/README_zh-CN.md

@@ -12,6 +12,7 @@
 
 ## 目录
 
+- [本次更新 (2026-06-19) — 反应式观察器](#本次更新-2026-06-19--反应式观察器)
 - [本次更新 (2026-06-19) — WCAG 2.2 审计](#本次更新-2026-06-19--wcag-22-审计)
 - [本次更新 (2026-06-19) — 记忆与确定性](#本次更新-2026-06-19--记忆与确定性)
 - [本次更新 (2026-06-19) — Office 读写](#本次更新-2026-06-19--office-读写)
@@ -68,6 +69,13 @@
 
 ---
 
+## 本次更新 (2026-06-19) — 反应式观察器
+
+非阻塞的屏幕观察器(SikuliX `observe` 模型),走完整五层(facade、`AC_*`、MCP、Script Builder)。完整参考:[`docs/source/Zh/doc/new_features/v17_features_doc.rst`](../docs/source/Zh/doc/new_features/v17_features_doc.rst)。
+
+- **`ScreenObserver`**(`AC_observe_add` / `AC_observe_remove` / `AC_observe_list` / `AC_observe_poll` / `AC_observe_start` / `AC_observe_stop`、`ac_observe_*`):注册监看,在图像/文本/像素的 **appear** / **vanish** / **change** 时触发回调或执行 action list——在主流程继续的同时对对话框/进度/状态做出反应。
+- **为可测试而设计**——检测是可注入的 `predicate`;转换逻辑用 `poll_once()` 以合成值做单元测试。内建 `image_predicate` / `text_predicate` / `pixel_predicate` 包装既有的 locate/OCR/pixel 辅助函数。
+
 ## 本次更新 (2026-06-19) — WCAG 2.2 审计
 
 无障碍审计新增 WCAG 2.2 / EN 301 549 成功准则层,走完整五层(facade、`AC_*`、MCP、Script Builder)。完整参考:[`docs/source/Zh/doc/new_features/v16_features_doc.rst`](../docs/source/Zh/doc/new_features/v16_features_doc.rst)。

README/README_zh-TW.md

@@ -12,6 +12,7 @@
 
 ## 目錄
 
+- [本次更新 (2026-06-19) — 反應式觀察器](#本次更新-2026-06-19--反應式觀察器)
 - [本次更新 (2026-06-19) — WCAG 2.2 稽核](#本次更新-2026-06-19--wcag-22-稽核)
 - [本次更新 (2026-06-19) — 記憶與決定性](#本次更新-2026-06-19--記憶與決定性)
 - [本次更新 (2026-06-19) — Office 讀寫](#本次更新-2026-06-19--office-讀寫)
@@ -68,6 +69,13 @@
 
 ---
 
+## 本次更新 (2026-06-19) — 反應式觀察器
+
+非阻塞的螢幕觀察器(SikuliX `observe` 模型),走完整五層(facade、`AC_*`、MCP、Script Builder)。完整參考:[`docs/source/Zh/doc/new_features/v17_features_doc.rst`](../docs/source/Zh/doc/new_features/v17_features_doc.rst)。
+
+- **`ScreenObserver`**(`AC_observe_add` / `AC_observe_remove` / `AC_observe_list` / `AC_observe_poll` / `AC_observe_start` / `AC_observe_stop`、`ac_observe_*`):註冊監看,在影像/文字/像素的 **appear** / **vanish** / **change** 時觸發回呼或執行 action list——在主流程繼續的同時對對話框/進度/狀態做出反應。
+- **為可測試而設計**——偵測是可注入的 `predicate`;轉換邏輯用 `poll_once()` 以合成值做單元測試。內建 `image_predicate` / `text_predicate` / `pixel_predicate` 包裝既有的 locate/OCR/pixel 輔助函式。
+
 ## 本次更新 (2026-06-19) — WCAG 2.2 稽核
 
 無障礙稽核新增 WCAG 2.2 / EN 301 549 成功準則層,走完整五層(facade、`AC_*`、MCP、Script Builder)。完整參考:[`docs/source/Zh/doc/new_features/v16_features_doc.rst`](../docs/source/Zh/doc/new_features/v16_features_doc.rst)。

docs/source/Eng/doc/new_features/v17_features_doc.rst

@@ -0,0 +1,57 @@
+==================================================
+New Features (2026-06-19) — Reactive Observer
+==================================================
+
+A non-blocking **screen observer**: register watches on a region/predicate
+and get a callback (or run an action list) when the watched thing
+**appears**, **vanishes**, or **changes**. This is the complement to the
+blocking ``wait_for_*`` helpers — a flow can react to dialogs, progress, or
+status changes *while doing other work* (the SikuliX ``observe`` model).
+
+Pure standard library; wired through the full stack (facade, ``AC_*``
+executor commands, MCP tools, Script Builder).
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Python API
+=========
+
+::
+
+    from je_auto_control import ScreenObserver, image_predicate, EVENT_APPEAR
+
+    obs = ScreenObserver(poll_interval_s=0.5)
+    obs.add("error-dialog",
+            image_predicate("error.png", threshold=0.9),
+            on_event=lambda event, value: dismiss(),
+            events=(EVENT_APPEAR,))
+    obs.start()                 # background polling thread
+    ...
+    obs.stop()
+
+Detection is decoupled from the screen: a watch's ``predicate`` just
+returns the current value (truthy = present), so transition logic is
+unit-tested with synthetic values via ``poll_once()``. Built-in predicate
+builders — :func:`image_predicate`, :func:`text_predicate`,
+:func:`pixel_predicate` — wrap the existing locate / OCR / pixel helpers.
+
+Transitions: ``appear`` (absent -> present), ``vanish`` (present ->
+absent), ``change`` (present, value differs). Subscribe to a subset via
+``events=``.
+
+
+Executor / MCP commands
+======================
+
+* ``AC_observe_add`` — watch ``kind`` (``image`` / ``text`` / ``pixel``)
+  for ``event`` and run ``actions`` when it fires (the watchdog pattern,
+  generalised to screen content).
+* ``AC_observe_remove`` / ``AC_observe_list`` — manage watches.
+* ``AC_observe_poll`` — evaluate every watch once and return fired events
+  (deterministic, thread-free — ideal in scripts/tests).
+* ``AC_observe_start`` / ``AC_observe_stop`` — background poll thread.
+
+The matching ``ac_observe_*`` MCP tools expose the same surface.

docs/source/Eng/eng_index.rst

@@ -39,6 +39,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v14_features_doc
    doc/new_features/v15_features_doc
    doc/new_features/v16_features_doc
+   doc/new_features/v17_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v17_features_doc.rst

@@ -0,0 +1,53 @@
+==========================================
+新功能 (2026-06-19) — 反應式觀察器
+==========================================
+
+非阻塞的**螢幕觀察器**:在區域/條件上註冊監看,當被監看的目標**出現**、
+**消失**或**改變**時,觸發回呼(或執行一段 action list)。這是阻塞式
+``wait_for_*`` 的互補——流程可以在做其他事的同時對對話框、進度或狀態
+變化做出反應(SikuliX 的 ``observe`` 模型)。
+
+純標準庫;走完整五層(facade、``AC_*`` 執行器指令、MCP 工具、Script
+Builder)。
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Python API
+==========
+
+::
+
+    from je_auto_control import ScreenObserver, image_predicate, EVENT_APPEAR
+
+    obs = ScreenObserver(poll_interval_s=0.5)
+    obs.add("error-dialog",
+            image_predicate("error.png", threshold=0.9),
+            on_event=lambda event, value: dismiss(),
+            events=(EVENT_APPEAR,))
+    obs.start()                 # 背景輪詢執行緒
+    ...
+    obs.stop()
+
+偵測與螢幕解耦:監看的 ``predicate`` 只回傳當前值(truthy 代表存在),
+因此轉換邏輯可用合成值透過 ``poll_once()`` 做單元測試。內建的條件建構器
+——:func:`image_predicate`、:func:`text_predicate`、
+:func:`pixel_predicate`——包裝既有的 locate / OCR / pixel 輔助函式。
+
+轉換:``appear``(不存在→存在)、``vanish``(存在→不存在)、``change``
+(存在且值改變)。可用 ``events=`` 只訂閱其中一部分。
+
+
+執行器 / MCP 指令
+=================
+
+* ``AC_observe_add`` — 監看 ``kind``(``image`` / ``text`` / ``pixel``)
+  的 ``event``,觸發時執行 ``actions``(watchdog 模式,推廣到螢幕內容)。
+* ``AC_observe_remove`` / ``AC_observe_list`` — 管理監看。
+* ``AC_observe_poll`` — 評估每個監看一次並回傳觸發事件(決定性、免執行緒
+  ——適合腳本/測試)。
+* ``AC_observe_start`` / ``AC_observe_stop`` — 背景輪詢執行緒。
+
+對應的 ``ac_observe_*`` MCP 工具提供相同介面。

docs/source/Zh/zh_index.rst

@@ -39,6 +39,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v14_features_doc
    doc/new_features/v15_features_doc
    doc/new_features/v16_features_doc
+   doc/new_features/v17_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -144,6 +144,11 @@
 from je_auto_control.utils.deterministic import (
     DeterministicRun, seed_everything,
 )
+# Reactive screen observer (appear / vanish / change -> callback)
+from je_auto_control.utils.observer import (
+    ScreenObserver, WatchRule, default_observer,
+    image_predicate, pixel_predicate, text_predicate,
+)
 # Background popup/interrupt watchdog (unattended automation)
 from je_auto_control.utils.watchdog import (
     PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -559,6 +564,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "read_presentation", "write_presentation",
     "AgentMemory", "Episode",
     "DeterministicRun", "seed_everything",
+    "ScreenObserver", "WatchRule", "default_observer",
+    "image_predicate", "pixel_predicate", "text_predicate",
     # MCP server
     "AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
     "MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

@@ -669,6 +669,46 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         ),
         description="WCAG 2.2 audit: SC-tagged findings + Target Size 2.5.8.",
     ))
+    _add_observer_specs(specs)
+
+
+def _add_observer_specs(specs: List[CommandSpec]) -> None:
+    specs.append(CommandSpec(
+        "AC_observe_add", "Flow", "Observe: Add Watch",
+        fields=(
+            FieldSpec("name", FieldType.STRING),
+            FieldSpec("kind", FieldType.ENUM,
+                      choices=("image", "text", "pixel"), default="image"),
+            FieldSpec("event", FieldType.ENUM,
+                      choices=("appear", "vanish", "change"),
+                      default="appear"),
+            FieldSpec("image", FieldType.FILE_PATH, optional=True),
+            FieldSpec("threshold", FieldType.FLOAT, optional=True,
+                      default=0.8),
+            FieldSpec("text", FieldType.STRING, optional=True),
+            FieldSpec("x", FieldType.INT, optional=True),
+            FieldSpec("y", FieldType.INT, optional=True),
+        ),
+        description="Run 'actions' (JSON view) on appear/vanish/change of an "
+                    "image/text/pixel.",
+    ))
+    specs.append(CommandSpec(
+        "AC_observe_remove", "Flow", "Observe: Remove Watch",
+        fields=(FieldSpec("name", FieldType.STRING),),
+        description="Remove a registered watch.",
+    ))
+    specs.append(CommandSpec(
+        "AC_observe_list", "Flow", "Observe: List Watches",
+        description="List registered watch names."))
+    specs.append(CommandSpec(
+        "AC_observe_poll", "Flow", "Observe: Poll Once",
+        description="Evaluate all watches once; return fired events."))
+    specs.append(CommandSpec(
+        "AC_observe_start", "Flow", "Observe: Start",
+        description="Start the background observer thread."))
+    specs.append(CommandSpec(
+        "AC_observe_stop", "Flow", "Observe: Stop",
+        description="Stop the background observer thread."))
 
 
 def _add_memory_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/executor/action_executor.py

@@ -2572,6 +2572,71 @@ def _seed_everything(seed: int = 0) -> Dict[str, Any]:
     return {"seed": seed_everything(int(seed))}
 
 
+def _observe_handler(actions: List[Any]) -> Callable[[str, Any], None]:
+    """Build an observer callback that runs an action list on each event."""
+    def handler(_event: str, _value: Any) -> None:
+        if actions:
+            executor.execute_action(list(actions))
+    return handler
+
+
+def _observe_predicate(kind: str, params: Dict[str, Any]):
+    from je_auto_control.utils.observer import (
+        image_predicate, pixel_predicate, text_predicate)
+    builders = {
+        "image": lambda: image_predicate(params.get("image", ""),
+                                         params.get("threshold", 0.8)),
+        "text": lambda: text_predicate(params.get("text", "")),
+        "pixel": lambda: pixel_predicate(int(params.get("x", 0)),
+                                         int(params.get("y", 0))),
+    }
+    if kind not in builders:
+        raise AutoControlActionException(f"unknown observe kind: {kind!r}")
+    return builders[kind]()
+
+
+def _observe_add(name: str, kind: str = "image", event: str = "appear",
+                 actions: Optional[List[Any]] = None,
+                 **params: Any) -> Dict[str, Any]:
+    """Adapter: watch image/text/pixel; run ``actions`` on the event."""
+    from je_auto_control.utils.observer import default_observer
+    default_observer.add(name, _observe_predicate(kind, params),
+                         _observe_handler(actions or []), events=(event,))
+    return {"name": name, "kind": kind, "event": event}
+
+
+def _observe_remove(name: str) -> Dict[str, Any]:
+    """Adapter: remove a registered watch."""
+    from je_auto_control.utils.observer import default_observer
+    return {"removed": default_observer.remove(name)}
+
+
+def _observe_list() -> Dict[str, Any]:
+    """Adapter: list registered watch names."""
+    from je_auto_control.utils.observer import default_observer
+    return {"names": default_observer.names()}
+
+
+def _observe_poll() -> Dict[str, Any]:
+    """Adapter: evaluate all watches once; return fired events."""
+    from je_auto_control.utils.observer import default_observer
+    return {"fired": default_observer.poll_once()}
+
+
+def _observe_start() -> Dict[str, Any]:
+    """Adapter: start the background observer thread."""
+    from je_auto_control.utils.observer import default_observer
+    default_observer.start()
+    return {"running": default_observer.running}
+
+
+def _observe_stop() -> Dict[str, Any]:
+    """Adapter: stop the background observer thread."""
+    from je_auto_control.utils.observer import default_observer
+    default_observer.stop()
+    return {"running": default_observer.running}
+
+
 class Executor:
     """
     Executor
@@ -2764,6 +2829,12 @@ def __init__(self):
             "AC_memory_forget": _memory_forget,
             "AC_memory_stats": _memory_stats,
             "AC_seed_everything": _seed_everything,
+            "AC_observe_add": _observe_add,
+            "AC_observe_remove": _observe_remove,
+            "AC_observe_list": _observe_list,
+            "AC_observe_poll": _observe_poll,
+            "AC_observe_start": _observe_start,
+            "AC_observe_stop": _observe_stop,
             "AC_a11y_record_start": _a11y_record_start,
             "AC_a11y_record_stop": _a11y_record_stop,
             "AC_a11y_record_events": _a11y_record_events,