Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-19) — Tweened Drag](#whats-new-2026-06-19--tweened-drag)
- [What's new (2026-06-19) — Process-Doc (SOP) Generator](#whats-new-2026-06-19--process-doc-sop-generator)
- [What's new (2026-06-19) — Heal Analytics & Secret Scan](#whats-new-2026-06-19--heal-analytics--secret-scan)
- [What's new (2026-06-19) — CI Annotations & Clipboard History](#whats-new-2026-06-19--ci-annotations--clipboard-history)
Expand Down Expand Up @@ -81,6 +82,12 @@

---

## What's new (2026-06-19) — Tweened Drag

Deterministic eased drags. Full reference: [`docs/source/Eng/doc/new_features/v29_features_doc.rst`](docs/source/Eng/doc/new_features/v29_features_doc.rst).

- **`tween_points` / `tween_drag` / `easing_names`** (`AC_tween_drag`, `ac_tween_drag`): drag from `start` to `end` along an eased curve (linear / ease_in_out_quad / ease_out_cubic / ease_in_cubic) — deterministic, pure-math path, injectable sink for tests; complements the humanized jitter.

## What's new (2026-06-19) — Process-Doc (SOP) Generator

Turn an action list into a step-by-step SOP. Full reference: [`docs/source/Eng/doc/new_features/v28_features_doc.rst`](docs/source/Eng/doc/new_features/v28_features_doc.rst).
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-19) — 缓动拖拽](#本次更新-2026-06-19--缓动拖拽)
- [本次更新 (2026-06-19) — 流程文档(SOP)生成器](#本次更新-2026-06-19--流程文档sop生成器)
- [本次更新 (2026-06-19) — 修复分析与机密扫描](#本次更新-2026-06-19--修复分析与机密扫描)
- [本次更新 (2026-06-19) — CI 注解与剪贴板历史](#本次更新-2026-06-19--ci-注解与剪贴板历史)
Expand Down Expand Up @@ -80,6 +81,12 @@

---

## 本次更新 (2026-06-19) — 缓动拖拽

确定性的缓动拖拽。完整参考:[`docs/source/Zh/doc/new_features/v29_features_doc.rst`](../docs/source/Zh/doc/new_features/v29_features_doc.rst)。

- **`tween_points` / `tween_drag` / `easing_names`**(`AC_tween_drag`、`ac_tween_drag`):沿缓动曲线从 `start` 拖到 `end`(linear / ease_in_out_quad / ease_out_cubic / ease_in_cubic)——确定性、纯数学路径、测试可注入 sink;补足人性化抖动。

## 本次更新 (2026-06-19) — 流程文档(SOP)生成器

把动作列表转成逐步 SOP。完整参考:[`docs/source/Zh/doc/new_features/v28_features_doc.rst`](../docs/source/Zh/doc/new_features/v28_features_doc.rst)。
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-19) — 緩動拖曳](#本次更新-2026-06-19--緩動拖曳)
- [本次更新 (2026-06-19) — 流程文件(SOP)產生器](#本次更新-2026-06-19--流程文件sop產生器)
- [本次更新 (2026-06-19) — 修復分析與機密掃描](#本次更新-2026-06-19--修復分析與機密掃描)
- [本次更新 (2026-06-19) — CI 註解與剪貼簿歷史](#本次更新-2026-06-19--ci-註解與剪貼簿歷史)
Expand Down Expand Up @@ -80,6 +81,12 @@

---

## 本次更新 (2026-06-19) — 緩動拖曳

決定性的緩動拖曳。完整參考:[`docs/source/Zh/doc/new_features/v29_features_doc.rst`](../docs/source/Zh/doc/new_features/v29_features_doc.rst)。

- **`tween_points` / `tween_drag` / `easing_names`**(`AC_tween_drag`、`ac_tween_drag`):沿緩動曲線從 `start` 拖到 `end`(linear / ease_in_out_quad / ease_out_cubic / ease_in_cubic)——決定性、純數學路徑、測試可注入 sink;補足人性化抖動。

## 本次更新 (2026-06-19) — 流程文件(SOP)產生器

把動作清單轉成逐步 SOP。完整參考:[`docs/source/Zh/doc/new_features/v28_features_doc.rst`](../docs/source/Zh/doc/new_features/v28_features_doc.rst)。
Expand Down
29 changes: 29 additions & 0 deletions docs/source/Eng/doc/new_features/v29_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
==================================================
New Features (2026-06-19) — Tweened Drag
==================================================

Deterministic eased drags along a curved path (PyAutoGUI-style ``tween``),
complementing the existing humanized jitter. Pure standard library; full
stack. The point math is pure and unit-testable; dispatch goes through an
injectable sink.

.. contents::
:local:
:depth: 2


Usage
=====

::

from je_auto_control import tween_points, tween_drag, easing_names

tween_points((0, 0), (100, 50), steps=20, easing="ease_out_cubic")
tween_drag((0, 0), (300, 200), steps=40, easing="ease_in_out_quad")

``tween_points`` returns ``steps + 1`` eased points between two
coordinates; ``tween_drag`` presses at the start, moves through the points,
and releases at the end. Easings: ``linear`` / ``ease_in_out_quad`` /
``ease_out_cubic`` / ``ease_in_cubic`` (see :func:`easing_names`). Exposed
as ``AC_tween_drag`` / ``ac_tween_drag`` (``start`` / ``end`` as ``[x, y]``).
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v26_features_doc
doc/new_features/v27_features_doc
doc/new_features/v28_features_doc
doc/new_features/v29_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
28 changes: 28 additions & 0 deletions docs/source/Zh/doc/new_features/v29_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
==========================================
新功能 (2026-06-19) — 緩動拖曳
==========================================

沿曲線路徑的決定性緩動拖曳(PyAutoGUI 風格的 ``tween``),補足既有的
人性化抖動。純標準庫;走完整五層。座標數學為純函式、可單元測試;派發
透過可注入的 sink。

.. contents::
:local:
:depth: 2


用法
====

::

from je_auto_control import tween_points, tween_drag, easing_names

tween_points((0, 0), (100, 50), steps=20, easing="ease_out_cubic")
tween_drag((0, 0), (300, 200), steps=40, easing="ease_in_out_quad")

``tween_points`` 回傳兩點之間 ``steps + 1`` 個緩動點;``tween_drag`` 在
起點按下、沿各點移動、於終點放開。緩動函式:``linear`` /
``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic``(見
:func:`easing_names`)。對應 ``AC_tween_drag`` / ``ac_tween_drag``
(``start`` / ``end`` 以 ``[x, y]`` 表示)。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v26_features_doc
doc/new_features/v27_features_doc
doc/new_features/v28_features_doc
doc/new_features/v29_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
5 changes: 5 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,10 @@
from je_auto_control.utils.process_doc import (
describe_step, generate_sop, write_sop,
)
# Eased / tweened interpolated drag
from je_auto_control.utils.tween_drag import (
easing_names, tween_drag, tween_points,
)
# Background popup/interrupt watchdog (unattended automation)
from je_auto_control.utils.watchdog import (
PopupWatchdog, WatchdogRule, default_popup_watchdog,
Expand Down Expand Up @@ -628,6 +632,7 @@ def start_autocontrol_gui(*args, **kwargs):
"ClipboardHistory", "default_clipboard_history",
"analyze_heal_log", "heal_stats", "scan_secrets",
"describe_step", "generate_sop", "write_sop",
"easing_names", "tween_drag", "tween_points",
# MCP server
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",
Expand Down
14 changes: 14 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -667,6 +667,20 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
_add_resilience_specs(specs)
_add_devex_specs(specs)
_add_audit_specs(specs)
specs.append(CommandSpec(
"AC_tween_drag", "Mouse", "Tweened Drag",
fields=(
FieldSpec("steps", FieldType.INT, optional=True, default=30),
FieldSpec("easing", FieldType.ENUM,
choices=("linear", "ease_in_out_quad", "ease_out_cubic",
"ease_in_cubic"),
optional=True, default="ease_in_out_quad"),
FieldSpec("button", FieldType.ENUM, choices=_MOUSE_BUTTONS,
optional=True, default="mouse_left"),
),
description="Drag along an eased path; 'start'/'end' [x,y] via JSON "
"view.",
))
specs.append(CommandSpec(
"AC_generate_sop", "Report", "Generate SOP Document",
fields=(
Expand Down
11 changes: 11 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -2872,6 +2872,16 @@ def _generate_sop(actions: List[Any], title: str = "Automation Procedure",
return generate_sop(actions, title=title)


def _tween_drag(start: List[int], end: List[int], steps: int = 30,
easing: str = "ease_in_out_quad",
button: str = "mouse_left") -> Dict[str, Any]:
"""Adapter: drag along an eased path from start to end."""
from je_auto_control.utils.tween_drag import tween_drag
result = tween_drag(tuple(start), tuple(end), steps=int(steps),
easing=easing, button=button)
return {"points": result["points"]}


class Executor:
"""
Executor
Expand Down Expand Up @@ -3100,6 +3110,7 @@ def __init__(self):
"AC_heal_stats": _heal_stats,
"AC_scan_secrets": _scan_secrets,
"AC_generate_sop": _generate_sop,
"AC_tween_drag": _tween_drag,
"AC_a11y_record_start": _a11y_record_start,
"AC_a11y_record_stop": _a11y_record_stop,
"AC_a11y_record_events": _a11y_record_events,
Expand Down
22 changes: 21 additions & 1 deletion je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -2521,6 +2521,26 @@ def process_doc_tools() -> List[MCPTool]:
]


def tween_drag_tools() -> List[MCPTool]:
return [
MCPTool(
name="ac_tween_drag",
description=("Drag from 'start' [x,y] to 'end' [x,y] along an "
"eased path (easing: linear / ease_in_out_quad / "
"ease_out_cubic / ease_in_cubic). Returns {points}."),
input_schema=schema({
"start": {"type": "array", "items": {"type": "integer"}},
"end": {"type": "array", "items": {"type": "integer"}},
"steps": {"type": "integer"},
"easing": {"type": "string"},
"button": {"type": "string"}},
required=["start", "end"]),
handler=h.tween_drag,
annotations=SIDE_EFFECT_ONLY,
),
]


def unattended_tools() -> List[MCPTool]:
return [
MCPTool(
Expand Down Expand Up @@ -3578,7 +3598,7 @@ def media_assert_tools() -> List[MCPTool]:
checkpoint_tools, set_of_marks_tools, screen_state_tools,
input_macro_tools, resilience_tools,
ci_annotation_tools, clipboard_history_tools, audit_analysis_tools,
process_doc_tools,
process_doc_tools, tween_drag_tools,
screen_record_tools,
process_and_shell_tools, remote_desktop_tools, gamepad_tools,
usb_passthrough_tools, assertion_tools, data_source_tools,
Expand Down
7 changes: 7 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1229,6 +1229,13 @@ def generate_sop(actions, title="Automation Procedure", path=None):
return _gen(actions, title=title)


def tween_drag(start, end, steps=30, easing="ease_in_out_quad",
button="mouse_left"):
from je_auto_control.utils.tween_drag import tween_drag as _td
return {"points": _td(tuple(start), tuple(end), steps=int(steps),
easing=easing, button=button)["points"]}


def vlm_locate(description: str,
screen_region: Optional[List[int]] = None,
model: Optional[str] = None) -> Optional[List[int]]:
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/tween_drag/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Eased / tweened interpolated drag along a curved path."""
from je_auto_control.utils.tween_drag.tween_drag import (
easing_names, tween_drag, tween_points,
)

__all__ = ["easing_names", "tween_drag", "tween_points"]
89 changes: 89 additions & 0 deletions je_auto_control/utils/tween_drag/tween_drag.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
"""Eased / tweened interpolated drag along a curved path.

AutoControl has humanized *jitter* but no deterministic named easings.
:func:`tween_points` produces an eased sequence of points between two
coordinates (pure math), and :func:`tween_drag` presses at the start, moves
through the points, and releases at the end — for smooth, deterministic
drags (PyAutoGUI-style ``tween``).

The point math is pure and unit-testable; dispatch goes through an
injectable ``sink`` so the drag is tested without real input. Imports no
``PySide6``.
"""
from typing import Any, Callable, Dict, List, Optional, Tuple


def _linear(t: float) -> float:
return t


def _ease_in_out_quad(t: float) -> float:
return 2 * t * t if t < 0.5 else 1 - ((-2 * t + 2) ** 2) / 2


def _ease_out_cubic(t: float) -> float:
return 1 - (1 - t) ** 3


def _ease_in_cubic(t: float) -> float:
return t ** 3


_EASINGS: Dict[str, Callable[[float], float]] = {
"linear": _linear,
"ease_in_out_quad": _ease_in_out_quad,
"ease_out_cubic": _ease_out_cubic,
"ease_in_cubic": _ease_in_cubic,
}


def easing_names() -> List[str]:
"""Return the available easing-function names."""
return sorted(_EASINGS)


def tween_points(start: Tuple[int, int], end: Tuple[int, int],
steps: int = 30,
easing: str = "ease_in_out_quad") -> List[List[int]]:
"""Return ``steps + 1`` eased points from ``start`` to ``end``."""
curve = _EASINGS.get(easing, _linear)
count = max(1, int(steps))
start_x, start_y = start
end_x, end_y = end
points: List[List[int]] = []
for index in range(count + 1):
progress = curve(index / count)
points.append([round(start_x + (end_x - start_x) * progress),
round(start_y + (end_y - start_y) * progress)])
return points


def _default_sink(event: Dict[str, Any]) -> None:
from je_auto_control.wrapper.auto_control_mouse import (
press_mouse, release_mouse, set_mouse_position)
x, y = int(event["x"]), int(event["y"])
op = event["op"]
if op == "move":
set_mouse_position(x, y)
elif op == "press":
set_mouse_position(x, y)
press_mouse(event.get("button", "mouse_left"), x, y)
elif op == "release":
set_mouse_position(x, y)
release_mouse(event.get("button", "mouse_left"), x, y)


def tween_drag(start: Tuple[int, int], end: Tuple[int, int], *,
steps: int = 30, easing: str = "ease_in_out_quad",
button: str = "mouse_left",
sink: Optional[Callable[[Dict[str, Any]], None]] = None
) -> Dict[str, Any]:
"""Drag from ``start`` to ``end`` along an eased path; return point count."""
points = tween_points(start, end, steps, easing)
dispatch = sink or _default_sink
first, last = points[0], points[-1]
dispatch({"op": "press", "button": button, "x": first[0], "y": first[1]})
for x, y in points:
dispatch({"op": "move", "x": x, "y": y})
dispatch({"op": "release", "button": button, "x": last[0], "y": last[1]})
return {"points": len(points), "path": points}
Loading
Loading