diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..6710e6636 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,216 @@ +# AGENTS.md + +## Commits + +- Plain commit messages only: no `Generated with ` footers and no + `Co-Authored-By` trailers. The commit author is the repo owner; tool + attribution does not belong in history. +- Prefer `-m` with single quotes (or a quoted heredoc) over unquoted + `$(cat <= 3.12, managed with `uv`. The project venv lives at `.venv/`; + run tools directly (`.venv/bin/python -m pytest ...`) or via `uv run`. +- `make test` — deterministic unit suite (no device, credentials, or + private services needed). Testpaths: `tests/unit`, `tests/tools`, + `packages/artemis-client/tests` (see `[tool.pytest.ini_options]`). +- `make lint` — `ruff format --check` + `ruff check`. Run on changed + files before committing. +- `make test-integration` / `make test-device` — need configured model + credentials and an Android device respectively. +- The full suite is NOT green on a bare checkout: expect failures that + require `GOOGLE_API_KEY`/`GEMINI_API_KEY`, provider mocks, local config + values, and tmp-path fixtures. Verify work with targeted tests instead: + e.g. `pytest -q tests/unit/agents/flash tests/unit/config + tests/unit/core` for Flash/local-model changes. + +## Local Model Setup (Gemma 4 E4B) + +- Local OpenAI-compatible endpoint convention: `http://127.0.0.1:8080/v1`, + model alias `gemma4-e4b`, served via `artemis model + list/pull/serve/status/stop` (MLX on Apple Silicon). +- Opt-in config: `config/artemis.gemma4.jsonc`. `config/artemis.jsonc` is + the machine-local deployment config — see "Local-only files" below. +- `thinking_level`/`include_thoughts` in a config are Gemini-provider + kwargs only. The `custom` provider path (`artemis/llm/router.py`) sends + only `model`, `temperature`, `max_tokens`, `timeout` (+`reasoning_effort` + if set) — thinking fields are silently dropped for local models. +- Gemma-family local endpoints can emit malformed tool-call envelopes + (`call:name{...}`); the Flash runner has relaxed parsing + (`_GEMMA_CALL_RE`, `_loads_relaxed_args`) — keep new parsing tolerant. + +## PR Scope Discipline + +- Keep PRs single-topic. The local-model PR is "Artemis Flash improvement + using the on-device model" — general Artemis bug fixes and improvements + do not belong in it, even when they are real bugs. +- For a legitimate but unrelated issue: file a GitHub issue on + google/artemis, preserve the patch/tests outside the PR, and handle it + in a separate PR. Example: compact-XML `checked`/`enabled` omission is + issue #187, deliberately excluded from PR #176. +- Shared components (formatter, planner, operator prompts, Android + driver, graph) are not Flash scope — changes there need their own + justification. + +## Local-only files + +- `config/artemis.jsonc` is machine-specific deployment config, marked + `git update-index --assume-unchanged` so local edits are not committed. + Do not overwrite it from HEAD or from staging operations; a recovery + copy lives at `/tmp/artemis-pr176-scope-20261017/local-config-recovery.jsonc`. +- Other expected dirty/untracked files: `uv.lock`, `.angular/`, + `AGENTS.md` (this file, until committed), `GEMMA_FINETUNE.txt`. + +## Flash Runner Architecture (`artemis/agents/flash/runner.py`) + +- Hybrid loop (opt-in): `_hybrid_plan` (VLM milestones) → per-turn + `_hybrid_decide` (constrained action menu, `_HYBRID_MAX_MENU_ELEMENTS`, + meta-actions) → fallback to the full Gemma tool-call path on + low-confidence, no goal target, or repeat streak. +- Repeat machinery: `_element_signature` (strict set equality — + systemui/OCR volatile sources excluded, so a single appended digit on a + dense screen counts as change; `_signatures_similar` remains Jaccard + for ping-pong detection only), dead-verbatim-call suppression, A-B-A + ping-pong detection, streak escalation → honest failure. Before a + 'dead' verdict, `_screens_differ` cross-checks the pre/post screenshots + — a frozen accessibility tree can lie while pixels move (verified: + helper tree frozen across a whole dial run); then one re-observation + covers capture lag. +- Index-target feedback: `target_description` on an index click is + verified against the resolved element (`_description_mismatch_note`). + On disagreement the result names the real element — and when a listed + element confidently matches the description, the tap is refused and + the right index is handed back (`_matching_element_hint`); a + known-wrong tap on a dense pad corrupts state, correction beats + cleanup. Without a confident alternative the tap proceeds with the + note appended. +- Coordinate-target feedback: `[x, y]` taps are hit-tested against the + indexed element list (`_coordinate_target_note` + + `find_element_at_point`). A leaf hit is checked against the + description like an index resolution; a hit on a layout container or + on nothing listed is refused when the description identifies one + element (`_confident_element_hint` — widget words like "switch" + filter same-labelled nodes by class), else noted as a likely dead + tap. Container hits that agree with the description stay untouched — + dial-pad keys are clickable FrameLayouts. Verified live: a tap aimed + at a DeskClock switch landed on the expanded CardView's dead space + five times; the check refuses it and names the switch's index. +- Unsatisfied invariants inject a `[GOAL CHECK]` notice with + `describe_failures` evidence (digit-level prefix diff for text + attributes) before the next action — the model narrates its own plan + over the field's real contents without this. +- Declared conditions: `agent.flash.goal_invariants` (config) → + `execution_setup.goal_invariants` + `state_predicates.py` → + `satisfied`/`unsatisfied`/`unknown` verdicts. All-satisfied suppresses + further actions and can close the run; unsatisfied rejects 'completed' + reports; unknown defers to the checker. Both the pre-action check and + the completion gate re-observe fresh (`_fresh_invariant_elements`) — + post-action captures race UI transitions, so lagged elements must + never decide suppression or release. +- Completion: `_verify_completion` evaluates invariants first, then the + Pro `run_final_check` when `final_check_enabled` — fail-open on + checker errors, findings feed back on rejection + (`_completion_rejections`). + +## Evidence From Live Tests (Oct 2026) + +Measured on the emulator with real local Gemma inference — treat as the +current empirical baseline before proposing model additions: + +- Best arm: Gemma + deterministic state checks (3/4 verified, 4/4 + correct final state). Kev/Laya-style decision models did not improve + completion and made harmful vetoes. +- Gemma's failures were missing-evidence failures (invisible `checked` + state, label-vs-control grounding), not shallow-reasoning failures. +- Deployed E4B emits inline prose reasoning, not `` blocks; + `operator_native_thinking` is always null on the local path. +- Decision models stay optional/shadow until fine-tuned on Artemis + traces; no cloud decision service (Jev) — on-device only. +- Oct 17: goal_invariants verified live — `already_off` closed in 2 + turns (action suppressed, completion released on `checked=false`), + `dark_off` toggled once then released deterministically without the + model checker. Both independently confirmed by ADB oracle. +- Oct 17: vision-token sweep (280/560/1120 × 3 tasks): no accuracy + gradient at this sample size, but vt1120 `dark_off` exposed a real + harness bug — post-action XML capture raced the theme transition, so + the pre-action check evaluated lagged `checked` and never suppressed; + four taps ping-ponged the switch. Fixed by re-observing fresh before + every invariant decision (pre-action and completion gate). Post-fix + vt1120 runs closed in 2 turns. Model addressing varies run to run — + index clicks and normalized-0-1000 coordinate clicks both resolve + correctly; latency is turn-count-bound, not token-bound. +- Oct 17: complex-UI milestone — `dial_5550155` on the Android dial pad + (~50 elements, 3 lookalike entries per key, list that grows as + suggestions appear) completed and verified: `digits` field reached + `555-0155`, the text invariant released completion deterministically, + and a satisfied-verdict suppression stopped post-goal actions. + Harness: `/tmp/artemis-live-flash-20261016/run_invariant_task.py` + (task `dial_5550155`, `hierarchy_backend: uiautomator` — the a11y + helper serves a byte-identical frozen tree for dial-pad text updates; + UIAutomator2 stays fresh but makes external `uiautomator dump` exit + 137, so the final oracle falls back to the trace DB's last element + snapshot). Earlier same-task runs showed: frozen helper tree → false + dead-verdicts; Jaccard signature → real digit taps read as unchanged; + index drift + narrated-vs-observed divergence → corrupted fields. + E4B-4B's residual limit is plan-level (digit-sequence transcription, + never choosing backspace), not grounding once the guards hold. +- Oct 18: expanded-card dead zones — DeskClock alarm toggles aimed by + coordinates landed on the CardView body, not the Switch inside it + (verified by trace: 5 taps, `checked` stayed false). Coordinate taps + are now hit-tested and refused-with-hint like index targets. Also + observed: a post-run shell `uiautomator dump` can serve stale node + attributes — it read `checked=false` while the same-moment + screenshot showed the switch on and the in-run captures had been + stable true for ~2 min; DB-backed widgets additionally revert if the + app is force-stopped before their async write flushes. Final-state + oracles should read the run's last live capture (dumps only as + fallback) and never follow a `force-stop`. +- Oct 18: 102-run complex-UI campaign (17 task families × 6 seeds, + resumable driver, trace-primary oracle with dump-vote + corroboration): **54/102 = 52.9% verified end-to-end** by local + Gemma 4 E4B. Per-family rates: alarm toggles 6/6 both directions, + dial_911 5/6, dark theme on/off 5/6 each, stopwatch 5/6, + appinfo_chrome 5/6, nav_battery 4/6, settings_search 4/6, + battery_pct_on 2/6 (system-setting oracle), sms_5550155 2/6, + 7-digit dials 1/6 each, nav_sound/nav_storage 1/6, sms_1234567 0/6. + Failure mix: timeout 25, claimed-without-state 14 (all rejected by + the oracle — e.g. sms_1234567 claimed done 6/6 while never picking + the "Send to" suggestion), dead-tap escalations 7, runner_failed 2. + `oracle_disagreement` fired 29× — post-run dumps genuinely disagree + with live trace state often enough that a single-source oracle + would mis-score runs. Harness: `tests/integration/flash_live/` + (env-parameterized; `python spec.py` once, then `run_batch.py`). +- Oct 18: APC (automatic prefix caching) on the local `mlx_vlm.server` + — `APC_ENABLED=1`, `APC_EXACT_CACHE_ENTRIES=8`. Stock APC salts every + block with the request's media hash, so a fresh screenshot each turn + made reuse exactly 0. Vendored patch adds a media-free companion salt + (`image_hash=0`) with checkpoints stored at (a) the last Gemma + turn-start (`<|turn>` id 105) before the first media span and (b) the + first-media boundary itself; lookups fall back to the media-free salt + bounded at first-media-start (causally valid: pre-image text KV + cannot see the image). To make the transcript prefix-stable, the + observation's screenshot is now the LAST content block — pruning then + removes a trailing span instead of a mid-message one. Measured on + real turns: `cached_tokens≈5231/~8000` (~65% reuse; the cap is the + strip of the second-newest observation each turn), prefill + 16–19s→6–10s, turn cadence ~40–45s→~12–15s. `nav_battery` and a live + dialer run verified correct on the patched server. Diagnostics: + `MLXVLM_APC_DEBUG=1` (server) and `ARTEMIS_DEBUG_PROMPTS=` + (per-turn per-block prompt digests for divergence localization). + Deeper reuse needs media-aware checkpoints over retained images — + deferred; the strip-point cap is structural while old observations + are rewritten. +- Oct 18: APC + 900s-budget re-run (17 families × 2 seeds, same + suite_spec/oracles): **23/34 = 67.6% verified** vs the 52.9% + pre-APC baseline. Zero timeouts — with 900s budgets the wall clock + can never fire before `max_turns=18`; wandering runs now end as + honest `runner_failed` status (model self-reports after exhausting + productive actions) instead of being cancelled mid-flight. Per-family + at 2/2: alarms both directions, appinfo, battery_pct, both dark + themes, dial_911, nav_battery, settings_search, stopwatch. Residual + failures concentrate exactly where the plan-level limit already + showed: 7-digit dials 0/6 (`55500155`/`555055` accumulation), + sms_1234567 0/2 (claimed while "Send to" suggestion untapped), + nav_sound/nav_storage 1/2 (post-launch confusion). diff --git a/README.md b/README.md index 8683bcce3..03d9ffd04 100644 --- a/README.md +++ b/README.md @@ -301,7 +301,7 @@ ARTEMIS supports two execution profiles tailored for different automation requir - [ ] **Android Studio Integration**: Native IDE plugin and workflow integration to enable in-editor debugging, test recording, and automated device control directly within Android Studio. - [ ] **iOS Platform Expansion**: Extending multimodal perception and mobile automation to iOS devices and simulators. -- [ ] **On-Device Lightweight VLMs**: Local execution with lightweight edge vision models for low-latency, privacy-first automation. +- [x] **On-Device Lightweight VLMs**: Local execution with lightweight edge vision models for low-latency, privacy-first automation via OpenAI-compatible local providers — see the [local models guide](./docs/local_models.md) for install/pull/run steps (`config/artemis.gemma4.jsonc` is the tested Gemma 4 E4B setup). - [ ] **Real-time Duplex Voice Interaction**: Voice-driven task dispatch with real-time conversational control and interruption handling. ## Community & Contributing diff --git a/apps/admin_console/routers/system.py b/apps/admin_console/routers/system.py index c8ce18343..be27dbeef 100644 --- a/apps/admin_console/routers/system.py +++ b/apps/admin_console/routers/system.py @@ -16,6 +16,7 @@ import ipaddress import os +import re import secrets from urllib.parse import urlsplit @@ -624,3 +625,188 @@ async def _shutdown_after_response() -> None: "message": "Artemis server is shutting down.", "pid": os.getpid(), } + + +# --- Local model lifecycle (artemis model pull/serve/stop) ------------------ +# +# The console surfaces the local_model_endpoint probe's guided actions; these +# endpoints let the user run them without leaving the browser. Each action +# spawns `python -m artemis model ...` and streams its output into a capped +# in-memory log the frontend polls. + + +class LocalModelActionRequest(BaseModel): + """Payload for a local-model lifecycle action.""" + + alias: str = Field(..., description="Catalog alias or org/repo id (e.g. gemma4-e4b)") + port: int | None = Field( + default=None, + ge=1, + le=65535, + description="Serve/stop port; the CLI default (8080) applies when omitted", + ) + + +_MODEL_TASKS: dict[str, dict] = {} +_MODEL_TASK_LOG_LINES = 80 + + +def _model_cli_available() -> bool: + """Whether the ``artemis model`` command group exists in this build.""" + import importlib.util + + try: + return importlib.util.find_spec("artemis.interfaces.cli.commands.model") is not None + except Exception: + return False + + +def _task_view(key: str, task: dict) -> dict: + return { + "key": key, + "action": task["action"], + "alias": task["alias"], + "cmd": task["cmd"], + "status": task["status"], + "returncode": task["returncode"], + "started_at": task["started_at"], + "log": task["log"][-12:], + } + + +def _task_snapshot() -> dict: + return {k: _task_view(k, t) for k, t in _MODEL_TASKS.items()} + + +async def _drain_model_task(task: dict) -> None: + proc = task["process"] + try: + if proc.stdout is not None: + async for raw in proc.stdout: + line = raw.decode(errors="replace").rstrip() + if line: + task["log"].append(line) + del task["log"][:-_MODEL_TASK_LOG_LINES] + rc = await proc.wait() + except Exception as e: # pylint: disable=broad-exception-caught + rc = -1 + task["log"].append(f"task monitor error: {e}") + task["returncode"] = rc + task["status"] = "done" if rc == 0 else "failed" + + +async def _start_model_task(action: str, alias: str, extra_args: list[str]) -> dict: + """Spawn `artemis model ` as a subprocess with a capped log.""" + import asyncio + import sys + import time + + key = f"{action}:{alias}" + existing = _MODEL_TASKS.get(key) + if existing and existing["status"] == "running": + return _task_view(key, existing) + + # ``stop`` is port-scoped on the CLI (no positional model arg). + cmd = [sys.executable, "-m", "artemis", "model", action] + if action != "stop": + cmd.append(alias) + cmd += extra_args + try: + proc = await asyncio.create_subprocess_exec( + *cmd, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + ) + except Exception as e: + raise HTTPException( + status_code=500, detail=f"Failed to start `artemis model {action}`: {e}" + ) from e + + task = { + "action": action, + "alias": alias, + "cmd": " ".join(cmd), + "status": "running", + "returncode": None, + "started_at": time.time(), + "log": [], + "process": proc, + } + _MODEL_TASKS[key] = task + task["watcher"] = asyncio.create_task(_drain_model_task(task)) + return _task_view(key, task) + + +def _require_model_cli() -> None: + if not _model_cli_available(): + raise HTTPException( + status_code=501, + detail=( + "`artemis model` commands are not available in this build. " + "Run the shown command in a terminal, or manage the model with " + "your own OpenAI-compatible server." + ), + ) + + +_MODEL_REF_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]*(/[A-Za-z0-9][A-Za-z0-9_.-]*)?$") + + +def _require_model_ref(alias: str) -> None: + """Reject aliases the managed CLI could never resolve. + + ``resolve_model_ref`` accepts catalog aliases and ``org/repo`` ids — the + same inputs ``artemis model pull`` takes. Anything else (bare junk, flag + injection like ``--port``, or argv that doesn't look like a model ref) is + refused before a subprocess is spawned. + """ + from artemis.config.local_models import resolve_model_ref + + if not _MODEL_REF_RE.match(alias) or resolve_model_ref(alias) is None: + raise HTTPException( + status_code=400, + detail=( + f"Unknown model '{alias}'. `artemis model` accepts catalog " + "aliases (see `artemis model list`) or a Hugging Face " + "'org/repo' id." + ), + ) + + +@router.get("/local-models/status") +async def local_model_status(): + """Command availability plus in-flight/finished model task states.""" + return { + "commands_available": _model_cli_available(), + "tasks": _task_snapshot(), + } + + +@router.post("/local-models/pull") +async def pull_local_model(payload: LocalModelActionRequest, request: Request): + """Download model weights via `artemis model pull` (long-running).""" + _require_loopback_request(request, "Local model lifecycle is local-only.") + _require_model_cli() + alias = payload.alias.strip() + _require_model_ref(alias) + return await _start_model_task("pull", alias, []) + + +@router.post("/local-models/serve") +async def serve_local_model(payload: LocalModelActionRequest, request: Request): + """Start the managed model server via `artemis model serve`.""" + _require_loopback_request(request, "Local model lifecycle is local-only.") + _require_model_cli() + alias = payload.alias.strip() + _require_model_ref(alias) + extra = ["--port", str(payload.port)] if payload.port else [] + return await _start_model_task("serve", alias, extra) + + +@router.post("/local-models/stop") +async def stop_local_model(payload: LocalModelActionRequest, request: Request): + """Stop a managed model server via `artemis model stop`.""" + _require_loopback_request(request, "Local model lifecycle is local-only.") + _require_model_cli() + extra = ["--port", str(payload.port)] if payload.port else [] + return await _start_model_task("stop", payload.alias.strip(), extra) diff --git a/apps/showcase_ui/src/app/pages/home/home.component.html b/apps/showcase_ui/src/app/pages/home/home.component.html index 060a01d25..ef11bb5a4 100644 --- a/apps/showcase_ui/src/app/pages/home/home.component.html +++ b/apps/showcase_ui/src/app/pages/home/home.component.html @@ -349,7 +349,7 @@

AI Model Setup

Custom Configuration

Define models in config/artemis.jsonc & keys in .env - (OpenAI, Claude, Ollama).

+ (OpenAI, Claude, Ollama — or a local model via artemis model serve).

@@ -604,6 +604,87 @@

AI Model Setup

} + + @if (localModelProbe(); as lm) { + @if (lm.status !== 'pass' || modelConfigEnv()?.default_model?.provider === 'custom') { +
+
+
+ dns +
+ Local Model Endpoint + {{ lm.summary }} +
+
+
+ @if (lm.status === 'pass') { + + check_circle + Ready + + } @else { + + error + Action Needed + + } +
+
+ +

{{ lm.description }}

+ + + @for (act of lm.actions; track act.label) { + @if (act.action_type === 'command') { +
+ {{ act.payload }} + @if (modelActionArgs(act.payload); as args) { + @if (localModelCommandsAvailable()) { + @if (localModelTaskFor(args.action, args.alias)?.status === 'running') { + progress_activity + Running… + } @else { + + } + } + } + +
+ @if (modelActionArgs(act.payload); as args) { + @if (localModelTaskFor(args.action, args.alias); as task) { + @if (task.status === 'failed' || task.log.length) { +
+ @if (task.status === 'done') { +
✓ {{ act.label }} finished
+ } + @if (task.status === 'failed') { +
✗ {{ act.label }} failed (exit {{ task.returncode }}) — check the log below or run the command in a terminal
+ } + @for (line of task.log.slice(-4); track $index) { +
{{ line }}
+ } +
+ } + } + } + } @else { +

{{ act.label }}: {{ act.payload }}

+ } + } +
+ } + } +
diff --git a/apps/showcase_ui/src/app/pages/home/home.component.scss b/apps/showcase_ui/src/app/pages/home/home.component.scss index c1a5e68d6..8d426c814 100644 --- a/apps/showcase_ui/src/app/pages/home/home.component.scss +++ b/apps/showcase_ui/src/app/pages/home/home.component.scss @@ -1421,6 +1421,18 @@ $text-light: #94a3b8; border-radius: 10px; overflow: hidden; + .inspector-desc { + font-size: 12px; + line-height: 1.5; + color: $text-secondary; + margin: 0; + padding: 10px 14px 0; + } + + > .code-box { + margin: 10px 14px; + } + .inspector-card-header { display: flex; align-items: center; @@ -3778,6 +3790,89 @@ $text-light: #94a3b8; } } } + + .btn-run-cmd { + display: inline-flex; + align-items: center; + justify-content: center; + gap: 3px; + padding: 0 8px; + height: 22px; + background: rgba(56, 189, 248, 0.25); + color: #7dd3fc !important; + border: 1px solid rgba(56, 189, 248, 0.4); + border-radius: 5px; + font-family: $font-sans; + font-size: 10.5px; + font-weight: 600; + cursor: pointer; + transition: all 0.18s ease; + flex-shrink: 0; + + .material-symbols-outlined, span { + color: #7dd3fc !important; + } + + .material-symbols-outlined { + font-size: 13px; + } + + &:hover { + background: rgba(56, 189, 248, 0.4); + border-color: rgba(56, 189, 248, 0.6); + } + + &:active { + transform: scale(0.96); + } + } + + .lm-spin { + font-size: 15px; + color: #38bdf8; + flex-shrink: 0; + animation: spin 1s linear infinite; + } + + .lm-running-label { + font-size: 10.5px; + font-weight: 600; + color: #7dd3fc; + flex-shrink: 0; + white-space: nowrap; + } + } + + .lm-task-log { + margin-top: 6px; + padding: 8px 10px; + background: #0f172a; + border-radius: 6px; + font-family: $font-mono; + font-size: 10.5px; + line-height: 1.5; + max-height: 110px; + overflow-y: auto; + + .lm-log-line { + color: #94a3b8; + white-space: pre-wrap; + word-break: break-all; + } + + .lm-log-done { + color: #34d399; + font-weight: 600; + } + + .lm-log-failed { + color: #f87171; + font-weight: 600; + } + + &.lm-task-failed { + border: 1px solid rgba(248, 113, 113, 0.35); + } } /* ======================================================================= */ diff --git a/apps/showcase_ui/src/app/pages/home/home.component.ts b/apps/showcase_ui/src/app/pages/home/home.component.ts index e189b9ce5..b7446bc73 100644 --- a/apps/showcase_ui/src/app/pages/home/home.component.ts +++ b/apps/showcase_ui/src/app/pages/home/home.component.ts @@ -358,6 +358,9 @@ export class HomeComponent implements OnInit, OnDestroy { public configProbe = computed(() => this.systemService.configProbe()); public adbProbe = computed(() => this.systemService.adbProbe()); public llmProbe = computed(() => this.systemService.llmProbe()); + public localModelProbe = computed(() => this.systemService.localModelProbe()); + public localModelTasks = computed(() => this.systemService.localModelTasks()); + public localModelCommandsAvailable = computed(() => this.systemService.localModelCommandsAvailable()); public geminiProbe = computed(() => this.systemService.geminiProbe()); public ocrProbe = computed(() => this.systemService.ocrProbe()); public toolchainProbe = computed(() => this.systemService.toolchainProbe()); @@ -541,6 +544,7 @@ export class HomeComponent implements OnInit, OnDestroy { // Initial fetch of system readiness & model configuration this.systemService.fetchReadiness().subscribe(); this.systemService.fetchModelConfigEnv().subscribe(); + this.systemService.fetchLocalModelStatus().subscribe(); this.systemService.fetchAdbServerStatus().subscribe({ next: status => { if (status.endpoint.mode === 'remote') { @@ -564,6 +568,7 @@ export class HomeComponent implements OnInit, OnDestroy { if (tab === 'diagnostics') { this.systemService.fetchReadiness().subscribe(); this.systemService.fetchModelConfigEnv().subscribe(); + this.systemService.fetchLocalModelStatus().subscribe(); } } @@ -1035,6 +1040,35 @@ export class HomeComponent implements OnInit, OnDestroy { this.selectedProfile.set(profile); } + // --- Local model lifecycle (artemis model pull/serve/stop) --------------- + + /** Parse an `artemis model [--port N]` action payload. */ + public modelActionArgs(payload: string): { action: 'pull' | 'serve' | 'stop'; alias: string; port?: number } | null { + const m = payload.match(/artemis\s+model\s+(pull|serve|stop)\s+([^\s]+)/); + if (!m) return null; + const portMatch = payload.match(/--port\s+(\d+)/); + return { + action: m[1] as 'pull' | 'serve' | 'stop', + alias: m[2], + port: portMatch ? Number(portMatch[1]) : undefined, + }; + } + + /** Latest task state for an action+alias pair, if any. */ + public localModelTaskFor(action: string, alias: string) { + return this.localModelTasks()[`${action}:${alias}`] || null; + } + + public runLocalModelAction(args: { action: 'pull' | 'serve' | 'stop'; alias: string; port?: number }): void { + if (args.action === 'pull') { + this.systemService.pullLocalModel(args.alias).subscribe(); + } else if (args.action === 'serve') { + this.systemService.serveLocalModel(args.alias, args.port).subscribe(); + } else { + this.systemService.stopLocalModel(args.alias, args.port).subscribe(); + } + } + public setCategory(cat: SuggestionCategory): void { this.selectedCategory.set(cat); } diff --git a/apps/showcase_ui/src/app/services/system.service.ts b/apps/showcase_ui/src/app/services/system.service.ts index b69697d9f..1cb5b49e9 100644 --- a/apps/showcase_ui/src/app/services/system.service.ts +++ b/apps/showcase_ui/src/app/services/system.service.ts @@ -58,6 +58,7 @@ export class SystemService { public adbProbe = computed(() => this.probes().find(p => p.id === 'android_adb') || null); public llmProbe = computed(() => this.probes().find(p => p.id === 'gemini_api_key' || p.id === 'llm_api_key') || null); public geminiProbe = computed(() => this.llmProbe()); + public localModelProbe = computed(() => this.probes().find(p => p.id === 'local_model_endpoint') || null); public ocrProbe = computed(() => this.probes().find(p => p.id === 'vision_ocr_key' || p.id === 'ocr_api_key') || null); // Grouped readiness helpers for the 3-step onboarding flow @@ -80,7 +81,12 @@ export class SystemService { return true; } const llm = this.llmProbe(); - return llm?.status === 'pass'; + // The local-endpoint probe is a blocker server-side: when artemis.jsonc + // binds a node to a custom api_base (Artemis-served or BYO), a dead + // endpoint or missing model must gate the run the same way a missing + // API key does. It self-passes ("Not used") for cloud-only configs. + const local = this.localModelProbe(); + return llm?.status === 'pass' && (local == null || local.status === 'pass'); }); public isDeviceReady = computed(() => { @@ -314,6 +320,79 @@ export class SystemService { } } + // --- Local model lifecycle (artemis model pull/serve/stop) --------------- + + public localModelCommandsAvailable = signal(null); + public localModelTasks = signal>({}); + private localModelPollTimer: any = null; + + public fetchLocalModelStatus(): Observable { + return this.http.get('/api/system/local-models/status').pipe( + tap({ + next: (resp) => { + this.localModelCommandsAvailable.set(resp.commands_available); + this.localModelTasks.set(resp.tasks || {}); + const anyRunning = Object.values(resp.tasks || {}).some(t => t.status === 'running'); + if (!anyRunning) { + this.stopLocalModelPolling(); + } + } + }) + ); + } + + public startLocalModelPolling(): void { + if (this.localModelPollTimer) { + clearInterval(this.localModelPollTimer); + } + this.localModelPollTimer = setInterval(() => { + this.fetchLocalModelStatus().subscribe({ error: () => {} }); + }, 2000); + } + + public stopLocalModelPolling(): void { + if (this.localModelPollTimer) { + clearInterval(this.localModelPollTimer); + this.localModelPollTimer = null; + // A finished action may have changed model readiness — re-check. + this.fetchReadiness().subscribe({ error: () => {} }); + } + } + + private runLocalModelAction( + endpoint: 'pull' | 'serve' | 'stop', + alias: string, + port?: number + ): Observable { + return this.http + .post(`/api/system/local-models/${endpoint}`, { alias, port: port ?? null }) + .pipe( + tap({ + next: (task) => { + this.localModelTasks.update(tasks => ({ ...tasks, [task.key]: task })); + if (task.status === 'running') { + this.startLocalModelPolling(); + } else { + this.fetchReadiness().subscribe({ error: () => {} }); + } + }, + error: () => {} + }) + ); + } + + public pullLocalModel(alias: string): Observable { + return this.runLocalModelAction('pull', alias); + } + + public serveLocalModel(alias: string, port?: number): Observable { + return this.runLocalModelAction('serve', alias, port); + } + + public stopLocalModel(alias: string, port?: number): Observable { + return this.runLocalModelAction('stop', alias, port); + } + /** * Launch a local AVD emulator in the background and continuously track boot progress */ @@ -608,3 +687,19 @@ export interface ModelConfigEnvResponse { }>; } + +export interface LocalModelTask { + key: string; + action: 'pull' | 'serve' | 'stop' | string; + alias: string; + cmd: string; + status: 'running' | 'done' | 'failed' | string; + returncode: number | null; + started_at: number; + log: string[]; +} + +export interface LocalModelStatusResponse { + commands_available: boolean; + tasks: Record; +} diff --git a/artemis/agents/checker/checker.py b/artemis/agents/checker/checker.py index 52dff2ca2..ad474395b 100644 --- a/artemis/agents/checker/checker.py +++ b/artemis/agents/checker/checker.py @@ -67,6 +67,7 @@ ) from artemis.utils.visualization import format_minimal_list_with_elements from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -836,15 +837,18 @@ async def run_final_check( f"# Final Screen Elements\n{minimal_list}" ) - content: list = [{"type": "text", "text": human_text}] + content: list = [] if screenshot_b64: - content.append({"type": "text", "text": "--- Final Screenshot ---"}) + # Image-first ordering: vision models ground better when the + # screenshot precedes the instruction/context text. content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"}, + "image_url": {"url": image_data_uri(screenshot_b64)}, } ) + content.append({"type": "text", "text": "--- Final Screenshot ---"}) + content.append({"type": "text", "text": human_text}) messages = [ SystemMessage(content=system_content), diff --git a/artemis/agents/diagnoser/diagnoser.py b/artemis/agents/diagnoser/diagnoser.py index 13d80317c..d94376a5d 100644 --- a/artemis/agents/diagnoser/diagnoser.py +++ b/artemis/agents/diagnoser/diagnoser.py @@ -40,6 +40,7 @@ from artemis.tools.log_tool import get_analyze_logs_tool from artemis.tools.mobile.read_hierarchy import get_ui_hierarchy_tool from artemis.tools.scratchpad import get_list_notes_tool, get_read_note_tool +from artemis.utils.image_codec import image_data_uri from artemis.tools.tool_wrapper import ( get_tool_result_content, invoke_tool_with_injection, @@ -227,7 +228,7 @@ def _build_initial_messages( content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"}, + "image_url": {"url": image_data_uri(latest_screenshot_b64)}, } ) diff --git a/artemis/agents/explorer/universal_runner.py b/artemis/agents/explorer/universal_runner.py index c5b5bc89a..633be4eb9 100644 --- a/artemis/agents/explorer/universal_runner.py +++ b/artemis/agents/explorer/universal_runner.py @@ -36,6 +36,7 @@ from artemis.graph.state import State from artemis.services.llm import get_llm from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -161,6 +162,10 @@ def _build_universal_messages( img_b64 = base64.b64encode(f.read()).decode("utf-8") user_content: list[dict[str, Any]] = [ + { + "type": "image_url", + "image_url": {"url": image_data_uri(img_b64)}, + }, { "type": "text", "text": ( @@ -170,10 +175,6 @@ def _build_universal_messages( f"{minimal_list}\n" ), }, - { - "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}, - }, ] messages: list[BaseMessage] = [ @@ -266,16 +267,14 @@ def _build_tool_images_message(self, images: list[tuple[str, str]]) -> HumanMess continue if tool_name not in tool_names: tool_names.append(tool_name) - blocks.append( - {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}} - ) + blocks.append({"type": "image_url", "image_url": {"url": image_data_uri(img_b64)}}) if not blocks: return None header = { "type": "text", "text": f"[Annotated image(s) returned by: {', '.join(tool_names)}]", } - return HumanMessage(content=[header, *blocks]) + return HumanMessage(content=[*blocks, header]) async def _universal_dispatch_tools( self, tool_calls: list, messages: list[BaseMessage], turn: int diff --git a/artemis/agents/flash/runner.py b/artemis/agents/flash/runner.py index 3ea3f6865..56c6d6141 100644 --- a/artemis/agents/flash/runner.py +++ b/artemis/agents/flash/runner.py @@ -41,6 +41,7 @@ import base64 from dataclasses import dataclass, field import json +import os from pathlib import Path import re import uuid @@ -70,17 +71,27 @@ MemoryTranscriptConfig, StepSummarizerConfig, load_agent_config, + settings, ) from artemis.context import ArtemisContext from artemis.controllers.unified_controller import UnifiedMobileController from artemis.data_engine.trace import trace from artemis.graph.perception import _check_injected_instruction_file from artemis.graph.state import State +from artemis.llm.router import ModelProvider from artemis.llm.structured import ParseFailure, parse_structured +from artemis.core.decision_client import DecisionClient +from artemis.agents.flash.state_predicates import ( + describe_failures, + evaluate_invariants, +) from artemis.mcp.action_executor import McpActionExecutor +from artemis.utils.image_codec import image_data_uri +from artemis.utils.image_hash import dhash_hex, hamming_distance_hex from artemis.mcp.observation import observe from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger, mark_ephemeral from artemis.services.llm import ( + LLMCallError, RobustChatModelWrapper, acomplete, get_google_llm, @@ -103,8 +114,133 @@ " action tool or 'report_task_status'." ) +# Hard ceiling on consecutive narration-only turns. Some local VLMs (e.g. +# Gemma 4 E4B under mlx_vlm) can fall into an apology loop — seeing the +# goal, narrating a plan, never emitting a tool call — and with an +# unlimited turn budget that loop never exits on its own. +_MAX_CONSECUTIVE_NO_TOOL_CALLS = 8 + +# Gemma-family tool-call envelopes observed in the wild: the model emits +# `call:{}` and some servers (mlx_vlm) wrap it into a function +# literally named "call" whose arguments are the unparseable remainder, or +# the envelope leaks into text content. +_GEMMA_CALL_RE = re.compile(r"(?:<\|?tool_call\|?>?\s*)?call:([a-zA-Z_]\w*)\s*\{") +_GEMMA_ENVELOPED_RE = re.compile(r'\s*\{?\s*"?([a-zA-Z_]\w*)"?\s*\{') +_GEMMA_QUOTE_RE = re.compile(r'<\|\\?\"\|>') + + +def _loads_relaxed_args(raw: str): + """Best-effort JSON parse of a mangled Gemma `name{key:value}` args object.""" + s = raw.strip() + s = _GEMMA_QUOTE_RE.sub('"', s) + # Quote bare keys: {key: -> {"key": and ,key: -> ,"key": + s = re.sub(r'([{,])\s*"?([a-zA-Z_]\w*)"?\s*:', r'\1"\2":', s) + try: + return json.loads(s) + except ValueError: + return None + + +def _parse_call_syntax(raw: str, match=None): + """Recovers `name{args}` at ``match`` — `call:name{...}` or `{name{...}`.""" + if match is None: + match = _GEMMA_ENVELOPED_RE.match(raw) + if not match: + return None + name = match.group(1) + start = match.end() - 1 # opening '{' of the args object + args = None + depth = 0 + for i in range(start, len(raw)): + ch = raw[i] + if ch == '{': + depth += 1 + elif ch == '}': + depth -= 1 + if depth == 0: + args = _loads_relaxed_args(raw[start:i + 1]) + break + if depth: + args = _loads_relaxed_args(raw[start:] + '}') + if not isinstance(args, dict): + return None + return {"name": name, "args": args, "id": str(uuid.uuid4())} + + _FINAL_TURN_WARNING = "[WARNING] This is your final turn; only 'report_task_status' is available." +# Hybrid Flash loop (decision.flash_loop): the VLM perceives, the typed +# decision model acts. The describe call carries NO tool declarations, so +# the tool-call envelope syntax that local VLMs mangle can never be +# emitted; action selection is a fixed-space `choose` over the live +# element list plus meta-actions instead. +_HYBRID_PLAN_PROMPT = """You are the planning stage of a UI testing agent. +GOAL: {{ goal }} + +Given the attached CURRENT screenshot and element list, decompose the goal +into 3-6 short ordered milestones (one per line, starting with '- '). Each +milestone is a SCREEN STATE to reach or a concrete interaction ON THE GOAL +PATH — e.g. "open the app drawer (swipe up on home)", "Settings main list +open", "Dark theme toggle enabled". Only include steps that move toward the +goal; NEVER enumerate visible elements unrelated to it (do not list "tap +" for every icon — a milestone is progress, not an inventory). +If the first target is not visible, the first milestone is how to reach it. +Output only the milestone list.""" + +_HYBRID_DESCRIBE_PROMPT = """You are the perception stage of a UI testing agent. +GOAL: {{ goal }} +PLAN: +{{ plan }} + +Describe the CURRENT screenshot factually in 3-5 short sentences: which app +and screen is shown, what has changed since the previous observation, and +where the elements relevant to the goal sit. If the goal's next target is +NOT visible, say so and note how it would be reached (e.g. open the app +drawer, scroll, navigate back). When a goal-relevant toggle, switch, or +control is visible, state its CURRENT value explicitly (on/off, checked, +enabled, the shown selection) — and if the goal state is already satisfied, +say so plainly rather than describing the screen generically. Then list +EVERY interactive element from the attached list, one per line: [index] +label ~[x,y] — the approximate normalized 0-1000 coordinate is only +required for elements visible on screen but NOT in the list. End with +exactly one marker line: +TARGET: visible — the goal's next interactive target is on this screen +TARGET: not-visible — it is not; navigation is needed first +TARGET: reached — the goal state is already achieved +Never claim an action was taken. Do not decide what to do next — a +separate decision stage reads your description.""" + +# Menu entries the decision model may always pick, alongside the element +# taps generated from the live index list. Free-text input is excluded by +# design: a fixed answer space cannot produce the text to type, so a goal +# that needs text input escalates to the standard tool-calling turn. +_HYBRID_META_ACTIONS = { + "swipe_up": "scroll the screen up to reveal content below", + "swipe_down": "scroll the screen down to reveal content above", + "swipe_left": "swipe left (e.g. carousel, page forward)", + "swipe_right": "swipe right (e.g. carousel, page back)", + "press_back": "press the Android back button", + "press_home": "press the Android home button", + "wait": "wait briefly for the screen to settle or load", + "done": "the goal is satisfied — finish the task as completed", + "fail": "the goal cannot be achieved from here — finish as failed", +} + +# Cap on element-tap entries in the decision menu: the decision model's +# context is short (~512-1024 tokens) and taps beyond the first screenful +# are rarely the right action anyway. +_HYBRID_MAX_MENU_ELEMENTS = 15 + +# Consecutive turns that re-executed an already-tried menu key before the +# hybrid decide is skipped entirely and the turn escalates to the VLM — +# the masked menu plus the notice usually break the loop first, but the +# standard path is the fallback when even that fails. +_HYBRID_REPEAT_ESCALATE = 2 + +# Distilled state sent to the decision model is capped for the same reason: +# goal + plan + screen description, truncated tail-first. +_HYBRID_MAX_STATE_CHARS = 2000 + @dataclass class _TurnRecord: @@ -178,6 +314,19 @@ def __init__(self, ctx: ArtemisContext, goal: str, max_turns: int | None = None) ctx.step_memory = self.summarizer except (AttributeError, TypeError, ValueError): pass + # Terminal-boundary verification state: the hybrid plan text (for the + # final check's context) and how many 'completed' reports the checker + # has already rejected this run. + self._plan_text: str | None = None + self._completion_rejections = 0 + # Declared success conditions (execution_setup.goal_invariants): when + # every one evaluates satisfied against the live element list the run + # may close early; an unsatisfied one vetoes 'completed' reports. + self._goal_invariants: list[dict] = [] + # One-line "last action → outcome" fed into the decision model's + # state: Laya has no transcript view, so without it a dead tap and + # real progress look identical. + self._last_action_note: str | None = None @property def turn_limit(self) -> int | None: @@ -286,6 +435,378 @@ def _init_llm(self): return RobustChatModelWrapper(get_google_llm(model_name="gemini-2.5-flash"), self.ctx) + # ------------------------------------------------------------------ + # Hybrid loop (decision.flash_loop): VLM describes, decision model acts + # ------------------------------------------------------------------ + + async def _hybrid_plan(self, llm, img_bytes=None, xml_list=None) -> str | None: + """The one upfront planning call — Pro's planner leg grafted into Flash. + + A plain completion (no tools): the VLM emits a milestone list that the + per-turn describe prompt carries as context. The call is grounded on + the first observation — a plan made blind degenerates to boilerplate + ("load login screen" for a Settings task, observed live). Failure is + non-fatal — the loop still runs, just without plan guidance. + """ + prompt = Template(_HYBRID_PLAN_PROMPT).render(goal=self.goal) + blocks: list[dict] = [{"type": "text", "text": prompt}] + if img_bytes: + blocks.append({"type": "text", "text": "--- Current Screenshot ---"}) + blocks.append( + { + "type": "image_url", + "image_url": {"url": image_data_uri(base64.b64encode(img_bytes).decode())}, + } + ) + if xml_list: + blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"}) + try: + response = await invoke_llm_with_timeout_message( + acomplete(llm, [HumanMessage(content=blocks)]), + timeout_seconds=10, + hard_timeout=settings.LLM_FLASH_HARD_TIMEOUT_SECONDS, + ) + except LLMCallError as e: + logger.warning(f"Hybrid plan call failed, running unplanned: {e}") + return None + plan = self._extract_response_text(response).strip() + logger.info(f"Hybrid plan ({len(plan)} chars): {plan[:200]}") + return plan or None + + @staticmethod + def _element_label(el: dict) -> str: + return str( + el.get("text") + or el.get("content_desc") + or el.get("resource_id") + or el.get("class_name") + or "element" + )[:60] + + @staticmethod + def _element_signature(elements) -> frozenset: + """(label, center, state) set used to tell whether the screen changed. + + ``checked``/``enabled`` join the identity so a toggle flipping + ON→OFF counts as a screen change — with geometry alone the dead-tap + machinery treated a working switch tap as a no-op (observed live: + the Dark theme switch flipped while the signature stayed identical). + Volatile attributes (``focused``/``selected``) are excluded: focus + moves on scroll/hover and would fake 'changed' on dead taps. + Inherently volatile elements are excluded entirely — status-bar + items (``com.android.systemui``: clock, badges) tick on their own + and OCR text jitters between dumps — so a strict set comparison + can mean 'unchanged' instead of a similarity ratio that swallows + small real changes on dense screens (observed live: one changed + digit label on the 30-node dial pad read 'dead' at Jaccard 0.8). + """ + return frozenset( + ( + FlashRunner._element_label(el), + tuple(el.get("center") or ()), + el.get("checked"), + el.get("enabled"), + ) + for el in elements + if not el.get("is_ocr") + and el.get("package") != "com.android.systemui" + ) + + @staticmethod + def _signatures_similar(a: frozenset, b: frozenset, threshold: float = 0.8) -> bool: + """Jaccard-similar element signatures — tolerates volatile items like + the clock, so a screen is 'unchanged' only when it really didn't move.""" + union = a | b + if not union: + return True + return len(a & b) / len(union) >= threshold + + @staticmethod + def _hybrid_action_menu( + state: State, + tried_taps: set[tuple[int, str]] | None = None, + tried_labels: set[str] | None = None, + ) -> dict[str, str]: + """The fixed action menu for one turn: element taps + meta-actions. + + Tap keys embed the element index of the CURRENT observation's list — + index drift across turns is impossible because the menu is rebuilt + every turn and the executor resolves the index against the same list + snapshot the decision was made on. + + ``tried_taps`` masks elements already tapped by (index, label) + identity; ``tried_labels`` additionally masks any element whose + distinctive label was tapped before, surviving re-indexing. A screen + signature can't express "same screen" here — volatile text (clocks, + badges) perturbs it — so identity matching is used instead. + """ + elements = getattr(state, "indexed_elements", None) or [] + menu: dict[str, str] = {} + for el in elements[:_HYBRID_MAX_MENU_ELEMENTS]: + idx = el.get("index") + if idx is None: + continue + label = FlashRunner._element_label(el) + if tried_taps and (idx, label) in tried_taps: + continue + if tried_labels and len(label) >= 4 and label in tried_labels: + continue + menu[f"tap_{idx}"] = f"tap element {idx}: {label}" + menu.update(_HYBRID_META_ACTIONS) + return menu + + @staticmethod + def _menu_choice_to_call(choice: str, description: str) -> dict | None: + """Maps a menu key to a tool-call dict for the normal execution path.""" + tid = f"call_{uuid.uuid4().hex[:8]}" + if choice.startswith("tap_"): + try: + idx = int(choice[len("tap_"):]) + except ValueError: + return None + return {"id": tid, "name": "click", "args": {"target": idx}} + if choice in ("swipe_up", "swipe_down", "swipe_left", "swipe_right"): + return { + "id": tid, + "name": "swipe", + "args": {"direction": choice.split("_", 1)[1]}, + } + if choice in ("press_back", "press_home"): + return { + "id": tid, + "name": "press_key", + "args": {"key": "back" if choice == "press_back" else "home"}, + } + if choice == "wait": + return {"id": tid, "name": "wait_for_delay", "args": {"timeout_ms": 1500}} + if choice in ("done", "fail"): + return { + "id": tid, + "name": "report_task_status", + "args": { + "status": "completed" if choice == "done" else "failed", + "explanation": description[:500] or "Decision layer verdict.", + }, + } + return None + + _NAV_LABEL_RE = re.compile( + r"navigate up|back|close|cancel|done|collapse|dismiss|arrow", re.I + ) + + def _navigation_suggestions(self, elements) -> str: + """Labeled elements that look like exits/nav — offered as concrete + alternatives when the model is re-issuing a dead action.""" + hints = [] + for el in elements or []: + idx = el.get("index") + label = self._element_label(el) + if idx is None or not label: + continue + if self._NAV_LABEL_RE.search(label): + hints.append(f"[{idx}] {label}") + return ", ".join(hints[:6]) + + _GOAL_STOPWORDS = frozenset( + { + "the", "and", "for", "with", "app", "open", "into", "enable", + "disable", "turn", "tap", "click", "press", "then", "this", + "that", "screen", "page", "goto", "select", + } + ) + + @classmethod + def _menu_has_goal_target(cls, menu: dict[str, str], goal: str) -> bool: + """True when some tap label shares a meaningful token with the goal.""" + goal_tokens = { + t + for t in re.findall(r"[a-z0-9']+", goal.lower()) + if len(t) >= 4 and t not in cls._GOAL_STOPWORDS + } + if not goal_tokens: + return True + for key, label in menu.items(): + if not key.startswith("tap_"): + continue + label_tokens = set(re.findall(r"[a-z0-9']+", label.lower())) + if label_tokens & goal_tokens: + return True + return False + + @staticmethod + def _menu_key_for_call(name: str, args: dict) -> str | None: + """Maps an executed tool call back to its hybrid menu key, so actions + taken on a screen can be masked from the menu on a later visit.""" + args = args or {} + if name == "click": + target = args.get("target") + return f"tap_{target}" if isinstance(target, int) else None + if name == "swipe": + direction = str(args.get("direction") or args.get("gesture") or "").lower() + return f"swipe_{direction}" if direction in ("up", "down", "left", "right") else None + if name == "press_key": + return {"back": "press_back", "home": "press_home"}.get( + str(args.get("key") or "").lower() + ) + if name == "wait_for_delay": + return "wait" + if name == "report_task_status": + return "done" if args.get("status") == "completed" else "fail" + return None + + async def _hybrid_decide( + self, + llm, + state: State, + tail: HumanMessage, + plan_text: str | None, + cfg, + tried_taps: set[tuple[int, str]] | None = None, + tried_labels: set[str] | None = None, + ) -> tuple | None: + """One hybrid turn: describe → choose → tool call. + + Returns ``(describe_response, [tool_call])`` on a confident decision, + ``None`` to escalate to the standard tool-calling model turn. The + describe call binds no tools — the VLM answers with a screen + narrative (plus approximate coords for unlisted targets) that becomes + the turn's thought text, so envelope-mangling and narration-only + stalls cannot occur on this path. + + ``tried_taps``/``tried_labels`` mask menu entries already attempted — + the decision model has no transcript view and would otherwise re-pick + a dead element indefinitely. Terminal meta-actions + (``wait``/``done``/``fail``) are never masked. + """ + describe_prompt = Template(_HYBRID_DESCRIBE_PROMPT).render( + goal=self.goal, plan=plan_text or "(no plan — describe and list elements)" + ) + try: + response = await invoke_llm_with_timeout_message( + acomplete(llm, [SystemMessage(content=describe_prompt), tail]), + timeout_seconds=10, + hard_timeout=settings.LLM_FLASH_HARD_TIMEOUT_SECONDS, + ) + except LLMCallError as e: + logger.warning(f"Hybrid describe call failed, escalating: {e}") + return None + description = self._extract_response_text(response) + logger.info(f"Hybrid describe: {description[:200]}") + + # Goal-achieved marker: a 'reached' describe is a perception claim, + # not a menu pick — hand the turn to the VLM, which can look at the + # screen itself and call report_task_status. The terminal-boundary + # check then audits that claim against device state, so a premature + # 'reached' costs a verification round-trip, not a false completion. + if re.search(r"(?im)^\s*target:\s*reached\b", description): + logger.info( + "Hybrid describe reports TARGET: reached — escalating to the " + "VLM turn for verification" + ) + return None + + menu = self._hybrid_action_menu( + state, tried_taps=tried_taps, tried_labels=tried_labels + ) + # Visibility gate: the goal's next target must exist among the menu's + # element labels for an in-screen pick to be meaningful at all. Token + # overlap between labels and the goal is the check — describe + # narration can't be trusted (observed: 'TARGET: visible' on a home + # screen listing no Settings). No match → the VLM handles the + # between-screen navigation turn instead. + if not self._menu_has_goal_target(menu, self.goal): + tap_labels = [v for k, v in menu.items() if k.startswith("tap_")] + logger.info( + "Hybrid menu holds no goal-relevant element — escalating " + f"the navigation turn to the VLM path (labels: {tap_labels[:12]})" + ) + return None + tried_line = "" + if tried_taps or tried_labels: + masked = sorted( + set(self._hybrid_action_menu(state)) - set(menu) + ) + if masked: + tried_line = ( + "\nAlready attempted without moving the task forward and" + f" unavailable: {', '.join(masked)}." + ) + client = DecisionClient( + cfg.url, model=cfg.model, timeout_seconds=cfg.timeout_seconds + ) + last_line = ( + f"\nLast action outcome: {self._last_action_note}" + if self._last_action_note + else "" + ) + state_text = ( + f"Goal: {self.goal}\n" + f"Plan: {plan_text or 'none'}\n" + f"Current screen: {description}{tried_line}{last_line}" + ) + verdict = await client.achoose( + state_text[-_HYBRID_MAX_STATE_CHARS:], + f"Which single action best advances this goal: {self.goal}", + menu, + ) + # For `choice` answers the decision gate is the top-pick probability + # (answer_confidence): the calibrated `confidence` field measures overall + # answer trustworthiness and sits near zero on any menu-shaped spread — + # using it would escalate every turn. + top_p = verdict.confidence + raw = getattr(verdict, "raw", None) or {} + try: + top_p = float(raw.get("answer_confidence") or top_p) + except (TypeError, ValueError): + pass + floor = getattr(cfg, "flash_choice_threshold", 0.25) + # Scale the floor to the menu size: on a 20-option menu a uniform + # spread is 0.05, so a 0.20 top pick is 4x uniform — decisive — yet a + # fixed 0.25 gate escalates every real-screen verdict (observed: all + # post-navigation picks landed 0.16–0.23). Require 4x uniform, capped + # at the configured floor, with an absolute minimum of 0.12. + effective_floor = max(0.12, min(floor, 4.0 / max(len(menu), 1))) + if ( + not verdict.answered + or top_p < effective_floor + or verdict.verdict not in menu + ): + logger.info( + f"Hybrid decision inconclusive (verdict={verdict.verdict!r} " + f"top_p={top_p:.2f} floor={effective_floor:.2f}), escalating to tool-call turn" + ) + return None + if verdict.verdict in ("done", "fail"): + # Terminal picks can never execute blind: the decision model sees + # no pixels and cannot verify goal completion — observed live + # declaring 'done' on a screen where the target app wasn't even + # open. The standard turn re-verifies against the screenshot and + # keeps report_task_status available to do it honestly. + logger.info( + f"Hybrid terminal pick {verdict.verdict!r} (top_p={top_p:.2f}) " + "— escalating to the VLM turn for verification" + ) + return None + tool_call = self._menu_choice_to_call(str(verdict.verdict), description) + if tool_call is None: + return None + logger.info( + f"Hybrid decision: {tool_call['name']} (top_p={top_p:.2f})" + ) + # Stamp the chosen call onto the describe message: the ToolMessages the + # executor appends then have a legitimate parent tool_call, which strict + # providers require when the ledger replays them into later prompts. + response.tool_calls = [ + { + "name": tool_call["name"], + "args": tool_call["args"], + "id": tool_call["id"], + "type": "tool_call", + } + ] + response.invalid_tool_calls = [] + return response, [tool_call] + def _render_system_prompt(self, tools_declaration: list) -> str: """Renders the system prompt from the flash_runner.md template. @@ -366,12 +887,6 @@ def _build_tail( blocks: list[dict] = [ {"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"} ] - if img_bytes: - img_b64 = base64.b64encode(img_bytes).decode("utf-8") - blocks.append({"type": "text", "text": "--- Current Screenshot ---"}) - blocks.append( - {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}} - ) if xml_list: blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"}) ephemeral: list[int] = [] @@ -390,6 +905,13 @@ def _build_tail( if is_ephemeral: ephemeral.append(len(blocks)) blocks.append({"type": "text", "text": text}) + # The screenshot goes last so intermediate-image pruning removes a + # trailing span: the surviving text prefix stays byte-identical across + # turns, which prefix-caching servers (APC) can then reuse. + if img_bytes: + img_b64 = base64.b64encode(img_bytes).decode("utf-8") + blocks.append({"type": "text", "text": "--- Current Screenshot ---"}) + blocks.append({"type": "image_url", "image_url": {"url": image_data_uri(img_b64)}}) tail = HumanMessage(content=blocks) if ephemeral: mark_ephemeral(tail, ephemeral) @@ -534,8 +1056,40 @@ def _resolve_tool_calls(self, response, raw_text: str) -> list: "id": str(uuid.uuid4()), } ] + + # Local-server envelope repair: mlx_vlm can wrap the model's native + # `call:name{args}` output into a function literally named "call" whose + # arguments are the unparseable remainder — langchain then drops it to + # invalid_tool_calls and the call is silently lost. + if not tool_calls: + tool_calls = self._repair_enveloped_tool_calls( + getattr(response, "invalid_tool_calls", None) + ) + + # Same envelope can leak into text content outright. + if not tool_calls and raw_text: + call_match = _GEMMA_CALL_RE.search(raw_text) + if call_match: + repaired = _parse_call_syntax(raw_text, match=call_match) + if repaired: + tool_calls = [repaired] return tool_calls + @staticmethod + def _repair_enveloped_tool_calls(invalid_tool_calls) -> list: + """Recovers calls mangled into `name="call"` + raw `name{args}` args.""" + repaired = [] + for tc in invalid_tool_calls or []: + name = tc.get("name") + raw_args = tc.get("args") + if name != "call" or not isinstance(raw_args, str): + continue + parsed = _parse_call_syntax(raw_args) + if parsed: + parsed["id"] = tc.get("id") or parsed["id"] + repaired.append(parsed) + return repaired + # ------------------------------------------------------------------ # Per-tool-call helpers (act / record / report) # ------------------------------------------------------------------ @@ -554,21 +1108,15 @@ async def _finalize_task_report( ) -> dict: """Records the final step, acknowledges the tool call, and flushes.""" final_report = args - if self.ctx.data_engine: - try: - if self.ctx.data_engine.current_step_id is None: - self.ctx.data_engine.allocate_step_id() - self.ctx.data_engine.record_step( - pre_screenshot_bytes=pre_screenshot_bytes, - ui_tree=xml_list, - action_taken={"action": "report_task_status", "args": args}, - operator_raw_thinking=raw_text, - operator_native_thinking=native_text or None, - last_execution_result={"result": "Task completed with final report."}, - extra_metadata={"token_usage": step_token_usage}, - ) - except Exception as step_err: - logger.warning(f"Error recording final step in FlashRunner: {step_err}") + self._record_terminal_step( + args, + raw_text, + step_token_usage, + pre_screenshot_bytes, + xml_list, + last_execution_result={"result": "Task completed with final report."}, + native_text=native_text, + ) messages.append( ToolMessage( tool_call_id=tc_id, @@ -581,6 +1129,101 @@ async def _finalize_task_report( await self.summarizer.flush() return final_report + def _record_terminal_step( + self, + args: dict, + raw_text: str, + step_token_usage: dict, + pre_screenshot_bytes, + xml_list, + last_execution_result: dict, + native_text: str | None = None, + ) -> None: + """Writes a report_task_status step to the data engine (accepted or + rejected), mirroring the final-step record shape.""" + if not self.ctx.data_engine: + return + try: + if self.ctx.data_engine.current_step_id is None: + self.ctx.data_engine.allocate_step_id() + self.ctx.data_engine.record_step( + pre_screenshot_bytes=pre_screenshot_bytes, + ui_tree=xml_list, + action_taken={"action": "report_task_status", "args": args}, + operator_raw_thinking=raw_text, + operator_native_thinking=native_text or None, + last_execution_result=last_execution_result, + extra_metadata={"token_usage": step_token_usage}, + ) + except Exception as step_err: + logger.warning(f"Error recording terminal step in FlashRunner: {step_err}") + + async def _verify_completion(self, state: State) -> tuple[bool, str]: + """Terminal-boundary check on a 'completed' report_task_status claim. + + Declared ``goal_invariants`` run first and are authoritative: + ``unsatisfied`` is a deterministic contradiction and rejects the + claim regardless of checker gates; all-``satisfied`` releases it + (the invariants are the declared definition of done); ``unknown`` + falls through to the model final check — or fail-open when that + check is disabled. + + The final review — final screenshot, element list, and the + enumerated read-only device probes — then judges the goal from + device state rather than the agent's narration (observed live: a + double tap netted zero on the dark-theme switch, then the model + reported completed). Fail-open on checker errors: a broken verifier + must never trap the loop. + + Returns ``(allowed, findings)``; ``findings`` lists failed verify + verdicts and unmet subgoals for feedback on rejection. + """ + if self._goal_invariants: + elements = await self._reobserve_elements(state) + report = evaluate_invariants(self._goal_invariants, elements) + if report["verdict"] == "unsatisfied": + logger.info("Completion rejected: declared invariant unsatisfied.") + return False, describe_failures(report) + if report["verdict"] == "satisfied": + logger.info("Completion released: declared invariants satisfied.") + return True, "" + # unknown: insufficient evidence — defer to the model check. + + setup = getattr(self.ctx, "execution_setup", None) + if not (setup and getattr(setup, "final_check_enabled", False)): + return True, "" + try: + from artemis.agents.checker.checker import ( + run_final_check, + verdicts_allow_release, + ) + + timeout = float(getattr(setup, "checkpoint_timeout", 180.0) or 180.0) + report = await asyncio.wait_for( + run_final_check( + self.ctx, + goal=self.goal, + plan_text=self._plan_text or "", + ledger=[], + check_items=[], + ), + timeout=timeout, + ) + except Exception as e: + logger.warning(f"Final check errored ({e}); releasing fail-open.") + return True, "" + if verdicts_allow_release(report) and not report.unmet_subgoals: + logger.info("Final check released the completion report.") + return True, "" + findings = [ + f"[verify failed] '{v.item_text}': {v.evidence}" + + (f" Suggestion: {v.suggestion}" if v.suggestion else "") + for v in report.verdicts + if v.kind == "verify" and v.status == "failed" + ] + findings += [f"[unmet subgoal] '{t}'" for t in report.unmet_subgoals] + return False, "\n".join(findings) or "Final check did not confirm completion." + async def _capture_post_screenshot( self, exec_result, name: str, action_names, state: State | None = None ): @@ -617,6 +1260,77 @@ async def _capture_post_screenshot( state.latest_screenshot = obs.screenshot_path return post_img_bytes + async def _refresh_idle_observation( + self, state: State | None, img_bytes, xml_list + ): + """Re-observes the screen after a turn that executed no device action. + + Turns that end without a tool call never reach the post-action + capture in ``_process_tool_calls``; without this the next turn's tail + (and the hybrid decision menu built from ``state.indexed_elements``) + would describe a stale frame — e.g. one captured mid-transition. + """ + try: + obs, screenshot_bytes = await observe(self.ctx, self.controller, settle_ms=0) + except Exception as obs_err: + logger.warning(f"Idle-turn re-observation failed: {obs_err}") + return img_bytes, xml_list + if not obs.ok: + return img_bytes, xml_list + if state is not None: + if obs.hierarchy_ok: + state.indexed_points = [el["center"] for el in obs.elements] + state.indexed_elements = obs.elements + if obs.screenshot_path: + state.latest_screenshot = obs.screenshot_path + if obs.elements_text: + xml_list = obs.elements_text + return screenshot_bytes or img_bytes, xml_list + + @staticmethod + def _screens_differ(before: bytes | None, after: bytes | None) -> bool: + """Pixel-level 'did the screen change' oracle for stale-tree detection. + + A 16x16 dHash comparison: deterministic screencap encoding means a + same-screen pair scores 0 bits, while even a single appended digit + scored ~9 bits on the emulator (measured) — a threshold of 3 keeps + real small changes while staying far above capture noise. + """ + if not before or not after: + return False + dist = hamming_distance_hex( + dhash_hex(before, hash_size=16), dhash_hex(after, hash_size=16) + ) + return dist is not None and dist >= 3 + + async def _reobserve_elements(self, state: State | None) -> list: + """Re-observes the screen when a decision needs the settled truth. + + Post-action captures race UI updates — a theme toggle applies + after the ~0.4 s capture delay, and the accessibility helper's + event-driven tree can trail a text commit too — so + ``state.indexed_elements`` can hold the pre-action state for a + whole turn (observed live: four allowed taps ping-ponged the + dark-theme switch on lagged ``checked``; a working dial-pad tap + read 'dead' on a lagged dump). Verification decisions — suppress + an action, release a completion, call a tap dead — need the + settled state, and a fresh dump is cheap next to the model call + that just ran. Falls back to the last captured elements on + failure. + """ + elements = getattr(state, "indexed_elements", None) or [] + try: + obs, _ = await observe(self.ctx, self.controller) + except Exception as obs_err: + logger.warning(f"Re-observation failed: {obs_err}") + return elements + if obs.ok and obs.hierarchy_ok: + elements = obs.elements + if state is not None: + state.indexed_points = [el["center"] for el in elements] + state.indexed_elements = elements + return elements + def _extract_normalized_coordinates(self, name: str, args: dict, resolved=None): """Extracts and enriches coordinate metadata for the recorded action. @@ -883,14 +1597,50 @@ async def _execute_and_record_action( async def _invoke_model(self, llm, current_tools: list, messages: list[BaseMessage]): """Binds the active tools and invokes the model through the LLM gateway.""" - # Bind active tools - bound_llm = llm.bind_tools(current_tools) + # Keep only the latest screenshot: strict gateways cap images per + # request, and earlier frames are stale context anyway. + self._prune_intermediate_screenshots(messages) + if os.environ.get("ARTEMIS_DEBUG_PROMPTS"): + # Per-turn prompt digests: [msg_type, per-block (kind,len,hash)] + # so prefix-cache divergence points can be localized by diffing. + import hashlib + import json as _json + + dump_path = Path(os.environ["ARTEMIS_DEBUG_PROMPTS"]) + digest = [] + for msg in messages: + blocks = msg.content if isinstance(msg.content, list) else [{"type": "text", "text": msg.content}] + digest.append([ + getattr(msg, "type", "?"), + [ + ( + b.get("type", "?"), + len(str(b.get("text") or b.get("image_url") or "")), + hashlib.md5(str(b).encode()).hexdigest()[:8], + ) + for b in blocks + ], + ]) + with dump_path.open("a") as fh: + fh.write(_json.dumps(digest) + "\n") + # Bind active tools. Local OpenAI-compatible servers (mlx_vlm and + # friends) may let the model answer with narration only, which the + # loop reads as a no-progress turn; tool_choice="required" holds the + # provider to the loop's contract that every turn is a tool call. + endpoint = getattr(llm, "endpoint", None) + provider = getattr(endpoint, "provider", None) + if provider == ModelProvider.CUSTOM: + bound_llm = llm.bind_tools(current_tools, tool_choice="required") + else: + bound_llm = llm.bind_tools(current_tools) # Invoke Model. Streaming, live-token UI deltas, classified # retries, and pause/resume are all owned by the LLM gateway # (acomplete); a typed LLMCallError propagates if it gives up. return await invoke_llm_with_timeout_message( - acomplete(bound_llm, messages), timeout_seconds=10, hard_timeout=180 + acomplete(bound_llm, messages), + timeout_seconds=10, + hard_timeout=settings.LLM_FLASH_HARD_TIMEOUT_SECONDS, ) async def _process_tool_calls( @@ -939,6 +1689,76 @@ async def _process_tool_calls( logger.info(f"Executing Flash tool: {name}({args})") if name == "report_task_status": + # Terminal-boundary verification: a 'completed' claim is only + # released when the final check confirms the goal from device + # state. A rejected claim feeds the verdicts back and the run + # continues; once the retry budget (final_check_max_attempts) + # is spent, the report is finalized as failed — a blocked + # outcome, mirroring the Pro graph. + if args.get("status") == "completed": + allowed, findings = await self._verify_completion(state) + if not allowed: + self._completion_rejections += 1 + setup = getattr(self.ctx, "execution_setup", None) + max_attempts = int( + getattr(setup, "final_check_max_attempts", 3) or 3 + ) + if self._completion_rejections < max_attempts: + logger.info( + "Completion report rejected by final check" + f" ({self._completion_rejections}/{max_attempts}):" + f" {findings}" + ) + self._record_terminal_step( + args, + raw_text, + step_token_usage, + pre_screenshot_bytes, + xml_list, + last_execution_result={ + "result": ( + "Completion rejected by final check:" + f" {findings}" + ) + }, + native_text=native_text, + ) + messages.append( + ToolMessage( + tool_call_id=tc_id, + name=name, + content=json.dumps( + { + "status": "rejected", + "reason": ( + "Completion verification" + " failed." + ), + "findings": findings, + "instruction": ( + "The final check could not" + " confirm the goal from device" + " state. Continue the task, or" + " report status 'failed'" + " honestly." + ), + } + ), + status="error", + ) + ) + continue + logger.warning( + "Final-check retry budget exhausted; finalizing" + f" as failed. Last findings: {findings}" + ) + args = dict(args) + args["status"] = "failed" + args["explanation"] = ( + f"{args.get('explanation', '')}\n\n" + "Completion verification repeatedly failed;" + f" blocked outcome. {findings}" + ).strip() final_report = await self._finalize_task_report( name, args, @@ -1044,11 +1864,33 @@ async def run(self, state: State) -> dict: llm = self._init_llm() self._llm = llm + # Hybrid loop (decision.flash_loop): VLM describes, a typed decision + # model picks the action. Both flags must be on; a missing/unreachable + # decision endpoint escalates per-turn to the standard tool-call path. + decision_cfg = getattr(getattr(self.ctx, "agent_config", None), "decision", None) + hybrid = bool( + decision_cfg + and getattr(decision_cfg, "enabled", False) + and getattr(decision_cfg, "flash_loop", False) + ) + setup = getattr(self.ctx, "execution_setup", None) + self._goal_invariants = list(getattr(setup, "goal_invariants", None) or []) + if self._goal_invariants: + logger.info( + f"Goal invariants armed: {len(self._goal_invariants)} declared" + " condition(s) gate completion and may close the run early." + ) tools_declaration = self._get_tools() report_only_tools = [t for t in tools_declaration if t.name == "report_task_status"] ledger, img_bytes, xml_list = await self._prepare_conversation(state, tools_declaration) + plan_text = None + if hybrid: + logger.info("Hybrid Flash loop enabled: VLM describes, decision model acts.") + plan_text = await self._hybrid_plan(llm, img_bytes, xml_list) + self._plan_text = plan_text + turns = 0 action_sequence = 0 final_report = None @@ -1056,6 +1898,37 @@ async def run(self, state: State) -> dict: current_xml_list = xml_list previous_turn: _TurnRecord | None = None pending_notices: list[str] = [] + no_tool_call_streak = 0 + # Element taps already executed, by (index, label) identity plus a + # distinctive-label set that survives re-indexing. The decision model + # cannot see the transcript, so masked menu entries are the only + # thing stopping it from re-picking a dead element forever (observed + # live: three identical taps on an unchanged screen, and the same + # icon tapped again four turns later once it worked). Meta actions + # are tracked only as a consecutive-repeat streak — presses back/home + # can be legitimately repeated during navigation. + tried_taps: set[tuple[int, str]] = set() + tried_labels: set[str] = set() + hybrid_last_key: str | None = None + hybrid_last_call_sig = "" + hybrid_repeat_streak = 0 + # Whether the previous turn's actions left the screen unchanged — + # the pre-execution half of verbatim dead-call suppression. + prev_turn_unchanged = False + # Sliding histories for cycle detection: an A,B,A,B ping-pong (enter + # a page, back out, re-enter — observed live on Network & internet) + # never repeats a call verbatim and every screen "changes", so it + # escapes dead-call tracking. The give-away is re-issuing the call + # from two turns ago on a matching screen. + call_sig_history: list[str] = [] + pre_sig_history: list[frozenset] = [] + # How many turns in a row action calls were suppressed because the + # declared goal invariants already held on the live screen. Bounded + # like the other streaks: after the model ignores the notice a few + # times, the verified conditions close the run instead of letting a + # redundant action undo them (observed live: the 'dark theme off' + # run toggled a satisfied switch back on). + satisfied_suppression_streak = 0 while limit is None or turns < limit: turns += 1 @@ -1095,7 +1968,30 @@ async def run(self, state: State) -> dict: turn_base = len(messages) - 1 current_tools = report_only_tools if is_final else tools_declaration - response = await self._invoke_model(llm, current_tools, messages) + # Hybrid turn: describe → decision-model action pick. The final + # turn stays on the standard path (report_task_status only), and + # a repeat streak escalates to the full model entirely. + response = None + tool_calls = None + if ( + hybrid + and not is_final + and hybrid_repeat_streak < _HYBRID_REPEAT_ESCALATE + ): + decided = await self._hybrid_decide( + llm, + state, + tail, + plan_text, + decision_cfg, + tried_taps=tried_taps, + tried_labels=tried_labels, + ) + if decided is not None: + response, tool_calls = decided + + if response is None: + response = await self._invoke_model(llm, current_tools, messages) if response is None: break @@ -1121,7 +2017,8 @@ async def run(self, state: State) -> dict: self._record_llm_trace(step_token_usage, raw_text) - tool_calls = self._resolve_tool_calls(response, raw_text) + if tool_calls is None: + tool_calls = self._resolve_tool_calls(response, raw_text) if not tool_calls: logger.info( @@ -1131,7 +2028,232 @@ async def run(self, state: State) -> dict: if is_final: final_report = {"status": "failed", "explanation": raw_text} break + # Local VLM servers (mlx_vlm) accept tool_choice="required" yet + # still let the model narrate. Re-ask once, in-turn, with an + # explicit directive before counting a no-progress turn — the + # model does emit calls when pushed (observed: it narrates + # "the toggle is on" instead of calling report_task_status). + messages.append( + HumanMessage(content=[{"type": "text", "text": _NO_TOOL_CALL_NOTICE}]) + ) + retry_response = await self._invoke_model(llm, current_tools, messages) + if retry_response is not None: + retry_text = self._extract_response_text(retry_response) + retry_calls = self._resolve_tool_calls(retry_response, retry_text) + if retry_calls: + response, raw_text, tool_calls = retry_response, retry_text, retry_calls + messages.append(response) + + if not tool_calls: + no_tool_call_streak += 1 + if no_tool_call_streak >= _MAX_CONSECUTIVE_NO_TOOL_CALLS: + final_report = { + "status": "failed", + "explanation": ( + f"Stopped after {no_tool_call_streak} consecutive responses" + " without a tool call; the model kept narrating instead" + " of acting." + ), + } + break pending_notices.append(_NO_TOOL_CALL_NOTICE) + # No action ran, so nothing refreshes the observation — but + # the current frame may be a mid-transition capture (the prior + # action's post-screenshot raced a screen animation). Re-serve + # that stale frame next turn and both the describe pass and + # the hybrid menu keep acting on the wrong screen. Re-observe + # here so the next tail reflects what is actually displayed. + ( + current_pre_screenshot_bytes, + current_xml_list, + ) = await self._refresh_idle_observation( + state, current_pre_screenshot_bytes, current_xml_list + ) + ledger.stage_turn(messages[turn_base:]) + previous_turn = _TurnRecord() + continue + + no_tool_call_streak = 0 + + # Snapshot the pre-action screen for the repeat machinery. A tap + # only joins the masked sets when it did NOT change the screen — + # a tap that navigated somewhere is working navigation, and masking + # its label would eventually erase every goal-relevant element from + # the menu (observed: the dark-theme run masked 'Use Dark theme' + # itself, leaving the VLM to toggle the switch blind forever). + elements_now = getattr(state, "indexed_elements", None) or [] + pre_sig = self._element_signature(elements_now) + turn_pre_image = current_pre_screenshot_bytes + label_by_idx = { + el.get("index"): self._element_label(el) + for el in elements_now + if el.get("index") is not None + } + turn_keys: list[str] = [] + tap_records: list[tuple[int, str]] = [] + call_sig = "|".join( + sorted( + f"{tc.get('name')}:{json.dumps(tc.get('args') or {}, sort_keys=True, default=str)}" + for tc in tool_calls + ) + ) + for tc in tool_calls: + key = self._menu_key_for_call(tc.get("name"), tc.get("args")) + if key is None: + continue + turn_keys.append(key) + if key.startswith("tap_"): + idx = int(key[len("tap_"):]) + tap_records.append((idx, label_by_idx.get(idx, ""))) + + # Declared-invariant short-circuit: when every goal invariant is + # verified satisfied on the live screen, further actions can only + # risk undoing the achieved state. Suppress the action calls + # (a report_task_status in the set still goes through — the + # completion gate will release it on the same evidence) and tell + # the model to report; after enough ignored notices the verified + # conditions close the run themselves. + if self._goal_invariants: + has_report_call = any( + str(tc.get("name") or "").split(":")[-1] == "report_task_status" + for tc in tool_calls + ) + # A report call defers to the completion gate (which re- + # observes itself); otherwise re-observe now — the elements + # snapshot above comes from the post-action capture, which + # can lag a still-applying transition by a whole turn. + inv_elements = ( + elements_now + if has_report_call + else await self._reobserve_elements(state) + ) + inv_report = evaluate_invariants(self._goal_invariants, inv_elements) + if inv_report["verdict"] == "unsatisfied" and not has_report_call: + # Evidence-first feedback: the model narrates its own plan + # ("I dialed 55501") and can drift from the field's real + # contents. The invariant already measured live state — + # put the observed-vs-expected values where they cannot + # be missed, before the next action proposal. + pending_notices.append( + "[GOAL CHECK] Declared conditions are not yet met on" + " the live screen — act on these observed values, not" + " on your plan:\n" + describe_failures(inv_report) + ) + if inv_report["verdict"] == "satisfied" and not has_report_call: + satisfied_suppression_streak += 1 + logger.info( + "Suppressing actions: declared goal invariants already" + f" satisfied ({satisfied_suppression_streak}/3)." + ) + notice = ( + "[NOTICE] The declared goal conditions are already" + " satisfied on the current screen. Do not act further" + " — call report_task_status('completed')." + ) + for tc in tool_calls: + messages.append( + ToolMessage( + tool_call_id=tc.get("id") or str(uuid.uuid4()), + name=str(tc.get("name") or ""), + content=json.dumps( + { + "status": "error", + "error": ( + "Goal conditions already satisfied;" + " action not executed. Report" + " task status instead." + ), + } + ), + status="error", + ) + ) + if satisfied_suppression_streak >= 3: + ledger.stage_turn(messages[turn_base:]) + if self.summarizer: + await self.summarizer.flush() + return { + "status": "completed", + "explanation": ( + "Declared goal conditions verified satisfied" + " on device state: " + + ", ".join( + str(r.get("predicate_id") or "invariant") + for r in inv_report["results"] + ) + ), + } + pending_notices.append(notice) + ( + current_pre_screenshot_bytes, + current_xml_list, + ) = await self._refresh_idle_observation( + state, current_pre_screenshot_bytes, current_xml_list + ) + ledger.stage_turn(messages[turn_base:]) + previous_turn = _TurnRecord() + continue + satisfied_suppression_streak = 0 + + # Dead-verbatim suppression: when this turn's call set is identical + # to the previous turn's and that turn changed nothing, executing + # the same actions again can only produce the same dead result — + # the VLM is looping on an element it keeps mis-picking (observed: + # five identical click(10)s on a blank row). Answer the tool calls + # with an error instead so the next turn must choose differently; + # the notice names navigation alternatives. + if call_sig and call_sig == hybrid_last_call_sig and prev_turn_unchanged: + nav = self._navigation_suggestions(elements_now) + notice = ( + "[NOTICE] The exact same action was already executed and" + " changed nothing. It was not re-run. Pick a different" + " action." + + (f" Available navigation elements: {nav}." if nav else "") + ) + logger.info(f"Suppressing verbatim repeat of dead call set: {call_sig}") + for tc in tool_calls: + messages.append( + ToolMessage( + tool_call_id=tc.get("id") or str(uuid.uuid4()), + name=str(tc.get("name") or ""), + content=json.dumps( + { + "status": "error", + "error": ( + "This exact action was already executed" + " and changed nothing; it was not" + " re-run. Choose a different action." + ), + } + ), + status="error", + ) + ) + pending_notices.append(notice) + hybrid_repeat_streak += 1 + if hybrid_repeat_streak >= 3: + # Third verbatim re-issue of a call that was suppressed + # with an explicit "not re-run" error — the model cannot + # break this loop itself (observed: 30+ turns of the same + # suppressed click until the turn cap). Fail honestly + # instead of burning the rest of the budget. + ledger.stage_turn(messages[turn_base:]) + if self.summarizer: + await self.summarizer.flush() + return { + "status": "failed", + "explanation": ( + "The agent kept re-issuing an action that was" + " already suppressed for having no effect, and" + " could not find an alternative way forward." + ), + } + ( + current_pre_screenshot_bytes, + current_xml_list, + ) = await self._refresh_idle_observation( + state, current_pre_screenshot_bytes, current_xml_list + ) ledger.stage_turn(messages[turn_base:]) previous_turn = _TurnRecord() continue @@ -1156,6 +2278,116 @@ async def run(self, state: State) -> dict: injected=injected, native_text=native_text, ) + + # Did anything actually change? Strict set equality — volatile + # sources are filtered inside the signature, so a single real + # difference (one digit appended, one switch flipped) counts as + # changed even on a dense screen. Mask dead taps, count repeats + # of already-dead keys, and flag a verbatim re-issue of last + # turn's call set. + post_sig = self._element_signature( + getattr(state, "indexed_elements", None) or [] + ) + unchanged = pre_sig == post_sig + if unchanged and turn.actions: + # Two stale-evidence guards before a 'dead' verdict. The + # a11y tree can freeze while the screen still moves — + # observed live: a frozen helper tree served identical XML + # across a whole dial-pad run while digits accumulated. + # Pixels come through a different capture path, so a + # changed screenshot disproves 'dead' even when the tree + # lies. Failing that, the post-action dump may just lag a + # settling UI — re-observe once; a real dead tap stays + # dead on the second look. + if self._screens_differ(turn_pre_image, current_pre_screenshot_bytes): + unchanged = False + logger.info( + "Post-action screenshot changed while the element" + " tree did not — stale hierarchy, not a dead turn." + ) + else: + fresh = await self._reobserve_elements(state) + post_sig = self._element_signature(fresh) + unchanged = pre_sig == post_sig + if not unchanged: + logger.info( + "Re-observation resolved an ambiguous dead read:" + " the screen did change after all." + ) + repeat_keys: list[str] = [] + for key in turn_keys: + if key.startswith("tap_"): + idx = int(key[len("tap_"):]) + label = label_by_idx.get(idx, "") + ident = (idx, label) + if ident in tried_taps or ( + len(label) >= 4 and label in tried_labels + ): + repeat_keys.append(key) + elif key == hybrid_last_key: + repeat_keys.append(key) + hybrid_last_key = key + if unchanged: + for idx, label in tap_records: + tried_taps.add((idx, label)) + if len(label) >= 4: + tried_labels.add(label) + verbatim_repeat = ( + unchanged and call_sig and call_sig == hybrid_last_call_sig + ) + ping_pong = ( + bool(call_sig) + and len(call_sig_history) >= 2 + and call_sig == call_sig_history[-2] + and self._signatures_similar(pre_sig, pre_sig_history[-2]) + ) + hybrid_last_call_sig = call_sig + prev_turn_unchanged = unchanged + call_sig_history.append(call_sig) + pre_sig_history.append(pre_sig) + del call_sig_history[:-12] + del pre_sig_history[:-12] + if repeat_keys or verbatim_repeat or ping_pong: + hybrid_repeat_streak += 1 + noted = repeat_keys or ["the same action"] + nav = self._navigation_suggestions(elements_now) + if ping_pong and not repeat_keys and not verbatim_repeat: + notice = ( + "[NOTICE] You are alternating between two screens —" + " this action was already tried two steps ago and the" + " following action undid it. Do not go back and" + " forth: commit to a direction, finish the goal on" + " the current screen, or report the task status." + ) + else: + notice = ( + f"[NOTICE] {', '.join(sorted(set(noted)))} was already" + " attempted and did not move the task forward." + ) + if nav: + notice += f" Available navigation elements instead: {nav}." + if hybrid_repeat_streak >= 3: + notice += ( + " This action has now failed several times in a row —" + " do not repeat it; use a different element, a system" + " key (back/home), or report_task_status('failed')." + ) + else: + notice += " Pick a different action or report the task status." + pending_notices.append(notice) + else: + hybrid_repeat_streak = 0 + + # Feed the decision model a one-line outcome: it cannot see the + # transcript, so a no-op tap and real progress otherwise look the + # same on the next pick (observed: three re-picks of a dead row). + if tool_calls: + names = ", ".join(str(tc.get("name") or "?") for tc in tool_calls) + outcome = "no visible change" if unchanged else "screen changed" + self._last_action_note = f"{names} → {outcome}" + else: + self._last_action_note = None + ledger.stage_turn(messages[turn_base:]) previous_turn = turn if final_report_from_calls is not None: diff --git a/artemis/agents/flash/state_predicates.py b/artemis/agents/flash/state_predicates.py new file mode 100644 index 000000000..a533ce3b3 --- /dev/null +++ b/artemis/agents/flash/state_predicates.py @@ -0,0 +1,206 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic UI-state predicates for Flash verification. + +A predicate pins one element (by selector) to an expected attribute value +and evaluates it against the live element list (``state.indexed_elements``, +the dicts produced by ``format_minimal_list_with_elements``). The verdict is +always one of ``satisfied`` / ``unsatisfied`` / ``unknown`` — absence, +ambiguity or missing state is ``unknown``, never assumed ``unsatisfied``. + +A satisfied predicate proves only the condition it declares for the +observation it ran on — it does not by itself prove the whole task. +``evaluate_invariants`` aggregates the declared ``goal_invariants`` into a +single gate used at the pre-action short-circuit and the completion +boundary (``FlashRunner._verify_completion``). + +Predicate schema:: + + { + "id": "dark_theme_off", + "selector": { + "resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch", + "text": "Dark theme", + }, + "attribute": "checked", + "expected": False, + } + +Boolean attributes (``checked``/``enabled``/``selected``) take boolean +``expected`` values; ``text`` takes a string and compares exact equality. +""" + +from __future__ import annotations + +import re +from typing import Any + +_SELECTOR_KEYS = {"resource_id", "class", "text"} +_BOOL_ATTRIBUTES = {"checked", "enabled", "selected"} +_STATE_ATTRIBUTES = _BOOL_ATTRIBUTES | {"text"} + + +def _as_bool(value: Any) -> bool | None: + """Coerce uiautomator attribute strings to bool; None when not boolean.""" + if type(value) is bool: + return value + if isinstance(value, str): + return {"true": True, "false": False}.get(value.strip().lower()) + return None + + +def evaluate_predicate(predicate: dict[str, Any], elements: list[dict[str, Any]]) -> dict[str, Any]: + """Evaluates one predicate against the current element list. + + Returns ``{"status": ..., "reason": ..., "predicate_id": ..., "evidence": {...}}`` + where status is ``satisfied``, ``unsatisfied`` or ``unknown``. + """ + predicate_id = predicate.get("id") if isinstance(predicate, dict) else None + evidence: dict[str, Any] = {"source": "xml"} + + selector = predicate.get("selector") if isinstance(predicate, dict) else None + attribute = predicate.get("attribute") if isinstance(predicate, dict) else None + expected = predicate.get("expected") if isinstance(predicate, dict) else None + if ( + not isinstance(predicate, dict) + or not isinstance(selector, dict) + or not selector + or not set(selector) <= _SELECTOR_KEYS + or any(not isinstance(v, str) or not v.strip() for v in selector.values()) + or attribute not in _STATE_ATTRIBUTES + or (attribute in _BOOL_ATTRIBUTES and type(expected) is not bool) + or (attribute == "text" and not isinstance(expected, str)) + ): + return { + "status": "unknown", + "reason": "unsupported_predicate", + "predicate_id": predicate_id, + "evidence": evidence, + } + + # OCR-sourced elements are not XML state — a match on detected pixels + # would be a guess about a control the hierarchy never declared. + matches = [ + el + for el in elements + if isinstance(el, dict) + and not el.get("is_ocr") + and all(el.get(key) == value for key, value in selector.items()) + ] + evidence["matches"] = len(matches) + if len(matches) != 1: + return { + "status": "unknown", + "reason": "target_absent" if not matches else "ambiguous_target", + "predicate_id": predicate_id, + "evidence": evidence, + } + + target = matches[0] + observed = ( + _as_bool(target.get(attribute)) + if attribute in _BOOL_ATTRIBUTES + else target.get(attribute) + ) + evidence.update( + attribute=attribute, + observed=observed, + expected=expected, + target={ + key: target.get(key) for key in ("index", "resource_id", "class", "text", "bounds") + }, + ) + if observed is None: + return { + "status": "unknown", + "reason": "attribute_not_known", + "predicate_id": predicate_id, + "evidence": evidence, + } + return { + "status": "satisfied" if observed == expected else "unsatisfied", + "reason": "unique_target_known_value", + "predicate_id": predicate_id, + "evidence": evidence, + } + + +def evaluate_invariants( + predicates: list[dict[str, Any]] | None, elements: list[dict[str, Any]] +) -> dict[str, Any]: + """Aggregates declared goal invariants into one gate. + + ``verdict`` is ``satisfied`` only when EVERY predicate is satisfied; + ``unsatisfied`` when ANY is unsatisfied (deterministic contradiction); + ``unknown`` otherwise (nothing provable either way — including the + empty-predicate case, which must never auto-close a task). + """ + results = [evaluate_predicate(p, elements or []) for p in predicates or []] + if not results: + verdict = "unknown" + elif any(r["status"] == "unsatisfied" for r in results): + verdict = "unsatisfied" + elif all(r["status"] == "satisfied" for r in results): + verdict = "satisfied" + else: + verdict = "unknown" + return {"verdict": verdict, "results": results} + + +def _digit_prefix_diff(expected: str, observed: str) -> str | None: + """Locates the first differing digit between an expected and observed value. + + Text-equality predicates on formatted fields (a dial pad renders '555-0155', + then '(555) 015-5' once a formatter kicks in) need digit-level comparison — + raw prefix comparison would report a false mismatch at the first '(' or + '-'. Returns a short clause describing the common digit prefix and both + remainders, or None when digit counts/content can't be compared usefully. + """ + exp = re.sub(r"\D", "", expected) + obs = re.sub(r"\D", "", observed) + if not exp or not obs: + return None + shared = 0 + for a, b in zip(exp, obs): + if a != b: + break + shared += 1 + if shared == len(exp) == len(obs): + return None + return ( + f" digits 1-{shared} '{exp[:shared] or '(none)'}' match;" + f" expected continues '{exp[shared:]}', observed continues '{obs[shared:]}'" + ) + + +def describe_failures(report: dict[str, Any]) -> str: + """One-line-per-predicate findings for feedback into the model turn.""" + lines = [] + for r in report.get("results", []): + if r["status"] != "unsatisfied": + continue + ev = r.get("evidence", {}) + target = ev.get("target") or {} + detail = "" + if ev.get("attribute") == "text": + diff = _digit_prefix_diff(str(ev.get("expected")), str(ev.get("observed"))) + detail = f" —{diff}" if diff else "" + lines.append( + f"[invariant failed] '{r.get('predicate_id') or target.get('text')}'" + f" expected {ev.get('attribute')}={ev.get('expected')}" + f" but observed {ev.get('observed')}{detail}" + ) + return "\n".join(lines) or "Declared goal condition is not satisfied." diff --git a/artemis/agents/flash/summarizer.py b/artemis/agents/flash/summarizer.py index 5b8a3cf98..0147b652c 100644 --- a/artemis/agents/flash/summarizer.py +++ b/artemis/agents/flash/summarizer.py @@ -41,6 +41,7 @@ from artemis.utils.task_tree import format_actions_clean from artemis.utils.visualization import draw_action_overlay_on_image from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -430,7 +431,7 @@ async def _attempt(self, key: JobKey) -> bool: } ) content_blocks.append( - {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_pre}"}} + {"type": "image_url", "image_url": {"url": image_data_uri(b64_pre)}} ) if post_bytes: @@ -449,7 +450,7 @@ async def _attempt(self, key: JobKey) -> bool: content_blocks.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{b64_post}"}, + "image_url": {"url": image_data_uri(b64_post)}, } ) diff --git a/artemis/agents/image_processor/image_processor.py b/artemis/agents/image_processor/image_processor.py index 4ce526984..54e60a8e4 100644 --- a/artemis/agents/image_processor/image_processor.py +++ b/artemis/agents/image_processor/image_processor.py @@ -35,6 +35,7 @@ from artemis.services.llm import get_llm from artemis.utils.python_executor import PythonExecutor from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -115,11 +116,11 @@ async def run(self, instruction: str, target_image_path: str) -> dict: SystemMessage(content=prompt_template), HumanMessage( content=[ - {"type": "text", "text": "Here is the target image. Begin writing your code."}, { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}, + "image_url": {"url": image_data_uri(img_b64)}, }, + {"type": "text", "text": "Here is the target image. Begin writing your code."}, ] ), ] @@ -186,9 +187,7 @@ async def run(self, instruction: str, target_image_path: str) -> dict: image_blocks.append( { "type": "image_url", - "image_url": { - "url": f"data:image/jpeg;base64,{b64_data}" - }, + "image_url": {"url": image_data_uri(b64_data)}, } ) image_blocks.append({"type": "text", "text": f"\n{label}\n"}) diff --git a/artemis/agents/object_detector/object_detector.py b/artemis/agents/object_detector/object_detector.py index b2acb4f5f..acfb3e14f 100644 --- a/artemis/agents/object_detector/object_detector.py +++ b/artemis/agents/object_detector/object_detector.py @@ -16,7 +16,9 @@ import asyncio import base64 +import io import json +import math import os from pathlib import Path import re @@ -25,12 +27,176 @@ from artemis.llm.structured import ParseFailure, parse_structured from artemis.services.llm import get_llm +from artemis.utils.image_codec import image_data_uri from third_party.mobile_use.utils.logger import get_logger logger = get_logger(__name__) _DETECTOR_SEMAPHORE = asyncio.Semaphore(6) +# The detector's output contract: points are [x, y] on a normalized 0-1000 +# grid over the image that was sent (consumers divide by 1000 to get pixels). +# Items may also carry ``box_2d`` corner pairs in the same declared convention +# (Gemma 4's trained detection output is [ymin, xmin, ymax, xmax] / 1000, i.e. +# the default ``yx_1000``); a box is normalized in place to [x1, y1, x2, y2] +# and synthesizes ``point`` from its center when the item lacks one. +# Models differ in what they natively emit; ``coordinate_format`` on the +# resolved endpoint declares the convention so results can be normalized: +# axis order — ``yx`` (Gemini ER style) or ``xy`` (most vision models) +# scale — ``1000`` grid, ``px`` absolute pixels, ``norm`` 0-1 floats +_DEFAULT_COORDINATE_FORMAT = "yx_1000" +_PIXEL_GRID = 1000.0 + +# Qwen-family processors smart-resize inputs to a multiple of the patch-merge +# factor within min/max pixel bounds; pixel-space coordinates land on that +# resized grid, not the raw image grid. +_QWEN_RESIZE_FACTOR = 32 +_QWEN_MIN_PIXELS = 65536 +_QWEN_MAX_PIXELS = 16777216 # size.longest_edge in the shipped preprocessor config + + +def _image_dimensions(image_data: bytes) -> tuple[int, int] | None: + """Return (width, height) of encoded image bytes, or None when undecodable.""" + try: + from PIL import Image + + with Image.open(io.BytesIO(image_data)) as img: + return img.width, img.height + except Exception: + return None + + +def _qwen_resized_dimensions(width: int, height: int) -> tuple[int, int]: + """Grid a Qwen-family processor sees after its smart resize.""" + factor = _QWEN_RESIZE_FACTOR + w_bar = round(width / factor) * factor + h_bar = round(height / factor) * factor + if w_bar * h_bar > _QWEN_MAX_PIXELS: + beta = math.sqrt((width * height) / _QWEN_MAX_PIXELS) + w_bar = math.floor(width / beta / factor) * factor + h_bar = math.floor(height / beta / factor) * factor + elif w_bar * h_bar < _QWEN_MIN_PIXELS: + beta = math.sqrt(_QWEN_MIN_PIXELS / (width * height)) + w_bar = math.ceil(width * beta / factor) * factor + h_bar = math.ceil(height * beta / factor) * factor + return w_bar, h_bar + + +def _endpoint_coordinate_format(llm) -> str: + fmt = getattr(getattr(llm, "endpoint", None), "coordinate_format", None) + if isinstance(fmt, str) and fmt in _COORDINATE_FORMATS: + return fmt + return _DEFAULT_COORDINATE_FORMAT + + +_COORDINATE_FORMATS = frozenset({"yx_1000", "xy_1000", "yx_px", "xy_px", "yx_norm", "xy_norm"}) + + +def _detected_batch_scale(items: list) -> str | None: + """Infer the actual scale a model used across a whole response. + + The contract is 0-1000, but when a model ignores it the whole batch + shares one convention: every coordinate fitting in 0-1 means ``norm``; + any coordinate above 1000 means ``px``. Mixed magnitudes stay on the + declared grid so sub-1000 corner points are not rescaled twice. + """ + coords = [ + abs(float(c)) + for item in items + if isinstance(item, dict) + for key in ("point", "box_2d") + if isinstance(item.get(key), (list, tuple)) + and len(item[key]) == (2 if key == "point" else 4) + for c in item[key] + if isinstance(c, (int, float)) and not isinstance(c, bool) + ] + if not coords: + return None + max_coord = max(coords) + if max_coord <= 1.0 and any(c != 0.0 for c in coords): + return "norm" + if max_coord > _PIXEL_GRID: + return "px" + return None + + +def _normalize_detected_point( + item: dict, + coordinate_format: str, + image_size: tuple[int, int] | None, + batch_scale: str | None, +) -> None: + """Rewrite ``item['point']`` to the [x, y] 0-1000 contract in place. + + Axis order and scale come from the endpoint's declared coordinate + format; ``batch_scale`` (``norm``/``px``) overrides the declared scale + only when the whole response clearly violated the contract. + """ + point = item.get("point") + if not (isinstance(point, (list, tuple)) and len(point) == 2): + return + try: + first, second = float(point[0]), float(point[1]) + except (TypeError, ValueError): + return + + x, y = (second, first) if coordinate_format.startswith("yx") else (first, second) + + scale = batch_scale or coordinate_format.rsplit("_", 1)[-1] + if scale == "px": + if image_size: + grid_w, grid_h = _qwen_resized_dimensions(*image_size) + x, y = x * _PIXEL_GRID / grid_w, y * _PIXEL_GRID / grid_h + elif scale == "norm": + x, y = x * _PIXEL_GRID, y * _PIXEL_GRID + + item["point"] = [x, y] + + +def _normalize_detected_box( + item: dict, + coordinate_format: str, + image_size: tuple[int, int] | None, + batch_scale: str | None, +) -> bool: + """Rewrite ``item['box_2d']`` to [x1, y1, x2, y2] on the 0-1000 contract. + + When the item has no usable ``point``, the box center becomes it. Returns + True only in that case: a synthesized point is already on the contract + grid and must not go through ``_normalize_detected_point`` again. + """ + box = item.get("box_2d") + if not (isinstance(box, (list, tuple)) and len(box) == 4): + return False + try: + a, b, c, d = (float(v) for v in box) + except (TypeError, ValueError): + return False + + x1, y1, x2, y2 = (b, a, d, c) if coordinate_format.startswith("yx") else (a, b, c, d) + + scale = batch_scale or coordinate_format.rsplit("_", 1)[-1] + if scale == "px": + if image_size: + grid_w, grid_h = _qwen_resized_dimensions(*image_size) + x1, x2 = x1 * _PIXEL_GRID / grid_w, x2 * _PIXEL_GRID / grid_w + y1, y2 = y1 * _PIXEL_GRID / grid_h, y2 * _PIXEL_GRID / grid_h + elif scale == "norm": + x1, y1, x2, y2 = x1 * _PIXEL_GRID, y1 * _PIXEL_GRID, x2 * _PIXEL_GRID, y2 * _PIXEL_GRID + + item["box_2d"] = [x1, y1, x2, y2] + + point = item.get("point") + if isinstance(point, (list, tuple)) and len(point) == 2: + try: + float(point[0]) + float(point[1]) + return False + except (TypeError, ValueError): + pass + item["point"] = [(x1 + x2) / 2, (y1 + y2) / 2] + return True + async def _detect_single_label( llm, @@ -53,7 +219,7 @@ async def _detect_single_label( content=[ { "type": "image_url", - "image_url": {"url": f"data:{mime_type};base64,{img_b64}"}, + "image_url": {"url": image_data_uri(img_b64)}, }, {"type": "text", "text": prompt}, ] @@ -124,7 +290,7 @@ async def _run_object_detection( queries = queries or [] templates = templates or ["Point to the following objects: {labels_str}"] try: - llm = get_llm(ctx, name="object_detector") + llm = get_llm(ctx, name="object_detector", is_utils=True) except Exception: llm = get_llm(ctx, name="operator") @@ -165,16 +331,18 @@ async def _run_object_detection( except Exception as e: logger.error(f"Task raised exception: {e}") - # Swap coordinates from [y, x] normalized to [x, y] normalized if needed + # Normalize model points to the [x, y] 0-1000 contract. Axis order and + # scale follow the endpoint's declared coordinate_format (Gemini ER's + # [y, x] stays the default); a batch that clearly arrived in 0-1 or raw + # pixels is rescaled instead. + coordinate_format = _endpoint_coordinate_format(llm) + image_size = _image_dimensions(image_data) + batch_scale = _detected_batch_scale(fused_results) for item in fused_results: - if ( - isinstance(item, dict) - and "point" in item - and isinstance(item["point"], list) - and len(item["point"]) == 2 - ): - y_norm, x_norm = item["point"] - item["point"] = [x_norm, y_norm] + if isinstance(item, dict): + box_derived = _normalize_detected_box(item, coordinate_format, image_size, batch_scale) + if not box_derived: + _normalize_detected_point(item, coordinate_format, image_size, batch_scale) detected_labels = set( item["label"] for item in fused_results if isinstance(item, dict) and "label" in item diff --git a/artemis/agents/operator/prompts.py b/artemis/agents/operator/prompts.py index e800de234..3bbb4eee5 100644 --- a/artemis/agents/operator/prompts.py +++ b/artemis/agents/operator/prompts.py @@ -31,6 +31,7 @@ _format_long_output_response, _is_output_long, ) +from artemis.utils.image_codec import image_data_uri from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec from artemis.utils.task_tree import SELF_DESCRIBED_MARKER, action_intent_phrase from third_party.mobile_use.utils.logger import get_logger @@ -384,7 +385,7 @@ async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisConte builder.add_human_content( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"}, + "image_url": {"url": image_data_uri(latest_screenshot_b64)}, } ) builder.add_human_content(f"--- Visible UI Elements ---\n{minimal_list}") diff --git a/artemis/agents/outputter/outputter.py b/artemis/agents/outputter/outputter.py index 9b75b427f..9a63fcb0c 100644 --- a/artemis/agents/outputter/outputter.py +++ b/artemis/agents/outputter/outputter.py @@ -46,6 +46,7 @@ from artemis.utils.task_tree import get_active_subgoal_hashes from third_party.mobile_use.utils.logger import get_logger from pydantic import BaseModel +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -257,27 +258,31 @@ def _initial_messages( target_schema=_target_schema_json(output_config), plan_and_history=plan_and_history, ) - content: list[dict[str, Any]] = [{"type": "text", "text": human_message}] - - if ctx.data_engine: - verification = _verification_block(ctx, graph_output) - if verification: - content.append({"type": "text", "text": verification}) - - environment = _environment_block(ctx) - if environment: - content.append({"type": "text", "text": environment}) + content: list[dict[str, Any]] = [] raw_data = graph_output.operator_raw_data screenshot_b64 = raw_data.get("screenshot_b64") if raw_data else None if screenshot_b64: + # Image-first: vision models ground better when the screenshot + # precedes the instruction/context text. content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"}, + "image_url": {"url": image_data_uri(screenshot_b64)}, } ) + content.append({"type": "text", "text": human_message}) + + if ctx.data_engine: + verification = _verification_block(ctx, graph_output) + if verification: + content.append({"type": "text", "text": verification}) + + environment = _environment_block(ctx) + if environment: + content.append({"type": "text", "text": environment}) + return [SystemMessage(content=_SYSTEM_MESSAGE), HumanMessage(content=content)] diff --git a/artemis/agents/planner/planner.py b/artemis/agents/planner/planner.py index 02be5c7c4..20e51c427 100644 --- a/artemis/agents/planner/planner.py +++ b/artemis/agents/planner/planner.py @@ -54,6 +54,7 @@ from artemis.utils.cython_compat import CyFunctionDetector from third_party.mobile_use.utils.decorators import agent_lifecycle_logging from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -274,14 +275,15 @@ async def __call__(self, state: State): human_message = Template(human_content).render(**render_kwargs) - human_message_content = [{"type": "text", "text": human_message}] + human_message_content = [] if screenshot_b64: human_message_content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"}, + "image_url": {"url": image_data_uri(screenshot_b64)}, } ) + human_message_content.append({"type": "text", "text": human_message}) messages = [ SystemMessage(content=system_message), HumanMessage(content=human_message_content), diff --git a/artemis/agents/validator/precondition_pixel.py b/artemis/agents/validator/precondition_pixel.py index 80a542fb8..784bfe22c 100644 --- a/artemis/agents/validator/precondition_pixel.py +++ b/artemis/agents/validator/precondition_pixel.py @@ -36,6 +36,7 @@ from artemis.services.llm import acomplete_structured from artemis.utils import visualization from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -116,17 +117,16 @@ def _build_messages( action_name = action_item.get("action") user_content = [ - {"type": "text", "text": _describe_target(action_item)}, + # Images first, then the target description: vision models ground + # better when pixel content precedes the instruction text. + {"type": "image_url", "image_url": {"url": image_data_uri(orig_b64)}}, {"type": "text", "text": "[Image 1 (Reference)]"}, { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{orig_b64}"}, + "image_url": {"url": image_data_uri(live_b64)}, }, {"type": "text", "text": "[Image 2 (Current State)]"}, - { - "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{live_b64}"}, - }, + {"type": "text", "text": _describe_target(action_item)}, ] if state: thoughts = [] diff --git a/artemis/agents/validator/tool_declarations.py b/artemis/agents/validator/tool_declarations.py index fba8dd751..889b41cb2 100644 --- a/artemis/agents/validator/tool_declarations.py +++ b/artemis/agents/validator/tool_declarations.py @@ -45,6 +45,7 @@ READ_NOTE_DOCSTRING, ) from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -79,7 +80,7 @@ def to_langchain_tool_message(self) -> ToolMessage: content_blocks.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{b64_img}"}, + "image_url": {"url": image_data_uri(b64_img)}, } ) return ToolMessage( diff --git a/artemis/agents/video_analyzer/universal_engine.py b/artemis/agents/video_analyzer/universal_engine.py index 96ade3fe7..767a9ec98 100644 --- a/artemis/agents/video_analyzer/universal_engine.py +++ b/artemis/agents/video_analyzer/universal_engine.py @@ -42,6 +42,7 @@ ) from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -159,7 +160,7 @@ async def _build_universal_user_blocks( user_blocks.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{b64_str}"}, + "image_url": {"url": image_data_uri(b64_str)}, } ) diff --git a/artemis/config/__init__.py b/artemis/config/__init__.py index 46862a438..e8092005f 100644 --- a/artemis/config/__init__.py +++ b/artemis/config/__init__.py @@ -19,6 +19,7 @@ CheckerConfig, CommitteeConfig, DEFAULT_VERIFICATION_LEVEL, + DecisionLayerConfig, ExplorerConfig, FlashProfileConfig, MemoryChunkingConfig, @@ -211,6 +212,7 @@ "AgentGlobalConfig", "CheckerConfig", "CommitteeConfig", + "DecisionLayerConfig", "DEFAULT_VERIFICATION_LEVEL", "ExplorerConfig", "FlashProfileConfig", diff --git a/artemis/config/agent.py b/artemis/config/agent.py index 6fb00d398..56d9c3ed0 100644 --- a/artemis/config/agent.py +++ b/artemis/config/agent.py @@ -152,6 +152,61 @@ class PlannerValidationConfig(BaseModel): model_config = {"extra": "allow"} +class DecisionLayerConfig(BaseModel): + """System-One decision layer: cheap typed junctions (noul/choice/score) + answered by a local decision model (e.g. Laya) instead of a full LLM call. + + Every junction is advisory — a decision-layer miss or low-confidence + answer always escalates to the existing model path, never rolls back. + """ + + enabled: bool = Field( + default=False, + description="Master switch for the decision layer. Off = identical to upstream behavior.", + ) + url: str = Field( + default="http://127.0.0.1:11435/v1/systemone", + description=( + "Jev/Laya-compatible decision endpoint (POST /v1/systemone). " + "`artemis model pull laya` + `artemis model serve laya -p 11435` " + "provisions a matching local server." + ), + ) + model: str | None = Field( + default="typed-decisions", + description=( + "Checkpoint name sent in the request payload — laya-serve honors " + "its own checkpoint names ('typed-decisions', 'english', " + "'multilingual')." + ), + ) + timeout_seconds: float = Field( + default=5.0, + gt=0, + description="Per-request decision timeout; expiry escalates to the LLM path.", + ) + flash_loop: bool = Field( + default=False, + description=( + "Hybrid Flash loop: the VLM describes the screen (text only, no tool " + "calls), the decision model picks the next action from a fixed menu " + "built on the live element list. Off → the VLM emits tool calls " + "directly (upstream behavior)." + ), + ) + flash_choice_threshold: float = Field( + default=0.25, + ge=0.0, + le=1.0, + description=( + "Min top-pick probability (answer_confidence) for a hybrid action-menu " + "pick to be acted on. Menus carry ~20 options, so 0.25 is roughly 5x " + "uniform; below the floor the turn escalates to the VLM tool path." + ), + ) + model_config = {"extra": "allow"} + + class CommitteeConfig(BaseModel): """Configuration specific to Multi-Agent Committee / Council.""" @@ -777,6 +832,18 @@ class FlashProfileConfig(BaseModel): default_factory=StepSummarizerConfig, description="Asynchronous visual context compressor settings.", ) + goal_invariants: list[dict] = Field( + default_factory=list, + description=( + "Declarative success conditions for the Flash loop, evaluated by" + " artemis.agents.flash.state_predicates against live element state" + " (e.g. {id, selector: {resource_id, class, text}, attribute:" + " 'checked', expected: false}). All-satisfied suppresses further" + " actions and may close the run; any-unsatisfied rejects a" + " 'completed' report; unknown defers to the final check. Declare" + " only conditions that jointly define task success." + ), + ) model_config = {"extra": "allow"} @@ -880,6 +947,10 @@ class AgentGlobalConfig(BaseModel): " ``explorer.flash_mode`` / ``explorer.pro_mode`` decide." ), ) + decision: DecisionLayerConfig = Field( + default_factory=DecisionLayerConfig, + description="System-One decision layer (Laya/Jev) offloading cheap typed junctions.", + ) planner_validation: PlannerValidationConfig = Field( default_factory=PlannerValidationConfig, description="Planner milestone validation runtime options.", diff --git a/artemis/config/local_models.py b/artemis/config/local_models.py new file mode 100644 index 000000000..65d3746d4 --- /dev/null +++ b/artemis/config/local_models.py @@ -0,0 +1,77 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Catalog of local models ARTEMIS can pull and serve itself. + +Each alias maps to a Hugging Face repository servable by ``mlx_vlm.server`` +(``pip install artemis[local]`` on Apple Silicon). Aliases are what the +``"model"`` field in ``artemis.jsonc`` references when ``provider`` is +``"custom"`` and ``api_base`` points at the managed server. +""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class LocalModelSpec: + """A local model ARTEMIS can pull and serve.""" + + repo: str + description: str = "" + # Recommended ``max_soft_tokens`` for the model's image processor — + # the vision-token budget per image. Gemma 4 ships with 280, which + # downsamples a phone screenshot to ~528x1152 effective; 1120 keeps + # near-original detail (~1056x2352) and measurably improves detection + # and not-found discipline. ``None`` leaves the shipped default. + vision_soft_tokens: int | None = None + # Serving stack: "mlx" = HF weights via mlx_vlm.server; "laya" = the + # decision model's own `laya-serve` binary in a managed venv. + kind: str = "mlx" + # For kind="laya": the checkpoint name laya-serve preloads + # ("typed-decisions", "english", "multilingual"). + checkpoint: str = "" + + +LOCAL_MODELS: dict[str, LocalModelSpec] = { + "gemma4-e4b": LocalModelSpec( + repo="mlx-community/gemma-4-e4b-it-4bit", + description="Gemma 4 E4B 4-bit multimodal (tested; ~5 GB pull, ~6 GB resident)", + vision_soft_tokens=1120, + ), + "laya": LocalModelSpec( + repo="laya", + description=( + "Laya decision model (typed-decisions checkpoint, ~421M, CPU/MPS) — " + "serves POST /v1/systemone for the decision.* layer" + ), + kind="laya", + checkpoint="typed-decisions", + ), +} + +DEFAULT_LOCAL_MODEL = "gemma4-e4b" + + +def resolve_model_ref(alias_or_repo: str) -> tuple[str, str] | None: + """Map a catalog alias or ``org/repo`` id to ``(alias, repo)``. + + Returns ``None`` for bare names that match neither the catalog nor the + ``org/repo`` shape — those cannot be served by ARTEMIS itself. + """ + spec = LOCAL_MODELS.get(alias_or_repo) + if spec is not None: + return alias_or_repo, spec.repo + if "/" in alias_or_repo: + return alias_or_repo, alias_or_repo + return None diff --git a/artemis/config/settings.py b/artemis/config/settings.py index 182d7ec77..6558f22c4 100644 --- a/artemis/config/settings.py +++ b/artemis/config/settings.py @@ -144,6 +144,14 @@ class Settings(BaseSettings): " forever (legacy interactive behavior)." ), ) + LLM_FLASH_HARD_TIMEOUT_SECONDS: float | None = Field( + default=180.0, + description=( + "Hard per-call timeout for Flash operator turns, in seconds." + " Busy time only: paused waits (PAUSE_FILE) do not accrue." + " Raise for slow local providers; None waits indefinitely." + ), + ) # Paths & Storage TRACES_PATH: Path = Field(default_factory=get_default_traces_path) diff --git a/artemis/context.py b/artemis/context.py index 57360c8c8..233c899ff 100644 --- a/artemis/context.py +++ b/artemis/context.py @@ -72,6 +72,13 @@ class ExecutionSetup(ExecutionSetupBase): enable_committee: bool = False committee_debate_rounds: int = 2 disable_outputter: bool = False + goal_invariants: list[dict[str, Any]] = Field(default_factory=list) + """Declarative terminal-state conditions for the Flash runner + (``artemis.agents.flash.state_predicates``). Evaluated against live + element dicts: all ``satisfied`` → the loop may close early and a + 'completed' report is released; any ``unsatisfied`` → a 'completed' + report is rejected with evidence; ``unknown`` → falls through to the + model final check. Declare only conditions that jointly define success.""" outputter: OutputterConfig = Field(default_factory=OutputterConfig) explorer: ExplorerConfig = Field(default_factory=ExplorerConfig) explorer_versions: dict[str, str] = Field(default_factory=dict) diff --git a/artemis/core/decision_client.py b/artemis/core/decision_client.py new file mode 100644 index 000000000..8d6e120b2 --- /dev/null +++ b/artemis/core/decision_client.py @@ -0,0 +1,177 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Typed decision-model client — the "System One" layer. + +Decision models (Jev from TypeSafe AI, or the open-weight Laya from Convai +Innovations — served locally via Ollama MLX builds) answer *typed* questions +about a distilled text/JSON state in a single non-autoregressive pass: +``choice`` (pick one of labelled options), ``score`` (ordered rubric), or +``noul`` (calibrated yes/no probability). They never emit free text, so +there is nothing to parse and no format to hallucinate. + +Wire format (TypeSafe-compatible, which Laya's ``typed-decisions`` checkpoint +also speaks):: + + POST {url} + {"model": "...", "state": "...", + "questions": {"gate": {"type": "noul", "instructions": "..."}}} + -> + {"answers": {"gate": {"noul": 0.83, "confidence": 0.91}}, ...} + +Used to offload cheap boolean/categorical junctions (validator gates, +planner review, routing triage) from the heavyweight multimodal model. +Fails closed: any transport/parse problem yields ``DecisionResult`` with +``verdict=None`` and ``confidence=0`` so callers escalate to the LLM path. +""" + +import asyncio +from dataclasses import dataclass, field +import json +import urllib.error +import urllib.request + +from third_party.mobile_use.utils.logger import get_logger + +logger = get_logger(__name__) + +DEFAULT_TIMEOUT_SECONDS = 5.0 + + +@dataclass +class DecisionResult: + """One answered question. + + ``verdict`` is the typed answer — bool for ``noul`` (probability >= 0.5), + the winning key for ``choice``, a float for ``score`` — or ``None`` when + the decision failed and the caller should escalate to the model path. + """ + + verdict: bool | str | float | None + confidence: float = 0.0 + probabilities: dict = field(default_factory=dict) + raw: dict = field(default_factory=dict) + + @property + def answered(self) -> bool: + return self.verdict is not None + + +class DecisionClient: + """Minimal urllib client for Jev/Laya-compatible decision endpoints.""" + + def __init__( + self, + url: str, + model: str | None = None, + timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS, + ): + self._url = url + self._model = model + self._timeout = timeout_seconds + + def decide(self, state: str, questions: dict) -> dict[str, DecisionResult]: + """One request, all questions answered in a single forward pass.""" + payload: dict = {"state": state, "questions": questions} + if self._model: + payload["model"] = self._model + request = urllib.request.Request( + self._url, + data=json.dumps(payload).encode("utf-8"), + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=self._timeout) as resp: + body = json.loads(resp.read().decode("utf-8")) + except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e: + logger.warning(f"Decision endpoint {self._url} failed: {e}") + return {name: DecisionResult(verdict=None) for name in questions} + answers = body.get("answers") or {} + return { + name: self._parse_answer(spec, answers.get(name) or {}) + for name, spec in questions.items() + } + + async def adecide(self, state: str, questions: dict) -> dict[str, DecisionResult]: + return await asyncio.to_thread(self.decide, state, questions) + + def noul(self, state: str, question: str) -> DecisionResult: + """Calibrated yes/no — the gate/guardrail primitive.""" + results = self.decide( + state, {"q": {"type": "noul", "instructions": question}} + ) + return results["q"] + + async def anoul(self, state: str, question: str) -> DecisionResult: + return await asyncio.to_thread(self.noul, state, question) + + def choose( + self, state: str, question: str, criteria: dict[str, str] + ) -> DecisionResult: + """Categorical pick over labelled options.""" + results = self.decide( + state, + {"q": {"type": "choice", "instructions": question, "criteria": criteria}}, + ) + return results["q"] + + async def achoose( + self, state: str, question: str, criteria: dict[str, str] + ) -> DecisionResult: + return await asyncio.to_thread(self.choose, state, question, criteria) + + def _parse_answer(self, spec: dict, answer: dict) -> DecisionResult: + qtype = spec.get("type", "noul") + confidence = answer.get("confidence") + try: + confidence = float(confidence) if confidence is not None else 0.0 + except (TypeError, ValueError): + confidence = 0.0 + probabilities = answer.get("probabilities") or {} + if qtype == "noul": + prob = answer.get("noul") + if prob is None: + return DecisionResult(verdict=None, confidence=confidence, raw=answer) + try: + prob = float(prob) + except (TypeError, ValueError): + return DecisionResult(verdict=None, confidence=confidence, raw=answer) + return DecisionResult( + verdict=prob >= 0.5, + confidence=confidence, + probabilities={"yes": prob, "no": 1.0 - prob}, + raw=answer, + ) + if qtype == "choice": + verdict = answer.get("choice") + return DecisionResult( + verdict=verdict, + confidence=confidence, + probabilities=probabilities, + raw=answer, + ) + if qtype == "score": + score = answer.get("score") + try: + verdict = float(score) if score is not None else None + except (TypeError, ValueError): + verdict = None + return DecisionResult( + verdict=verdict, + confidence=confidence, + probabilities=probabilities, + raw=answer, + ) + return DecisionResult(verdict=None, confidence=confidence, raw=answer) diff --git a/artemis/core/diagnostics/engine.py b/artemis/core/diagnostics/engine.py index 97f9d20ca..5b3c91e26 100644 --- a/artemis/core/diagnostics/engine.py +++ b/artemis/core/diagnostics/engine.py @@ -27,6 +27,9 @@ LLMCredentialsProbe, VisionOCRProbe, ) +from artemis.core.diagnostics.probes.local_model_probe import ( + LocalModelEndpointProbe, +) from artemis.core.diagnostics.probes.runtime_probe import ( PythonRuntimeProbe, SystemConfigProbe, @@ -67,6 +70,7 @@ def __init__(self): self._credentials_probe = LLMCredentialsProbe() self._ocr_probe = VisionOCRProbe() self._adb_probe = AdbDeviceProbe() + self._local_model_probe = LocalModelEndpointProbe() self._report_cache: SystemReadinessReport | None = None self._report_cache_time = 0.0 self._report_cache_generation = -1 @@ -80,6 +84,7 @@ def __init__(self): self.register_probe(self._credentials_probe) self.register_probe(self._ocr_probe) self.register_probe(self._adb_probe) + self.register_probe(self._local_model_probe) def register_probe(self, probe: BaseProbe) -> None: """Register a new diagnostic probe.""" diff --git a/artemis/core/diagnostics/probes/credentials_probe.py b/artemis/core/diagnostics/probes/credentials_probe.py index 683e216c3..b07705383 100644 --- a/artemis/core/diagnostics/probes/credentials_probe.py +++ b/artemis/core/diagnostics/probes/credentials_probe.py @@ -144,6 +144,30 @@ async def probe(self) -> ProbeResult: ) api_keys_map[prov_id] = val.strip() + # Config-bound local endpoints (artemis-served models, Ollama, vLLM, + # any OpenAI-compatible server) need no API key: the custom provider + # authenticates to a loopback or self-hosted server. + try: + from artemis.core.diagnostics.probes.local_model_probe import ( + _local_endpoints, + ) + + for base, models in _local_endpoints().items(): + configured_providers.append( + { + "provider": "custom", + "label": "Local Model Endpoint", + "masked": base, + "raw_key": base, + "key": base, + } + ) + api_keys_map["custom"] = base + for alias in models: + api_keys_map.setdefault(alias, base) + except Exception: + pass + if ocr_key and not is_placeholder_key(ocr_key): api_keys_map["ocr"] = ocr_key.get_secret_value() diff --git a/artemis/core/diagnostics/probes/local_model_probe.py b/artemis/core/diagnostics/probes/local_model_probe.py new file mode 100644 index 000000000..f43dbc051 --- /dev/null +++ b/artemis/core/diagnostics/probes/local_model_probe.py @@ -0,0 +1,273 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Local Model Endpoint Readiness Probe. + +When artemis.jsonc routes nodes to a ``custom`` provider (an OpenAI-compatible +endpoint at ``api_base``), every configured model must actually be served +there. This probe verifies the endpoint is reachable and that each required +model appears in ``GET {api_base}/models``. + +Guidance is model-aware: aliases in the Artemis local-model catalog get +``artemis model pull`` / ``artemis model serve`` commands; other models are +BYO — the probe suggests serving them with the endpoint's own tooling +(Ollama, vLLM, LM Studio, ...) instead of a command Artemis cannot run. +""" + +from typing import Any +from urllib.parse import urlparse + +import httpx + +from artemis.config.local_models import resolve_model_ref +from artemis.core.diagnostics.probes.base import BaseProbe +from artemis.core.diagnostics.schema import ( + ProbeAction, + ProbeCategory, + ProbeResult, + ProbeStatus, +) +from third_party.mobile_use.config.llm import LLM +from third_party.mobile_use.utils.logger import get_logger + +logger = get_logger(__name__) + +_PROBE_HTTP_TIMEOUT_S = 5.0 + + +def _artemis_managed(model: str) -> bool: + """Whether the model is an Artemis catalog alias (``artemis model`` + lifecycle commands apply) or a BYO name the endpoint serves itself.""" + try: + from artemis.config.local_models import LOCAL_MODELS + + return model in LOCAL_MODELS + except Exception: + return False + + +def _serve_port(api_base: str) -> str | None: + """Port from an api_base URL, for ``--port`` in serve guidance.""" + try: + from urllib.parse import urlparse + + port = urlparse(api_base).port + return str(port) if port else None + except Exception: + return None + + +def _iter_configured_llms() -> list[LLM]: + """Every LLM/LLMWithFallback configured in the active artemis.jsonc, + including per-node fallbacks.""" + from artemis.config.llm import parse_llm_config + + llms: list[LLM] = [] + cfg = parse_llm_config() + for name in type(cfg).model_fields: + node = getattr(cfg, name, None) + if isinstance(node, LLM): + llms.append(node) + fallback = getattr(node, "fallback", None) + if isinstance(fallback, LLM): + llms.append(fallback) + elif node is not None and hasattr(node, "model_fields_set"): + # Container (e.g. LLMConfigUtils) - walk its LLM fields too. + for sub_name in type(node).model_fields: + sub = getattr(node, sub_name, None) + if isinstance(sub, LLM): + llms.append(sub) + fallback = getattr(sub, "fallback", None) + if isinstance(fallback, LLM): + llms.append(fallback) + return llms + + +def _local_endpoints() -> dict[str, set[str]]: + """Map api_base -> set of model aliases required on it (custom provider).""" + endpoints: dict[str, set[str]] = {} + for llm in _iter_configured_llms(): + if llm.provider == "custom" and llm.api_base: + base = llm.api_base.rstrip("/") + endpoints.setdefault(base, set()).add(llm.model) + return endpoints + + +class LocalModelEndpointProbe(BaseProbe): + """Verify local model endpoints serve every configured custom model.""" + + @property + def probe_id(self) -> str: + return "local_model_endpoint" + + @property + def category(self) -> ProbeCategory: + return ProbeCategory.RUNTIME + + @property + def is_blocker(self) -> bool: + return True + + async def probe(self) -> ProbeResult: + try: + endpoints = _local_endpoints() + except Exception as e: + # A broken config is the SystemConfigProbe's job - don't double-report. + logger.debug(f"local model probe skipped: cannot parse LLM config: {e}") + endpoints = {} + + if not endpoints: + return ProbeResult( + id=self.probe_id, + category=self.category, + title="Local Model Endpoint", + status=ProbeStatus.PASS, + is_blocker=self.is_blocker, + summary="Not used", + description="No node is bound to a local custom endpoint; cloud providers serve all models.", + metadata={"endpoints": {}}, + actions=[ + ProbeAction( + action_type="hint", + label="Cloud Providers", + payload="Configured providers are remote; no local model install is required.", + ) + ], + ) + + reports: list[dict[str, Any]] = [] + missing_actions: list[ProbeAction] = [] + worst_summary = "" + managed_all: set[str] = set() + async with httpx.AsyncClient(timeout=_PROBE_HTTP_TIMEOUT_S) as client: + for base, required in sorted(endpoints.items()): + managed_all.update(a for a in required if _artemis_managed(a)) + try: + resp = await client.get(f"{base}/models") + resp.raise_for_status() + served = { + m["id"] + for m in resp.json().get("data", []) + if isinstance(m, dict) and isinstance(m.get("id"), str) + } + except Exception as e: + reports.append({"api_base": base, "reachable": False, "error": str(e)}) + worst_summary = f"{base} unreachable" + managed = sorted(a for a in required if _artemis_managed(a)) + if managed: + port = _serve_port(base) + port_arg = f" --port {port}" if port else "" + missing_actions.append( + ProbeAction( + action_type="command", + label="Start Model Server", + payload=f"artemis model serve {managed[0]}{port_arg}", + ) + ) + else: + missing_actions.append( + ProbeAction( + action_type="hint", + label="Start Model Server", + payload=( + f"Start the OpenAI-compatible model server for {base}" + " and re-run diagnostics." + ), + ) + ) + continue + missing = sorted(required - served) + reports.append( + { + "api_base": base, + "reachable": True, + "required": sorted(required), + "served": sorted(served), + "missing": missing, + } + ) + for alias in missing: + worst_summary = f"model '{alias}' not installed" + if _artemis_managed(alias): + missing_actions.append( + ProbeAction( + action_type="command", + label=f"Pull {alias}", + payload=f"artemis model pull {alias}", + ) + ) + else: + missing_actions.append( + ProbeAction( + action_type="hint", + label=f"Serve {alias}", + payload=( + f"'{alias}' is not served at {base}. Pull and serve" + " it with that server's own tooling" + " (e.g. `ollama pull`, a vLLM/LM Studio config)" + " or choose a model the endpoint already serves." + ), + ) + ) + + unreachable = [r for r in reports if not r.get("reachable")] + with_missing = [r for r in reports if r.get("missing")] + + if not unreachable and not with_missing: + served_total = sorted({m for r in reports for m in r.get("served", [])}) + return ProbeResult( + id=self.probe_id, + category=self.category, + title="Local Model Endpoint", + status=ProbeStatus.PASS, + is_blocker=self.is_blocker, + summary=f"Serving {len(served_total)} model(s)", + description=( + f"{len(reports)} local endpoint(s) reachable and all configured " + f"model aliases are pulled and serving." + ), + metadata={"endpoints": reports, "managed": sorted(managed_all)}, + actions=[ + ProbeAction( + action_type="hint", + label="Models Ready", + payload=f"Serving: {', '.join(served_total)}", + ) + ], + ) + + if unreachable: + description = ( + "Local model endpoint(s) unreachable: " + + ", ".join(r["api_base"] for r in unreachable) + + ". The agent cannot run until the server is up." + ) + else: + missing_all = sorted({m for r in with_missing for m in r["missing"]}) + description = ( + "Configured model(s) not installed on the local endpoint: " + + ", ".join(missing_all) + + ". Pull them before running a task." + ) + return ProbeResult( + id=self.probe_id, + category=self.category, + title="Local Model Endpoint", + status=ProbeStatus.FAIL, + is_blocker=self.is_blocker, + summary=worst_summary, + description=description, + metadata={"endpoints": reports, "managed": sorted(managed_all)}, + actions=missing_actions, + ) diff --git a/artemis/interfaces/cli/commands/model.py b/artemis/interfaces/cli/commands/model.py new file mode 100644 index 000000000..de41e6028 --- /dev/null +++ b/artemis/interfaces/cli/commands/model.py @@ -0,0 +1,478 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Local on-device model management (artemis model). + +ARTEMIS can pull and serve a local multimodal model itself — no external +serving stack required. ``pull`` downloads weights through huggingface_hub; +``serve`` runs ``mlx_vlm.server`` (installed via the ``local`` extra) as a +managed background process exposing an OpenAI-compatible endpoint that the +``custom`` LLM provider and the local-model readiness probe consume. +""" + +import json +import os +from pathlib import Path +import platform +import shutil +import signal +import subprocess +import sys +import time +from typing import Annotated + +from artemis.config.local_models import DEFAULT_LOCAL_MODEL, LOCAL_MODELS, resolve_model_ref +from third_party.mobile_use.utils.logger import get_logger +from rich.console import Console +from rich.table import Table +import httpx +import typer + +logger = get_logger(__name__) +model_app = typer.Typer(help="Pull and serve local on-device models.") + +_STATE_DIR = Path.home() / ".artemis" / "local_models" +_SERVE_TIMEOUT_S = 300.0 + +# Laya is not an HF-weights/mlx_vlm model: `pull` provisions a managed venv +# with the `laya[serve]` extra (the serve entry point arrived in 0.3.7), and +# `serve` spawns `laya-serve`, which speaks TypeSafe's Jev wire API +# (POST /v1/systemone) for the `decision.*` config layer. +_LAYA_VENV_DIR = Path.home() / ".artemis" / "laya-venv" +_LAYA_BIN = _LAYA_VENV_DIR / "bin" / "laya-serve" +_LAYA_REQUIREMENT = "laya[serve]>=0.3.7" + + +def _state_file(port: int) -> Path: + return _STATE_DIR / f"server-{port}.json" + + +def _read_state(port: int) -> dict | None: + path = _state_file(port) + if not path.exists(): + return None + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return None + + +def _pid_alive(pid: int) -> bool: + if pid <= 0: + return False + try: + import os + + os.kill(pid, 0) + except OSError: + return False + return True + + +def _require_local_extra() -> None: + try: + import mlx_vlm # noqa: F401 + except ImportError as e: + typer.secho( + "mlx-vlm is not installed. Install the local-model extra first:\n" + " pip install 'artemis[local]' # Apple Silicon only\n" + "On other platforms, run an OpenAI-compatible server (e.g. Ollama) " + "and point api_base at it instead.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) from e + + +def _pull_laya(spec) -> None: + """Provision the managed laya-serve runtime: a dedicated venv holding the + pinned ``laya[serve]`` install, kept out of the Artemis dependency tree. + Model weights download on first ``serve`` (LAYA_PRELOAD warms them).""" + python_bin = _LAYA_VENV_DIR / "bin" / "python" + if _LAYA_BIN.exists(): + typer.secho( + f"laya-serve already provisioned at {_LAYA_VENV_DIR}", fg=typer.colors.GREEN + ) + return + uv = shutil.which("uv") + typer.secho( + f"Provisioning Laya runtime at {_LAYA_VENV_DIR} ({_LAYA_REQUIREMENT})...", + fg=typer.colors.CYAN, + ) + try: + if uv: + subprocess.run( + [uv, "venv", "--python", "3.12", str(_LAYA_VENV_DIR)], check=True + ) + subprocess.run( + [ + uv, + "pip", + "install", + "--python", + str(python_bin), + _LAYA_REQUIREMENT, + ], + check=True, + ) + else: + subprocess.run( + [sys.executable, "-m", "venv", str(_LAYA_VENV_DIR)], check=True + ) + subprocess.run( + [str(python_bin), "-m", "pip", "install", _LAYA_REQUIREMENT], + check=True, + ) + except (subprocess.CalledProcessError, OSError) as e: + typer.secho(f"Laya provisioning failed: {e}", fg=typer.colors.RED) + raise typer.Exit(code=1) from e + typer.secho( + f"Ready: laya-serve installed. The '{spec.checkpoint}' checkpoint " + "(~800 MB) downloads on first `artemis model serve laya`.", + fg=typer.colors.GREEN, + ) + + +def _serve_laya(alias: str, spec, host: str, port: int) -> None: + """Spawn ``laya-serve`` in the managed venv behind the pid-file state used + by `model stop`/`status`. Exposes POST /v1/systemone (Jev wire API) and + GET /health — point ``decision.url`` at ``http://:/v1/systemone``.""" + if not _LAYA_BIN.exists(): + typer.secho( + "Laya is not installed. Run `artemis model pull laya` first.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + existing = _read_state(port) + if existing and _pid_alive(int(existing.get("pid", -1))): + typer.secho( + f"A server (pid {existing['pid']}) is already running on {host}:{port}.", + fg=typer.colors.YELLOW, + ) + return + env = os.environ.copy() + # CPU is the default: measured ~1s/decision under memory pressure vs ~25s + # on a contended MPS. A caller-provided LAYA_DEVICE in the environment wins. + env.update( + { + "LAYA_HOST": host, + "LAYA_PORT": str(port), + "LAYA_MODELS": spec.checkpoint, + "LAYA_PRELOAD": "1", + } + ) + env.setdefault("LAYA_DEVICE", "cpu") + _STATE_DIR.mkdir(parents=True, exist_ok=True) + log_path = _STATE_DIR / f"server-{port}.log" + typer.secho( + f"Starting {alias} ({spec.checkpoint}) on http://{host}:{port} — log: {log_path}", + fg=typer.colors.CYAN, + ) + with open(log_path, "ab") as log: + proc = subprocess.Popen( + [str(_LAYA_BIN)], + stdout=log, + stderr=subprocess.STDOUT, + env=env, + start_new_session=True, + ) + _state_file(port).write_text( + json.dumps( + { + "pid": proc.pid, + "alias": alias, + "repo": spec.checkpoint, + "host": host, + "port": port, + "kind": "laya", + } + ) + ) + base = f"http://{host}:{port}" + deadline = time.monotonic() + _SERVE_TIMEOUT_S + while time.monotonic() < deadline: + if proc.poll() is not None: + break + try: + resp = httpx.get(f"{base}/health", timeout=2.0) + if resp.status_code == 200: + typer.secho( + f"Serving {alias} at {base}/v1/systemone (pid {proc.pid}). " + f"Set decision.url to that endpoint. Stop with " + f"`artemis model stop --port {port}`.", + fg=typer.colors.GREEN, + ) + return + except httpx.HTTPError: + pass + time.sleep(2.0) + typer.secho( + f"laya-serve did not become ready within {_SERVE_TIMEOUT_S:.0f}s; " + f"check the log at {log_path}.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + + +@model_app.command("list") +def list_models() -> None: + """List local model aliases ARTEMIS can pull and serve.""" + console = Console() + table = Table(title="Local Models") + table.add_column("Alias", style="cyan") + table.add_column("Repository", style="green") + table.add_column("Notes", style="white") + for alias, spec in LOCAL_MODELS.items(): + table.add_row(alias, spec.repo, spec.description) + console.print(table) + console.print( + "\nAliases resolve through [bold]artemis model pull/serve [/bold]; " + "an org/repo id may be used instead of an alias." + ) + + +@model_app.command("pull") +def pull_model( + model: Annotated[ + str, + typer.Argument(help="Catalog alias or org/repo id to download."), + ] = DEFAULT_LOCAL_MODEL, +) -> None: + """Download model weights so they are ready to serve.""" + ref = resolve_model_ref(model) + if ref is None: + typer.secho( + f"Unknown model '{model}'. Run `artemis model list` or pass an org/repo id.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + alias, repo = ref + spec = LOCAL_MODELS.get(alias) + if spec is not None and spec.kind == "laya": + _pull_laya(spec) + return + _require_local_extra() + typer.secho(f"Pulling {alias} ({repo})...", fg=typer.colors.CYAN) + try: + from huggingface_hub import snapshot_download + + path = snapshot_download(repo) + except ImportError as e: + typer.secho( + "huggingface_hub is not installed. Install 'artemis[local]' first.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) from e + except Exception as e: + typer.secho(f"Pull failed for {repo}: {e}", fg=typer.colors.RED) + raise typer.Exit(code=1) from e + typer.secho(f"Ready: {alias} cached at {path}", fg=typer.colors.GREEN) + + +_VISION_SOFT_TOKENS_SUPPORTED = (70, 140, 280, 560, 1120) + + +def _snapshot_path(repo: str) -> Path | None: + """Locate the local HF snapshot for ``repo`` without hitting the network.""" + try: + from huggingface_hub import snapshot_download + + return Path(snapshot_download(repo, local_files_only=True)) + except Exception: + return None + + +def _apply_vision_soft_tokens(repo: str, tokens: int) -> bool: + """Write ``image_processor.max_soft_tokens`` into the pulled snapshot. + + ``mlx_vlm.server`` exposes no flag for the vision-token budget; the + image processor reads it from ``processor_config.json`` at load time, + so we adjust the pulled snapshot before spawning. Idempotent. + """ + snapshot = _snapshot_path(repo) + if snapshot is None: + return False + cfg_path = snapshot / "processor_config.json" + try: + cfg = json.loads(cfg_path.read_text()) + except (OSError, json.JSONDecodeError): + return False + image_proc = cfg.get("image_processor") + if not isinstance(image_proc, dict) or "max_soft_tokens" not in image_proc: + return False + image_proc["max_soft_tokens"] = tokens + cfg_path.write_text(json.dumps(cfg, indent=2)) + return True + + +@model_app.command("serve") +def serve_model( + model: Annotated[ + str, + typer.Argument(help="Catalog alias or org/repo id to serve."), + ] = DEFAULT_LOCAL_MODEL, + host: Annotated[str, typer.Option("--host", help="Bind address.")] = "127.0.0.1", + port: Annotated[int, typer.Option("--port", "-p", help="Bind port.")] = 8080, + vision_tokens: Annotated[ + int | None, + typer.Option( + "--vision-tokens", + help="Image soft-token budget (Gemma 4 supports " + f"{_VISION_SOFT_TOKENS_SUPPORTED}; catalog default used when omitted).", + ), + ] = None, +) -> None: + """Start a managed OpenAI-compatible model server in the background.""" + ref = resolve_model_ref(model) + if ref is None: + typer.secho( + f"Unknown model '{model}'. Run `artemis model list` or pass an org/repo id.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + if vision_tokens is not None and vision_tokens not in _VISION_SOFT_TOKENS_SUPPORTED: + typer.secho( + f"--vision-tokens must be one of {_VISION_SOFT_TOKENS_SUPPORTED}.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + alias, repo = ref + spec = LOCAL_MODELS.get(alias) + if spec is not None and spec.kind == "laya": + if vision_tokens is not None: + typer.secho("--vision-tokens is ignored for laya.", fg=typer.colors.YELLOW) + _serve_laya(alias, spec, host, port) + return + _require_local_extra() + tokens = vision_tokens + if tokens is None: + tokens = LOCAL_MODELS[alias].vision_soft_tokens if alias in LOCAL_MODELS else None + if tokens is not None: + if _apply_vision_soft_tokens(repo, tokens): + typer.secho(f"Vision soft tokens per image: {tokens}", fg=typer.colors.CYAN) + else: + typer.secho( + "Could not set the vision-token budget (model not pulled, or its " + "processor has no max_soft_tokens); serving the model default. " + "Run `artemis model pull` first.", + fg=typer.colors.YELLOW, + ) + existing = _read_state(port) + if existing and _pid_alive(int(existing.get("pid", -1))): + typer.secho( + f"A server (pid {existing['pid']}) is already running on {host}:{port}.", + fg=typer.colors.YELLOW, + ) + return + _STATE_DIR.mkdir(parents=True, exist_ok=True) + log_path = _STATE_DIR / f"server-{port}.log" + argv = [ + sys.executable, + "-m", + "mlx_vlm.server", + "--model", + repo, + "--host", + host, + "--port", + str(port), + ] + typer.secho( + f"Starting {alias} ({repo}) on http://{host}:{port}/v1 — log: {log_path}", + fg=typer.colors.CYAN, + ) + with open(log_path, "ab") as log: + proc = subprocess.Popen( + argv, + stdout=log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + _state_file(port).write_text( + json.dumps({"pid": proc.pid, "alias": alias, "repo": repo, "host": host, "port": port}) + ) + base = f"http://{host}:{port}/v1" + deadline = time.monotonic() + _SERVE_TIMEOUT_S + while time.monotonic() < deadline: + if proc.poll() is not None: + break + try: + resp = httpx.get(f"{base}/models", timeout=2.0) + if resp.status_code == 200: + typer.secho( + f"Serving {alias} at {base} (pid {proc.pid}). Stop with " + f"`artemis model stop --port {port}`.", + fg=typer.colors.GREEN, + ) + return + except httpx.HTTPError: + pass + time.sleep(2.0) + typer.secho( + f"Server did not become ready within {_SERVE_TIMEOUT_S:.0f}s; " + f"check the log at {log_path}.", + fg=typer.colors.RED, + ) + raise typer.Exit(code=1) + + +@model_app.command("stop") +def stop_model( + port: Annotated[int, typer.Option("--port", "-p", help="Port the server was started on.")] = 8080, +) -> None: + """Stop a managed model server started by `artemis model serve`.""" + state = _read_state(port) + if not state: + typer.secho(f"No managed server recorded on port {port}.", fg=typer.colors.YELLOW) + return + pid = int(state.get("pid", -1)) + if not _pid_alive(pid): + _state_file(port).unlink(missing_ok=True) + typer.secho("Server was not running; cleaned up state.", fg=typer.colors.YELLOW) + return + import os + + os.kill(pid, signal.SIGTERM) + _state_file(port).unlink(missing_ok=True) + typer.secho(f"Stopped {state.get('alias', 'server')} (pid {pid}).", fg=typer.colors.GREEN) + + +@model_app.command("status") +def status_model( + port: Annotated[int, typer.Option("--port", "-p", help="Port to check.")] = 8080, + host: Annotated[str, typer.Option("--host", help="Server host.")] = "127.0.0.1", +) -> None: + """Report whether a managed local model server is up.""" + state = _read_state(port) + if state and _pid_alive(int(state.get("pid", -1))): + typer.secho( + f"Running: {state.get('alias')} (pid {state['pid']}) on {state.get('host')}:{port}", + fg=typer.colors.GREEN, + ) + return + # laya-serve speaks /health, not the OpenAI /v1/models route. + for probe in ("v1/models", "health"): + try: + resp = httpx.get(f"http://{host}:{port}/{probe}", timeout=3.0) + if resp.status_code == 200: + detail = "" + if probe == "v1/models": + detail = f" serving {[m.get('id') for m in resp.json().get('data', [])]}" + typer.secho( + f"An unmanaged server answered /{probe}{detail} on {host}:{port}.", + fg=typer.colors.YELLOW, + ) + return + except httpx.HTTPError: + pass + typer.secho(f"No model server on {host}:{port}.", fg=typer.colors.RED) diff --git a/artemis/interfaces/cli/main.py b/artemis/interfaces/cli/main.py index 629d6b2cb..301904a4f 100644 --- a/artemis/interfaces/cli/main.py +++ b/artemis/interfaces/cli/main.py @@ -22,6 +22,7 @@ from artemis.interfaces.cli.commands.helper import helper_app from artemis.interfaces.cli.commands.init import init_command from artemis.interfaces.cli.commands.mcp import mcp_command +from artemis.interfaces.cli.commands.model import model_app from artemis.interfaces.cli.commands.run import run_command from artemis.interfaces.cli.commands.server import server_app from artemis.interfaces.cli.commands.server_lifecycle import ( @@ -64,6 +65,9 @@ app.command(name="mcp", help="Start the Artemis Model Context Protocol (MCP) server.")(mcp_command) app.add_typer(server_app, name="server", help="Cloud Run proxy and web dashboard server.") app.add_typer(trace_app, name="trace", help="Inspect and query execution traces.") +app.add_typer( + model_app, name="model", help="Pull and serve local on-device models." +) app.add_typer( helper_app, name="helper", help="Manage the Accessibility Helper APK on attached devices." ) diff --git a/artemis/llm/router.py b/artemis/llm/router.py index f2e6b42b6..4e94a7151 100644 --- a/artemis/llm/router.py +++ b/artemis/llm/router.py @@ -111,6 +111,13 @@ class ModelEndpoint(BaseModel): default=False, description="Whether to enable Google Search grounding for Gemini endpoints", ) + coordinate_format: str | None = Field( + default=None, + description=( + "Vision detector point convention, e.g. 'yx_1000' (Gemini default)," + " 'xy_1000', 'yx_px', 'xy_px', 'yx_norm', 'xy_norm'" + ), + ) def cache_key(self) -> tuple: """Key the client cache by endpoint settings, hashing the API key.""" @@ -286,7 +293,8 @@ def create_model(cls, endpoint: ModelEndpoint) -> BaseChatModel: api_key = ( endpoint.api_key or (settings.OPENAI_API_KEY.get_secret_value() if settings.OPENAI_API_KEY else None) - or os.environ.get("OPENAI_API_KEY", "EMPTY") + or os.environ.get("OPENAI_API_KEY") + or "EMPTY" ) base_url = endpoint.api_base or ( str(settings.OPENAI_BASE_URL) if settings.OPENAI_BASE_URL else None @@ -369,7 +377,7 @@ def create_model(cls, endpoint: ModelEndpoint) -> BaseChatModel: elif provider in (ModelProvider.OLLAMA, ModelProvider.VLLM, ModelProvider.CUSTOM): from langchain_openai import ChatOpenAI - api_key = endpoint.api_key or os.environ.get("OPENAI_API_KEY", "EMPTY") + api_key = endpoint.api_key or os.environ.get("OPENAI_API_KEY") or "EMPTY" base_url = endpoint.api_base or os.environ.get( "OPENAI_BASE_URL", "http://localhost:8000/v1" ) diff --git a/artemis/mcp/action_executor.py b/artemis/mcp/action_executor.py index 79b399cda..1d3d3e1e6 100644 --- a/artemis/mcp/action_executor.py +++ b/artemis/mcp/action_executor.py @@ -33,6 +33,7 @@ import ast import hashlib import json +import re from pathlib import Path from typing import Any @@ -55,6 +56,7 @@ compute_smart_swipe_coordinates, parse_swipe_parameters, ) +from artemis.utils.element_hit_test import find_element_at_point from artemis.utils.notes import ( format_list_notes_failure, format_list_notes_success, @@ -177,6 +179,23 @@ async def _execute_device_action( wire_name, wire_args, finalize, target_semantics = self._translate( raw_name, args, state, index_elements=index_elements ) + action_elements = self._index_elements(state, index_elements) + if ( + raw_name in _POINT_TARGET_ACTIONS + and not target_semantics.get("target_text") + ): + # Coordinate target: the model picked a point, so resolve + # what is actually under it for the same feedback path an + # index target gets. + desc_note = self._coordinate_target_note( + args, wire_args.get("target"), action_elements + ) + else: + desc_note = self._description_mismatch_note( + args, + target_semantics, + action_elements, + ) if raw_name in _POINT_TARGET_ACTIONS: # The normalized point the action was sent to: the recorded # coordinates of an index target (the model named an index, @@ -184,7 +203,18 @@ async def _execute_device_action( target_coordinates = wire_args.get("target") extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS) - if extension: + if desc_note and desc_note.get("suggestion"): + # Deterministic guard: the model's own description says it + # wants a different element than the index resolves to, and + # a listed element confidently matches the description. On + # a dense pad the wrong tap still corrupts state (observed: + # '1 key' narrated over the 5-key's JKL cell kept appending + # 5s), so refuse the tap and hand back the right index. + message = desc_note["text"] + res = ActionResult.failure( + raw_name, message, code=ActionCode.INVALID_ARGS + ) + elif extension: raw_result = await session.call_raw(wire_name, wire_args) message = next( (b.text for b in raw_result.content if getattr(b, "type", "") == "text"), @@ -194,6 +224,8 @@ async def _execute_device_action( else: res = await session.call(wire_name, wire_args) message = finalize(res) if finalize else res.message + if res.ok and desc_note: + message = f"{message}\n{desc_note['text']}" if res.ok or self._observe_despite_failure(raw_name, res): obs = await session.observe(settle_ms=400) @@ -482,6 +514,236 @@ def _norm_y(v: Any) -> int: } return [_norm_x(center[0]), _norm_y(center[1])], recorded + _DESC_STOPWORDS = frozenset( + { + "key", "button", "the", "a", "an", "icon", "field", "box", "switch", + "option", "item", "label", "element", "text", "tap", "click", "on", + "to", "of", "in", "and", "or", + } + ) + + # Spelled-out digits describe the same glyph as the numeral — "the five key" + # *is* the '5' cell on a dial pad, not a mismatch. + _DESC_DIGIT_WORDS = { + "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4", + "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", + } + + @classmethod + def _description_mismatch_note( + cls, + args: dict[str, Any], + semantics: dict[str, Any], + elements: list[dict[str, Any]] | None = None, + ) -> str | None: + """Warns when a stated target_description disagrees with the resolved element. + + Index targets resolve to an observed element whose text is of record; + ``target_description`` rides along unverified (Pro parity — it is only + *required* for coordinate targets). On dense lookalike screens the model + can narrate the wrong index ('1 key' while the index is the JKL cell of + the 5-key): the tap still executes, but the note names the real target + so the next turn can correct instead of trusting its own description. + When the element list is available it also names the index that does + match the description — on a 50-line pad a small model cannot reliably + rescan the list to find it. + """ + description = args.get("target_description") + text = semantics.get("target_text") + if not (isinstance(description, str) and description.strip()) or not text: + return None + tokens = cls._description_tokens(description) + if not tokens or any(t in text.lower() for t in tokens): + return None + rid = semantics.get("target_resource_id") + where = f" ({rid})" if rid else "" + note = ( + f"[NOTE] That index resolves to '{text}'{where}, which does not match" + f" the described target '{description}'." + ) + suggestion = cls._matching_element_hint(tokens, elements) + if suggestion: + note += f" The element matching '{description}' is {suggestion}." + else: + note += " Re-check the current element list if you meant a different element." + return {"text": note, "suggestion": suggestion} + + # Container classes a coordinate tap can "hit" without doing anything — + # the expanded alarm card is a CardView covering half the screen, and a + # tap on its body lands here instead of on the switch inside it. + _CONTAINER_CLASSES = frozenset( + { + "framelayout", "linearlayout", "relativelayout", "constraintlayout", + "viewgroup", "cardview", "recyclerview", "scrollview", + "nestedscrollview", "coordinatorlayout", "drawerlayout", "gridview", + "listview", "adapterview", "appbarlayout", "linearlayoutcompat", + } + ) + + # Widget words in a description pick a candidate by class — "toggle switch + # for 8:30 AM alarm" means the Switch among the several '8:30 AM alarm' + # labelled nodes, not the CardView sharing every one of its tokens. + _DESC_CLASS_WORDS = { + "switch": "switch", "toggle": "switch", "checkbox": "checkbox", + "radio": "radio", "slider": "seekbar", "seekbar": "seekbar", + "field": "edittext", "input": "edittext", "search": "edittext", + "button": "button", "tab": "tab", + } + + @classmethod + def _description_tokens(cls, description: str) -> set[str]: + return { + cls._DESC_DIGIT_WORDS.get(t, t) + for t in re.findall(r"[a-z0-9]+", description.lower()) + if t not in cls._DESC_STOPWORDS + } + + @classmethod + def _confident_element_hint( + cls, tokens: set[str], description: str, elements: list[dict[str, Any]] | None + ) -> str | None: + """Names the single best element match, or None when ambiguous. + + Scores each labelled element by how many description tokens it covers; + widget words in the description ("switch", "button", "field") first + filter candidates to the matching widget class, which separates "the + 8:30 AM alarm switch" from the card and chevron carrying the same text. + A hint is returned only when one candidate uniquely out-scores the rest. + """ + if not elements or not tokens: + return None + class_words = { + cls._DESC_CLASS_WORDS[t] + for t in re.findall(r"[a-z]+", description.lower()) + if t in cls._DESC_CLASS_WORDS + } + scored: list[tuple[int, Any, str, dict[str, Any]]] = [] + for pos, el in enumerate(elements): + label = str(el.get("text") or "").strip() + if not label: + continue + lower = label.lower() + score = sum( + 1 for t in tokens if t == lower or (len(t) > 1 and t in lower) + ) + if score: + scored.append((score, el.get("index", pos + 1), label, el)) + if not scored: + return None + if class_words: + preferred = [ + entry + for entry in scored + if any( + w in str(entry[3].get("class") or "").rsplit(".", 1)[-1].lower() + for w in class_words + ) + ] + if preferred: + scored = preferred + best = max(entry[0] for entry in scored) + tops = [entry for entry in scored if entry[0] == best] + if len(tops) != 1: + return None + _, idx, label, _el = tops[0] + return f"[{idx}] '{label}'" + + def _coordinate_target_note( + self, + args: dict[str, Any], + target_norm: Any, + elements: list[dict[str, Any]] | None, + ) -> dict[str, str] | None: + """Evaluates a coordinate tap against what actually sits under the point. + + For index targets ``target_description`` is checked against the resolved + element; a coordinate target has no resolved element, so hit-test the + point against the same indexed list: + + * a leaf-element hit is checked against the description like an index + resolution (the element under the point *is* the resolved target); + * a hit on a layout container, or on nothing listed, cannot perform the + described action — reject with a hint when the description identifies + one element confidently, else only note the likely dead tap; + * a container hit whose own label matches the description is left alone + (many actionable targets, e.g. dial-pad keys, are FrameLayouts). + """ + description = args.get("target_description") + if not (isinstance(description, str) and description.strip()): + return None + if not (isinstance(target_norm, (list, tuple)) and len(target_norm) == 2): + return None + width, height = self._screen_size() + x_px = int(target_norm[0]) * width // 1000 + y_px = int(target_norm[1]) * height // 1000 + hit, _source = find_element_at_point(elements, x_px, y_px) + description = description.strip() + tokens = self._description_tokens(description) + if hit is not None: + cls_name = str(hit.get("class") or "").rsplit(".", 1)[-1].lower() + if cls_name not in self._CONTAINER_CLASSES: + return self._description_mismatch_note( + args, + { + "target_text": hit.get("text"), + "target_resource_id": hit.get("resource_id"), + }, + elements, + ) + hit_label = str(hit.get("text") or "").strip().lower() + if tokens and any(t in hit_label for t in tokens): + class_words = { + self._DESC_CLASS_WORDS[t] + for t in re.findall(r"[a-z]+", description.lower()) + if t in self._DESC_CLASS_WORDS + } + if not class_words or any(w in cls_name for w in class_words): + return None + what = f"the '{hit.get('text')}' {cls_name} container" + else: + what = "no listed element" + suggestion = self._confident_element_hint(tokens, description, elements) + note = ( + f"[NOTE] [{target_norm[0]}, {target_norm[1]}] lands on {what}, which" + f" cannot perform '{description}'." + ) + if suggestion: + note += f" The element matching '{description}' is {suggestion}." + return {"text": note, "suggestion": suggestion} + note += " Re-aim at the intended element or use its index from the list." + return {"text": note, "suggestion": None} + + @staticmethod + def _matching_element_hint( + tokens: set[str], elements: list[dict[str, Any]] | None + ) -> str | None: + """Names the listed element(s) whose text matches the described tokens. + + Exact-text equality ranks first ('1' is the 1-key's digit cell, not a + substring of '(555) 015-5'); longer tokens may substring-match so + 'the dark theme switch' finds the 'Dark theme' row. At most two + candidates are named — a longer list would become the same wall of + lookalikes the model just failed to read. + """ + if not elements or not tokens: + return None + exact: list[str] = [] + partial: list[str] = [] + for pos, el in enumerate(elements): + label = str(el.get("text") or "").strip() + if not label: + continue + idx = el.get("index", pos + 1) + lower = label.lower() + if any(t == lower for t in tokens): + exact.append(f"[{idx}] '{label}'") + elif any(len(t) > 1 and t in lower for t in tokens): + partial.append(f"[{idx}] '{label}'") + candidates = exact or partial + if not candidates: + return None + return " or ".join(candidates[:2]) + @staticmethod def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]: """Validate and trim the coordinate target description for the action record.""" diff --git a/artemis/mcp/observation.py b/artemis/mcp/observation.py index 7b40eec5b..807ca82f0 100644 --- a/artemis/mcp/observation.py +++ b/artemis/mcp/observation.py @@ -29,7 +29,11 @@ from artemis.controllers.unified_controller import UnifiedMobileController from artemis.mcp.action_types import ActionCode, ObserveResult from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml -from artemis.utils.visualization import format_minimal_list_with_elements +from artemis.utils.visualization import ( + format_minimal_list_with_elements, + get_center_coordinates, + parse_bounds, +) from third_party.mobile_use.utils.logger import get_logger logger = get_logger(__name__) @@ -37,6 +41,57 @@ __all__ = ["observe"] +_CONTROL_CLASSES = ("switch", "checkbox", "radiobutton", "togglebutton") +_STATE_ATTRS = ("checked", "enabled", "selected", "clickable", "scrollable", "package") + + +def _control_only_elements(fused_xml: list[dict], elements: list[dict]) -> list[dict]: + """Metadata for stateful controls the compact list never renders. + + Checkable/toggle nodes carry verified device state but usually no text, + so ``format_minimal_list_with_elements`` skips them — a bare Settings + switch is invisible to code consumers as well as to the model. Appended + AFTER the labeled sequence: their ``index`` values continue the + sequence so positional index resolution for listed elements is + unchanged, while signatures and goal invariants can see the state. + No text line is produced — the model-facing list is unchanged. + """ + seen = { + (el.get("resource_id"), el.get("class"), tuple(el.get("bounds") or ())) for el in elements + } + controls: list[dict] = [] + index = len(elements) + 1 + for node in fused_xml: + if node.get("ocr_elements"): + continue + cls = str(node.get("class") or "") + if not (node.get("checkable") == "true" or cls.lower().endswith(_CONTROL_CLASSES)): + continue + bounds = parse_bounds(node.get("bounds")) + key = (node.get("resource-id"), node.get("class"), bounds) + if not bounds or key in seen: + continue + seen.add(key) + left, top, right, bottom = bounds + cx, cy = get_center_coordinates(left, top, right, bottom) + element = { + "index": index, + "center": [cx, cy], + "text": str(node.get("text") or node.get("content-desc") or "").strip(), + "bounds": [left, top, right, bottom], + "class": node.get("class"), + "resource_id": node.get("resource-id"), + "is_ocr": False, + "is_control_only": True, + } + for attr in _STATE_ATTRS: + if node.get(attr) is not None: + element[attr] = node.get(attr) + controls.append(element) + index += 1 + return controls + + async def observe( ctx: ArtemisContext, controller: UnifiedMobileController, @@ -89,6 +144,7 @@ async def observe( fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results) minimal_list_str, elements, _ = format_minimal_list_with_elements(fused_xml, width, height) + elements = elements + _control_only_elements(fused_xml, elements) except Exception as e: logger.error(f"Failed to fetch or fuse UI hierarchy: {e}") minimal_list_str = "Not available due to hierarchy error." diff --git a/artemis/resources/config/artemis.jsonc b/artemis/resources/config/artemis.jsonc index 517d73840..71e59add7 100644 --- a/artemis/resources/config/artemis.jsonc +++ b/artemis/resources/config/artemis.jsonc @@ -45,6 +45,20 @@ "provider": "openai", "model": "llama3.2-vision", "fallback": { "provider": "openai", "model": "llama3.2-vision" } + }, + // Fully local route via any OpenAI-compatible server — Artemis-managed + // models (`artemis model serve `) or BYO (Ollama, vLLM, LM Studio). + // Copy the provider/model/api_base keys into "default" or a node below; + // "custom" skips cloud credential checks and needs no API key. + "local-qwen": { + "provider": "custom", + "model": "qwen3.8-9b", + "api_base": "http://127.0.0.1:8080/v1", + "fallback": { + "provider": "custom", + "model": "qwen3.8-9b", + "api_base": "http://127.0.0.1:8080/v1" + } } }, @@ -74,6 +88,11 @@ // CRITICAL REQUIREMENT: Standard LLMs (including standard Gemini Flash, GPT-4o, Claude 3.7) lack sub-pixel spatial coordinate fine-tuning. // To achieve accurate [x, y] point and bounding box localization, this node MUST use specialized Gemini ER (Embodied Reasoning / Robotics) models: // e.g., "gemini-robotics-er-2-preview". Non-ER models will fail spatial coordinate detection. + // For a local vision model (e.g. qwen-vl via a local server) also set + // "coordinate_format" — detector points are normalized to [x, y] 0-1000: + // "yx_1000" (default, Gemini ER) | "xy_1000" (Qwen-VL verified) | + // "xy_px"/"yx_px" (absolute pixels of the processed image grid) | + // "xy_norm"/"yx_norm" (0-1 floats) "object_detector": { "model": "gemini-robotics-er-2-preview", "fallback": { diff --git a/artemis/sdk/agent.py b/artemis/sdk/agent.py index c0c0da08b..d96cafbaf 100644 --- a/artemis/sdk/agent.py +++ b/artemis/sdk/agent.py @@ -306,6 +306,9 @@ def _prepare_tracing(self, task: Task, context: ArtemisContext): enable_committee=self._config.enable_committee, committee_debate_rounds=self._config.committee_debate_rounds, disable_outputter=self._config.disable_outputter, + goal_invariants=list( + getattr(self._config.flash, "goal_invariants", None) or [] + ), outputter=self._config.outputter, explorer=self._config.explorer, explorer_versions=self._config.explorer_versions, diff --git a/artemis/sdk/builders/agent_config_builder.py b/artemis/sdk/builders/agent_config_builder.py index 16893864d..fb435ca26 100644 --- a/artemis/sdk/builders/agent_config_builder.py +++ b/artemis/sdk/builders/agent_config_builder.py @@ -54,6 +54,7 @@ def __init__(self): else: self._video_recording_tools_enabled = detect_video_tools_enabled() self._disable_planner_validation = not agent_cfg.planner_validation.enabled + self._decision = agent_cfg.decision self._enable_committee = agent_cfg.committee.enabled self._committee_debate_rounds = agent_cfg.committee.debate_rounds self._disable_checker = not agent_cfg.checker.enabled @@ -439,6 +440,7 @@ def _extra_config_fields(self) -> dict[str, Any]: "assert_failure_policy": self._assert_failure_policy, "disable_device_probes": self._disable_device_probes, "disable_planner_validation": self._disable_planner_validation, + "decision": self._decision, "enable_committee": self._enable_committee, "committee_debate_rounds": self._committee_debate_rounds, "disable_outputter": self._disable_outputter, diff --git a/artemis/sdk/types/agent.py b/artemis/sdk/types/agent.py index 10f4e1d9a..3a3b9aa03 100644 --- a/artemis/sdk/types/agent.py +++ b/artemis/sdk/types/agent.py @@ -15,6 +15,7 @@ from typing import Literal from artemis.config import ( + DecisionLayerConfig, ExplorerConfig, FlashProfileConfig, OutputterConfig, @@ -55,6 +56,7 @@ class AgentConfig(AgentConfigBase): assert_failure_policy: Literal["continue", "halt"] = "continue" disable_device_probes: bool = False disable_planner_validation: bool = False + decision: DecisionLayerConfig = Field(default_factory=DecisionLayerConfig) enable_committee: bool = False committee_debate_rounds: int = 2 disable_outputter: bool = False diff --git a/artemis/services/llm.py b/artemis/services/llm.py index ec91e6fa0..d9e936c90 100644 --- a/artemis/services/llm.py +++ b/artemis/services/llm.py @@ -595,6 +595,11 @@ def with_config(self, *args, **kwargs): def with_structured_output(self, *args, **kwargs): if hasattr(self.base_model, "with_structured_output"): + if self._provider_value() == ModelProvider.CUSTOM.value: + # The local OpenAI-compatible endpoint rejects strict JSON-schema + # response_format requests; function calling yields the same + # structured result through tool calls it does support. + kwargs.setdefault("method", "function_calling") return RobustChatModelWrapper( self.base_model.with_structured_output(*args, **kwargs), self.ctx, @@ -929,6 +934,9 @@ def _get_val(obj, attr, expected_type): return ModelEndpoint( provider=ModelProvider.from_string(provider_val), model_name=str(model_val), + api_base=_get_val(cfg, "api_base", str), + max_tokens=_get_val(cfg, "max_tokens", int), + coordinate_format=_get_val(cfg, "coordinate_format", str), temperature=_get_val(cfg, "temperature", (int, float)) or 0.0, timeout_seconds=_get_val(cfg, "timeout", (int, float)) or 60.0, thinking_budget=_get_val(cfg, "thinking_budget", int), diff --git a/artemis/tools/committee_tool.py b/artemis/tools/committee_tool.py index 28735c903..fbb2da5ed 100644 --- a/artemis/tools/committee_tool.py +++ b/artemis/tools/committee_tool.py @@ -55,6 +55,7 @@ get_recent_subgoal_hashes, ) from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -273,19 +274,20 @@ async def _execute_committee( async def run_agent_turn(llm, system_prompt, tools, agent_name): current_blackboard = blackboard_path.read_text(encoding="utf-8") - human_content = [ - { - "type": "text", - "text": f"Current Blackboard:\n{current_blackboard}", - } - ] + human_content = [] if screenshot_b64: human_content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"}, + "image_url": {"url": image_data_uri(screenshot_b64)}, } ) + human_content.append( + { + "type": "text", + "text": f"Current Blackboard:\n{current_blackboard}", + } + ) messages = [ SystemMessage(content=system_prompt), @@ -367,19 +369,20 @@ async def run_tool(tc): # Final Synthesis by Planner Avatar current_blackboard = blackboard_path.read_text(encoding="utf-8") - final_human_content = [ - { - "type": "text", - "text": f"Current Blackboard:\n{current_blackboard}", - } - ] + final_human_content = [] if screenshot_b64: final_human_content.append( { "type": "image_url", - "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"}, + "image_url": {"url": image_data_uri(screenshot_b64)}, } ) + final_human_content.append( + { + "type": "text", + "text": f"Current Blackboard:\n{current_blackboard}", + } + ) msg_final = [ SystemMessage( diff --git a/artemis/tools/explorer_tool.py b/artemis/tools/explorer_tool.py index 14a95b94f..83e86c1f0 100644 --- a/artemis/tools/explorer_tool.py +++ b/artemis/tools/explorer_tool.py @@ -62,6 +62,7 @@ from artemis.utils.element_hit_test import find_element_at_point from artemis.utils.visualization import draw_dots from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -479,8 +480,8 @@ def render_operator_blocks( logger.error(f"Failed to read annotated explorer image: {e}") return text return [ + {"type": "image_url", "image_url": {"url": image_data_uri(img_b64)}}, {"type": "text", "text": text}, - {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}}, ] diff --git a/artemis/tools/history/screenshot.py b/artemis/tools/history/screenshot.py index 94f55d343..50b886a51 100644 --- a/artemis/tools/history/screenshot.py +++ b/artemis/tools/history/screenshot.py @@ -32,6 +32,7 @@ from artemis.core.tool_failure import ToolFailure from artemis.utils.visualization import overlay_action_on_screenshot from third_party.mobile_use.utils.logger import get_logger +from artemis.utils.image_codec import image_data_uri logger = get_logger(__name__) @@ -64,8 +65,8 @@ def to_content_blocks(self) -> list[dict[str, Any]] | str: return self.description encoded = base64.b64encode(self.image_bytes).decode("utf-8") return [ + {"type": "image_url", "image_url": {"url": image_data_uri(encoded)}}, {"type": "text", "text": self.description}, - {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}}, ] diff --git a/artemis/utils/apple_vision_ocr.py b/artemis/utils/apple_vision_ocr.py new file mode 100644 index 000000000..e03df770c --- /dev/null +++ b/artemis/utils/apple_vision_ocr.py @@ -0,0 +1,106 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Standalone Apple Vision OCR worker — run as ``python apple_vision_ocr.py``. + +Reads image bytes on stdin and writes a JSON list of +``{"text": str, "position": [{"x": int, "y": int}, ...]}`` on stdout, +matching the Google Vision ``boundingPoly.vertices`` contract. + +This module is deliberately free of artemis imports and runs in a clean +interpreter: once cv2 (and its GPU/OpenCL contexts) is loaded in the host +process, ``VNRecognizeTextRequest`` fails intermittently with e5rt +compute-stream errors. A subprocess keeps the ML stack isolated. +""" + +# pyright: reportAttributeAccessIssue=false +# pyobjc exposes AppKit/Vision classes dynamically; static stubs have no symbols. + +import io +import json +import sys + + +def _vision_box_to_vertices(bounding_box, width: int, height: int) -> list[dict[str, int]]: + """Convert a Vision normalized bounding box (origin bottom-left) to + pixel-space vertices in Google's boundingPoly order: TL, TR, BR, BL.""" + x0 = bounding_box.origin.x * width + x1 = (bounding_box.origin.x + bounding_box.size.width) * width + y_top = (1.0 - bounding_box.origin.y - bounding_box.size.height) * height + y_bottom = (1.0 - bounding_box.origin.y) * height + return [ + {"x": round(x0), "y": round(y_top)}, + {"x": round(x1), "y": round(y_top)}, + {"x": round(x1), "y": round(y_bottom)}, + {"x": round(x0), "y": round(y_bottom)}, + ] + + +def ocr_image(image_bytes: bytes) -> list[dict]: + """On-device text recognition via Apple's Vision framework.""" + import Vision + from Foundation import NSData + from PIL import Image + + with Image.open(io.BytesIO(image_bytes)) as img: + width, height = img.width, img.height + + request = Vision.VNRecognizeTextRequest.alloc().init() + request.setRecognitionLevel_(Vision.VNRequestTextRecognitionLevelAccurate) + request.setUsesLanguageCorrection_(True) + # The first entry in recognitionLanguages dominates per-request — en-US + # first garbles CJK text. Per-observation language detection (revision 3) + # avoids ordering games: every supported language is enabled and Vision + # picks the right one for each text region. + request.setAutomaticallyDetectsLanguage_(True) + langs, _ = ( + Vision.VNRecognizeTextRequest.supportedRecognitionLanguagesForTextRecognitionLevel_revision_error_( + Vision.VNRequestTextRecognitionLevelAccurate, + request.revision(), + None, + ) + ) + if langs: + request.setRecognitionLanguages_(list(langs)) + + ns_data = NSData.dataWithBytes_length_(image_bytes, len(image_bytes)) + handler = Vision.VNImageRequestHandler.alloc().initWithData_options_(ns_data, None) + ok, error = handler.performRequests_error_([request], None) + if not ok: + raise RuntimeError(f"VNRecognizeTextRequest failed: {error}") + + results = [] + for observation in request.results() or []: + candidates = observation.topCandidates_(1) + if not candidates: + continue + text = str(candidates[0].string()).strip() + if not text: + continue + results.append( + { + "text": text, + "position": _vision_box_to_vertices(observation.boundingBox(), width, height), + } + ) + return results + + +def main() -> None: + image_bytes = sys.stdin.buffer.read() + json.dump(ocr_image(image_bytes), sys.stdout) + + +if __name__ == "__main__": + main() diff --git a/artemis/utils/image_codec.py b/artemis/utils/image_codec.py new file mode 100644 index 000000000..f791019fb --- /dev/null +++ b/artemis/utils/image_codec.py @@ -0,0 +1,52 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Base64 image data-URI helpers. + +OpenAI-compatible servers differ in how strictly they treat the declared +MIME type: some (e.g. strict local gateways) validate it against the actual +payload and reject a PNG byte stream labeled ``image/jpeg``. Always sniff the +magic bytes instead of assuming JPEG. +""" + +import base64 + +_JPEG_SOI = b"\xff\xd8\xff" +_PNG_SIG = b"\x89PNG" +_WEBP_RIFF = b"RIFF" +_WEBP_TAG = b"WEBP" + + +def sniff_image_mime(image_bytes: bytes) -> str: + """Return the ``image/*`` MIME type for encoded image bytes.""" + if image_bytes[:4] == _PNG_SIG: + return "image/png" + if image_bytes[:3] == _JPEG_SOI: + return "image/jpeg" + if image_bytes[:4] == _WEBP_RIFF and image_bytes[8:12] == _WEBP_TAG: + return "image/webp" + return "image/jpeg" + + +def image_data_uri(b64_data: str) -> str: + """Wrap a base64 payload in a data URI with the MIME type sniffed from its + decoded head. Unknown or corrupt prefixes keep the historic ``image/jpeg`` + label. + """ + try: + head = base64.b64decode(b64_data[:24]) + mime = sniff_image_mime(head) + except Exception: + mime = "image/jpeg" + return f"data:{mime};base64,{b64_data}" diff --git a/artemis/utils/ocr_api.py b/artemis/utils/ocr_api.py index 56b7c2356..8ad937ea8 100644 --- a/artemis/utils/ocr_api.py +++ b/artemis/utils/ocr_api.py @@ -12,7 +12,12 @@ # See the License for the specific language governing permissions and # limitations under the License. +import asyncio +import base64 +import json import os +from pathlib import Path +import sys from typing import Any import httpx @@ -22,6 +27,13 @@ logger = get_logger(__name__) _HTTP_CLIENT: httpx.AsyncClient | None = None +# Provider selection via ARTEMIS_OCR_PROVIDER: "auto" (default) prefers +# on-device Apple Vision on macOS and falls back to Google Cloud Vision when +# an API key exists; "apple" / "google" pin a single provider. +_OCR_PROVIDER_ENV = "ARTEMIS_OCR_PROVIDER" +_APPLE_VISION_SUPPORTED: bool | None = None +_APPLE_OCR_WORKER = Path(__file__).with_name("apple_vision_ocr.py") + def get_http_client() -> httpx.AsyncClient: global _HTTP_CLIENT @@ -30,29 +42,131 @@ def get_http_client() -> httpx.AsyncClient: return _HTTP_CLIENT -def is_ocr_configured() -> bool: - """Checks if a valid Google Vision OCR API key is configured.""" +def _ocr_provider() -> str: + return os.environ.get(_OCR_PROVIDER_ENV, "auto").strip().lower() + + +def _apple_vision_supported() -> bool: + """True on macOS when the pyobjc Vision bindings are importable and the + standalone worker script exists. The real OCR runs in a subprocess: once + cv2 is loaded in this process, VNRecognizeTextRequest intermittently + fails compiling its ANE compute stream (e5rt errors).""" + global _APPLE_VISION_SUPPORTED + if _APPLE_VISION_SUPPORTED is None: + if sys.platform != "darwin" or getattr(sys, "frozen", False): + _APPLE_VISION_SUPPORTED = False + else: + try: + import Vision # noqa: F401 + + _APPLE_VISION_SUPPORTED = _APPLE_OCR_WORKER.is_file() + except ImportError: + _APPLE_VISION_SUPPORTED = False + return _APPLE_VISION_SUPPORTED + + +def _google_vision_key() -> str | None: + """Raw Google Vision API key, or None (placeholders are not filtered).""" ocr_secret = settings.get_api_key("ocr") - if ocr_secret and ocr_secret.get_secret_value().strip(): - val = ocr_secret.get_secret_value().strip() - if val not in ("API_KEY", "your_google_cloud_vision_api_key_here"): - return True - for env_var in ("OCR_API_KEY", "VISION_API_KEY"): - val = os.environ.get(env_var) + return ( + (ocr_secret.get_secret_value() if ocr_secret else None) + or os.environ.get("OCR_API_KEY") + or os.environ.get("VISION_API_KEY") + ) + + +def _google_vision_key_present() -> bool: + """True when a non-placeholder Google Vision key is configured.""" + for val in (_google_vision_key(),): if ( val and val.strip() - and val.strip() not in ("API_KEY", "your_google_cloud_vision_api_key_here") + and val.strip() + not in ( + "API_KEY", + "your_google_cloud_vision_api_key_here", + ) ): return True return False +def is_ocr_configured() -> bool: + """True when some OCR provider is usable: Apple Vision on macOS, or a + configured Google Cloud Vision API key.""" + provider = _ocr_provider() + if provider == "apple": + return _apple_vision_available() + if provider == "google": + return _google_vision_key_present() + return _apple_vision_available() or _google_vision_key_present() + + +# A failing Vision attempt can stall for minutes inside the ObjC call (e5rt +# compute-stream compile), so every subprocess invocation is time-bounded and +# retried once in a fresh interpreter before degrading. When the OS-level +# Vision ML service is wedged, every call would burn 2x timeout forever — +# after consecutive failures the provider is disabled until a success resets +# the counter. +_APPLE_OCR_TIMEOUT_S = float(os.environ.get("APPLE_OCR_TIMEOUT", "20")) +_APPLE_OCR_MAX_CONSECUTIVE_FAILURES = 2 +_apple_vision_consecutive_failures = 0 + + +def _apple_vision_available() -> bool: + return ( + _apple_vision_supported() + and _apple_vision_consecutive_failures < _APPLE_OCR_MAX_CONSECUTIVE_FAILURES + ) + + +async def _apple_vision_subprocess(image_bytes: bytes) -> list[dict[str, Any]]: + proc = await asyncio.create_subprocess_exec( + sys.executable, + str(_APPLE_OCR_WORKER), + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + out, err = await asyncio.wait_for( + proc.communicate(input=image_bytes), timeout=_APPLE_OCR_TIMEOUT_S + ) + except BaseException: + proc.kill() + await proc.wait() + raise + if proc.returncode != 0: + tail = err.decode(errors="replace")[-300:] if err else "" + raise RuntimeError(f"Apple Vision worker exited {proc.returncode}: {tail}") + return json.loads(out or b"[]") + + +async def _run_apple_vision(image_bytes: bytes) -> list[dict[str, Any]]: + global _apple_vision_consecutive_failures + try: + results = await _apple_vision_subprocess(image_bytes) + except Exception as e: + logger.warning(f"Apple Vision OCR attempt failed ({e}); retrying once") + try: + results = await _apple_vision_subprocess(image_bytes) + except Exception: + _apple_vision_consecutive_failures += 1 + raise + _apple_vision_consecutive_failures = 0 + return results + + async def perform_ocr( screenshot_b64: str, client: httpx.AsyncClient | None = None, ) -> list[dict[str, Any]]: - """Calls Google Vision API for text recognition on an image. + """Runs text recognition on an image with the selected provider. + + Provider order under "auto": Apple Vision on-device (macOS) first, then + Google Cloud Vision when an API key is configured. An empty Apple result + is authoritative — Google is only consulted when Apple Vision is + unavailable or raises. Args: screenshot_b64: Base64 encoded screenshot image. @@ -61,12 +175,20 @@ async def perform_ocr( Returns: A list of dictionaries containing detected text and position vertices. """ - ocr_secret = settings.get_api_key("ocr") - api_key = ( - (ocr_secret.get_secret_value() if ocr_secret else None) - or os.environ.get("OCR_API_KEY") - or os.environ.get("VISION_API_KEY") - ) + provider = _ocr_provider() + if provider in ("auto", "apple"): + if _apple_vision_available(): + try: + return await _run_apple_vision(base64.b64decode(screenshot_b64)) + except Exception as e: + logger.warning(f"Apple Vision OCR failed: {e}") + elif provider == "apple": + logger.warning("ARTEMIS_OCR_PROVIDER=apple but Apple Vision is unavailable") + return [] + if provider == "apple": + return [] + + api_key = _google_vision_key() if not api_key: return [] diff --git a/artemis/utils/visualization.py b/artemis/utils/visualization.py index c6c2ddd2e..ac76fc458 100644 --- a/artemis/utils/visualization.py +++ b/artemis/utils/visualization.py @@ -427,6 +427,12 @@ def is_duplicate(text_val: str, cx: int, cy: int) -> bool: "resource_id": node.get("resource-id"), "is_ocr": False, } + # Raw uiautomator state attributes, carried for code-side + # verification (signatures, state predicates). They are NOT + # rendered into the compact text — see issue #187 for that. + for state_attr in ("checked", "enabled", "selected", "package"): + if node.get(state_attr) is not None: + element[state_attr] = node.get(state_attr) if not text.strip(): element["is_hint"] = True if error: diff --git a/config/artemis.gemma4.jsonc b/config/artemis.gemma4.jsonc new file mode 100644 index 000000000..ea7e1618a --- /dev/null +++ b/config/artemis.gemma4.jsonc @@ -0,0 +1,333 @@ +{ + // ============================================================================== + // ☕ ARTEMIS Autonomous Agent & Framework Configuration — GEMMA 4 SPIKE + // ============================================================================== + // Single-model deployment: every node (reasoning + vision + grounding) resolves + // to one gemma4-e4b endpoint on the local OpenAI-compatible model server. + // + // ACTIVATION (pick one): + // export ARTEMIS_ARTEMIS_JSONC=config/artemis.gemma4.jsonc + // # or rename this file over artemis.jsonc + // + // PREREQUISITE: the :8080 platform must actually serve the "gemma4-e4b" alias + // (curl http://127.0.0.1:8080/v1/models). Until it does, keep artemis.jsonc. + // + // Grounding notes: + // - Gemma 4 emits detection natively as {"box_2d": [ymin,xmin,ymax,xmax]} + // on a 0-1000 grid = this repo's default "yx_1000" coordinate_format, so + // no override is set. The detector now derives "point" from the box center. + // - Operator targets already use [x,y] on the same 0-1000 grid. + + // ------------------------------------------------------------------------------ + // 1. Global Default Model (Applied to all sub-agents unless overridden below) + // ------------------------------------------------------------------------------ + "default": { + "provider": "custom", + // One unified multimodal model serves every role; the server alias must + // match the platform's registry entry (Gemma 4 E4B via the mlx-vlm provider, + // e.g. mlx-community/gemma-4-E4B-it-4bit). + "model": "gemma4-e4b", + "api_base": "http://127.0.0.1:8080/v1", + // Local reasoning + vision encoder; keep the same generous timeout the + // qwen deployment needed for thinking preambles. + "timeout": 300, + "fallback": { + "provider": "custom", + "model": "gemma4-e4b", + "api_base": "http://127.0.0.1:8080/v1", + "timeout": 300 + } + }, + + // ------------------------------------------------------------------------------ + // 2. Ready-to-Use Presets (Switch easily in code or CLI) + // ------------------------------------------------------------------------------ + "presets": { + "gemini-flagship": { + "provider": "google", + "model": "gemini-3.8-flash", + "fallback": { "provider": "google", "model": "gemini-3.7-flash" } + }, + "gemini-flash": { + "provider": "google", + "model": "gemini-2.5-flash", + "fallback": { "provider": "google", "model": "gemini-2.0-flash" } + }, + "openai-gpt4o": { + "provider": "openai", + "model": "gpt-4o", + "fallback": { "provider": "openai", "model": "gpt-4o-mini" } + }, + "cost-saving": { + "provider": "google", + "model": "gemini-3.5-flash-lite", + "fallback": { "provider": "google", "model": "gemini-3.1-flash-lite" } + }, + "local-ollama": { + "provider": "openai", + "model": "llama3.2-vision", + "fallback": { "provider": "openai", "model": "llama3.2-vision" } + }, + // Unified single-model route via a local OpenAI-compatible server. + "local-gemma4": { + "provider": "custom", + "model": "gemma4-e4b", + "api_base": "http://127.0.0.1:8080/v1", + "fallback": { + "provider": "custom", + "model": "gemma4-e4b", + "api_base": "http://127.0.0.1:8080/v1" + } + } + }, + + // ------------------------------------------------------------------------------ + // 3. Agent & Sub-Agent Node Customizations (Optional - Inherits from 'default') + // ------------------------------------------------------------------------------ + "nodes": { + // 🧠 Planner: High-level goal decomposition and strategic step planning + "planner": { + "thinking_level": "high" + }, + + // 🎯 Operator: vision-capable by inheritance now — one model serves all roles. + "operator": { + "thinking_level": "medium", + "include_thoughts": true + }, + + // 🎬 Video Analyzer: inherits the unified endpoint; universal multi-model path + // feeds extracted keyframes as images. + "video_analyzer": {}, + + // 👁️ Object Detector: Gemma 4 native box_2d output is yx_1000 = the detector's + // declared default, so no coordinate_format override. box_2d collapses to a + // center point (and stays available as normalized bounds) downstream. + "object_detector": { + "timeout": 300 + }, + + // 🔍 Explorer: inherits the unified endpoint. + "explorer": { + "thinking_level": "medium" + }, + + // 📋 Summarizer: Aggregates step history and generates structured outcome + "summarizer": {}, + + // 🔽 Hopper: Fast lightweight package name locator + "hopper": {}, + + // 📝 Outputter: Synthesizes execution trace into final output report + "outputter": {}, + + // 🪵 Log Analyzer: grounding is a Gemini-only feature — ineffective here. + "log_analyzer": {}, + + // 🩺 Diagnoser Expert + "diagnoser_expert": {}, + // 🏛️ Committee Council Members + "planner_avatar": {}, + "history_analyzer_expert": {}, + // 🔎 Checker: Plan-declared checkpoint verification and exit final review. + "checker": { + "thinking_level": "medium", + "include_thoughts": true + } + }, + + // ------------------------------------------------------------------------------ + // 4. Runtime Mode & Agent Capabilities Configuration (Dual-Engine: Flash vs. Pro) + // ------------------------------------------------------------------------------ + "agent": { + // ========================================================================== + // ⚡ 4.1 FLASH Profile Options (Reactive Single-Agent Lightweight Loop) + // ========================================================================== + // Best for: Simple, fast, deterministic tasks; runs unbounded by default. + // Execution: Observe-Think-Act single LLM reactive loop without graph overhead. + "flash": { + "max_turns": 0, // Maximum reactive turns before stopping (0 = unlimited; history is bounded by transcript compression, not by a turn cap) + "explorer_mode": "flash", // Explorer tier for the Flash runner: 'flash' | 'pro' | 'ultra' (see top-level "explorer") + // 🧠 Step Summarizer: Asynchronous visual context compressor + // Summarizes intermediate steps in background without blocking the runner, + // replacing images only after strictly objective, high-density summaries are ready. + // LOCAL-MODEL NOTE: the visual-transition lens bypasses the LLM router + // (get_google_llm) — with no GOOGLE_API_KEY it crashes runner init. + // Disabled for the local-provider deployment. + "step_summarizer": { + "enabled": false, // Whether background step summarization is active + "model": "gemini-3.5-flash-lite", // Lightweight model for fast background inference + "prune_history_xml": true // Prune outdated heavy XML trees from historical steps + } + }, + // ========================================================================== + // 🚀 4.2 PRO Profile Options (Deep Multi-Agent Graph Closed-Loop) + // ========================================================================== + // Best for: Complex, exploratory, dynamic multi-branch tasks. + // Execution: LangGraph multi-node graph with a living task plan, a pre-execution safety net (single actions) / fast-action bursts (multi-action turns), + // execution-incident recovery handled by the Operator itself (no separate repair agent), and a read-only Checker for checkpoint verification + exit final review. + "pro": { + // 🧭 Explorer: Perception sub-agent under Pro profile ('flash', 'pro', or 'ultra') + "explorer": { + "mode": "flash" // Explorer tier for Pro nodes (Operator/Validator): 'flash' | 'pro' | 'ultra' + }, + // 🎯 Planner Validation: Async verification when task plan milestones change + "planner_validation": { + "enabled": true // On by default: every milestone text change is reviewed (advisory: hint + reason only, never rolled back) + }, + // 🏛️ Committee: Multi-Agent Council debate tool for Operator + "committee": { + "enabled": false, + "debate_rounds": 2 + }, + // 🔎 Checker: Plan-declared checkpoint verification (optimistic async) + exit final review + "checker": { + "enabled": true, // Master switch (compat alias): false disables both gates below + "midway_checks": false, // Plan-declared midway checkpoints (off in the factory layering) + "final_check": true, // Exit final review against the user's original goal (on by default) + "max_iterations": 20, + "final_check_max_attempts": 3, + "checkpoint_max_repairs": 2, + "max_concurrent_checkpoints": 3, + "checkpoint_timeout": 180.0, + "settlement_timeout": 120.0, + "assert_failure_policy": "continue", // 'continue' or 'halt' on assert failures + "device_probes": true // Checker's enumerated read-only device probes + }, + // 🎬 Video Analyzer & Screen Recording: Continuous screen recording via scrcpy and temporal analysis + "video_analyzer": { + "enabled": true, // Whether continuous video recording and video tools are active (default: auto-detected) + "enable_ledger": true + } + }, + // ========================================================================== + // 🌐 4.3 Shared Capabilities & Global Overrides + // ========================================================================== + // Shared step memory and history compression + // Transcript ledger + layered compression for the Pro operator. + "memory": { + // Background summarization runtime (visual/capsule lenses) + "runtime": { + "max_concurrency": 2, // 1→2 per 2026-09-01 baseline: serial queue pushed summary-ready P90 past the scrub window + "retry_limit": 3, + "flush_timeout_s": 30 + }, + "transcript": { + // ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message + // prompt path byte-for-byte (L2/L3 chunk compression rides this same + // flag). See services/token_meter.py for measured session token usage + // and cache-hit ratios. + // LOCAL-MODEL NOTE: transcript + capsule lens are Google-bound + // (get_google_llm) — disabled under the local-provider deployment. + "enabled": false, + "context_budget_tokens": 80000, + // Below start_ratio of the operator's own measured prompt size, closed + // segments only prepare their summary; the original turns are kept in + // context and replaced once occupancy reaches this ratio. Measurements + // come from the operator; other agents' calls do not affect this gate. + "start_ratio": 0.35, + "soft_ratio": 0.7, + "hard_ratio": 0.9, + "min_active_steps": 5, + // Screenshot edge: 6 while the operator's measured context is below + // start_ratio (nothing is being replaced yet), 3 from there up. A + // screenshot reaching the edge is replaced in place by its visual + // summary (the 'Current Screenshot' label leaves with it) and never + // restored; the depth only decides how long new ones stay. + "image_scrub_depth": 3, + "image_scrub_depth_relaxed": 6, + "pending_grace_steps": 3, + // Text edge: once a newer observation exists (depth 1) the UI element + // list, the plan recitation and the per-turn ephemeral blocks + // (reminders, hints, guidance wrappers) leave the older observation. + // User instruction bodies remain in history. The text edge stays + // shallow on purpose: only the live observation carries an indexed + // element list, so a stale [n] index can never be picked up as a + // target. + "xml_scrub_depth": 1, + // Local historical-state hint (no model call); distance calibrated + // 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4, + // different screens >=7). + "similarity_hint": true, + "similarity_max_distance": 5 + }, + // Segment (L2) chunk compression: three-band chunks, era folding + "chunking": { + "max_steps": 12, + // Size/pressure closes wait until at least this many turns are + // eligible, so heavy turns do not become one-step chunks (one capsule + // call each); milestone closes are exempt, the hard line waives it. + "min_steps": 3, + "target_source_tokens": 2000, + "model": "gemini-3.8-flash", + "max_chunks": 8 + // "max_eras": 8 // independent era cap; omitted = follows max_chunks + }, + // search_history: deterministic keyword / step-range lookup over stored history + "recall": { + "enabled": true, + "max_results": 5, + "max_text_tokens": 2000, + "screen_scan_steps": 150 + }, + // replay_steps: full step replays (whole trailing steps are dropped over budget) + "replay": { + "max_steps": 5, + "max_tokens": 12000 + } + }, + // 🎬 Screen Recording & Video Analyzer Configuration + "video_analyzer": { + "enabled": true, // Whether continuous screen recording via scrcpy is active (default: auto-detected) + "enable_ledger": true + // "processing": "auto", // How Gemini sub-agents watch a clip: auto (agentic on Gemini 3.6+ Flash, static elsewhere) | agentic | static + // "agentic_chunk_size_seconds": 600, // Longest clip one agentic sub-agent takes; static mode keeps chunk_size_seconds (60) + // "agentic_call_timeout_seconds": 300 // Hard timeout for one agentic sub-agent response + }, + // 📝 Outputter: Post-execution report synthesis and structured schema extraction + // Reviews action history, notes, and verifies goal achievement to synthesize final output. + "outputter": { + "enabled": true, // Whether Outputter is mounted to synthesize final outputs + "force_synthesis": false // Force Outputter synthesis even without explicit output schema + }, + // 🧠 Decision Layer (System One): a local decision model (Laya via Ollama + // or a Jev-compatible endpoint) answers cheap typed junctions — yes/no + // gates, categorical picks, rubric scores — in ~100ms instead of a full + // Gemma turn. Advisory only: below-threshold or unreachable answers + // escalate to the existing model path. Text/JSON state only — no pixels. + "decision": { + "enabled": false, // opt-in; off = upstream-identical behavior + // Jev wire API (POST /v1/systemone). `artemis model pull laya` + + // `artemis model serve laya -p 11435` provisions a matching server. + "url": "http://127.0.0.1:11435/v1/systemone", + "model": "typed-decisions", + // Generous: laya-serve serializes requests — a stalled predecessor plus + // memory pressure pushes single decisions well past 5s (measured ~10-25s). + "timeout_seconds": 15.0, + // Hybrid Flash loop: Gemma plans once, then per turn describes the + // screen (text only — no tool-call envelopes to mangle), and the + // decision model picks the next action from a fixed menu of element + // taps + meta-actions. Inconclusive/low-confidence verdicts escalate + // to the standard tool-call turn. + "flash_loop": false + }, + // 🧭 Explorer tier: the perception sub-agent behind `ask_explorer`. + // Tiers are a user setting; calling agents never see which tier is active. + // flash - one-shot visual detection on the screenshot (fastest) + // pro - 3-turn reasoning loop: UI-tree search + coordinate audit + detection + // ultra - 8-turn loop adding zoom, OCR and pixel-level image processing (slowest) + // `flash.explorer_mode` and `pro.explorer.mode` above are the per-profile knobs; + // `caching` (Gemini explicit context caching) defaults per tier: off for pro, on for ultra. + "explorer": { + "default_version": "flash" // Fallback tier when no profile knob applies + // "caching": true, // Uncomment to force context caching for pro/ultra + }, + // Advanced: per-agent tier override, e.g. {"validator": "ultra"}. Takes + // precedence over the profile knobs; leave empty to let them decide. + "explorer_versions": {}, +// 🚫 Per-agent Tool Denylists + "denylisted_tools": { + "explorer": [] // Perception tools to withhold from the Explorer, e.g. "ask_image_processor" + } + } +} diff --git a/docs/assets/gemma4-e4b-flash-safari.png b/docs/assets/gemma4-e4b-flash-safari.png new file mode 100644 index 000000000..839cf178c Binary files /dev/null and b/docs/assets/gemma4-e4b-flash-safari.png differ diff --git a/docs/local_models.md b/docs/local_models.md new file mode 100644 index 000000000..8f2efa8f3 --- /dev/null +++ b/docs/local_models.md @@ -0,0 +1,110 @@ +# Running ARTEMIS on a Local Model + +ARTEMIS can drive every agent node (planner, operator, checker, flash, +grounding, OCR helpers) from a single local multimodal model — and it can +pull and serve that model itself, no external serving stack or cloud key +required. + +Reference setup: **Gemma 4 E4B (4-bit, MLX)** on Apple Silicon. +This guide covers local serving and the Flash profile. Runtime memory and +latency depend on image resolution, context length, and background load. + +## 1. Install the local-model extra (Apple Silicon) + +```bash +pip install 'artemis[local]' # or: uv pip install -e '.[local]' +``` + +This installs `mlx-vlm` + `huggingface-hub`, the serving stack ARTEMIS +manages for you. (On other platforms, skip to +[BYO server](#byo-any-openai-compatible-server).) + +## 2. Pull and serve the model + +```bash +artemis model list # known aliases +artemis model pull gemma4-e4b # download weights (~5 GB) +artemis model serve gemma4-e4b # OpenAI-compatible server on :8080 +``` + +`serve` runs `mlx_vlm.server` as a managed background process, waits for +readiness, and prints the endpoint. Lifecycle: + +```bash +artemis model status # is the managed server up? +artemis model stop # shut it down +``` + +The served aliases appear under `GET /v1/models` — the same names the +ARTEMIS config's `"model"` field must match. + +## 3. Point ARTEMIS at it + +A ready-made config ships with this repo: + +```bash +export ARTEMIS_ARTEMIS_JSONC=config/artemis.gemma4.jsonc +# or copy it over your artemis.jsonc +``` + +The `default` block (every node inherits it unless overridden): + +```jsonc +"default": { + "provider": "custom", // skips cloud credential checks + "model": "gemma4-e4b", // must match /v1/models exactly + "api_base": "http://127.0.0.1:8080/v1", // the managed server + "timeout": 300, // local decode is slower; keep generous +} +``` + +## 4. Verify + +```bash +uv run artemis doctor +``` + +The **Local Model Endpoint** probe checks the endpoint is reachable and that +every configured alias is actually served: + +- ✅ `Serving N model(s)` — ready +- ❌ `model 'X' not installed` → action: `artemis model pull X` +- ❌ `endpoint unreachable` → action: `artemis model serve X --port 8080` + +The same probe backs the web console's setup wizard and `mobile_diagnose`. +For catalog aliases the console shows **Run** buttons that execute +`artemis model pull` / `serve` in the background with live logs; other +servers get copyable commands for their own tooling. + +## 5. Run + +```bash +uv run artemis run "Open Settings" --profile flash +``` + +## BYO: any OpenAI-compatible server + +ARTEMIS only needs `/v1/chat/completions` with text + `image_url` + tool +calls. Ollama, LM Studio, llama.cpp, vLLM all work — pull the model there, +set `api_base` + `"model"` to match its `/v1/models`, and `artemis doctor` +verifies the same way (remediation hints fall back to generic pull/serve +guidance for non-catalog aliases). + +## Tuning notes for small local models + +- **Timeouts**: `LLM_FLASH_HARD_TIMEOUT_SECONDS` (default 180 s) caps Flash + turns; keep `timeout` ≥ 300 in the config for thinking-style models. +- **Images**: strict servers cap images-per-request — Flash prunes + intermediate screenshots automatically; no action needed. +- **Vision detail**: Gemma 4's image processor defaults to 280 soft tokens + per image (≈528×1152 effective for a phone screenshot). `artemis model + serve` applies the catalog's recommended 1120 (≈near-original detail, + +840 prompt tokens/image) — measured to fix grounding misses and + hallucinated detections on dense screens. Override with + `--vision-tokens {70,140,280,560,1120}`; BYO servers keep their own + defaults. +- **Structured output**: strict JSON-schema `response_format` is not + required — ARTEMIS uses function-calling for structured nodes. +- **Termination**: Flash uses `report_task_status` for completion and failure. + This change does not alter the Pro plan/convergence contract. +- **Memory**: budget ~6 GB for a 4-bit 4B vision model + KV cache. diff --git a/pyproject.toml b/pyproject.toml index e5cd095fe..98772717a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -68,6 +68,9 @@ dependencies = [ "scipy>=1.13.0", # System, Utilities & Telemetry "psutil>=5.9.0", + # On-device OCR via Apple Vision (macOS only); guarded import — the OCR + # seam degrades to Google Cloud Vision or disabled without it. + "pyobjc-framework-Vision>=10.0,<13; sys_platform == 'darwin'", "retry>=0.9.2", "posthog>=3.5.0", "uuid-utils>=0.9.0", @@ -86,6 +89,13 @@ dev = [ "pyright>=1.1.389", "pre-commit>=3.5.0", ] +# `artemis model pull/serve`: managed local model serving via mlx-vlm. +# Apple Silicon only; other platforms should run an OpenAI-compatible +# server (e.g. Ollama) and point api_base at it. +local = [ + "mlx-vlm>=0.7.6,<1.0; sys_platform == 'darwin' and platform_machine == 'arm64'", + "huggingface-hub>=0.30,<2.0", +] [project.scripts] artemis = "artemis.interfaces.cli.main:cli" diff --git a/tests/conftest.py b/tests/conftest.py index b99e8d4dd..0b148fed0 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -32,6 +32,20 @@ def mock_driver(): return MockDeviceDriver(device_id="fixture-mock-device", width=1080, height=2400) +@pytest.fixture(autouse=True) +def _reset_apple_vision_ocr_breaker(): + """Isolate the Apple Vision OCR circuit-breaker counter between tests. + + Tests exercising the real ``perform_ocr`` path with undecodable images + trip the process-global failure counter and would silently disable the + provider for every later test in the session.""" + import artemis.utils.ocr_api as ocr_api + + ocr_api._apple_vision_consecutive_failures = 0 + yield + ocr_api._apple_vision_consecutive_failures = 0 + + def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: """Attach test-layer markers according to the owning test directory.""" for item in items: diff --git a/tests/integration/flash_live/.gitignore b/tests/integration/flash_live/.gitignore new file mode 100644 index 000000000..0b0669897 --- /dev/null +++ b/tests/integration/flash_live/.gitignore @@ -0,0 +1,3 @@ +local-config.json +__pycache__/ +*.pyc diff --git a/tests/integration/flash_live/analyze.py b/tests/integration/flash_live/analyze.py new file mode 100644 index 000000000..ebc6c1ba5 --- /dev/null +++ b/tests/integration/flash_live/analyze.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +"""Aggregate the suite campaign into a failure-mode report. + +Reads every /__s/result.json + run.log, groups by task and by +failure class, and writes /analysis.json + a printed table. +""" +import json +import re +import statistics +import sys +from pathlib import Path + +OUT = Path(sys.argv[1] if len(sys.argv) > 1 else "/tmp/suite-out") + + +def classify(result, log_text): + if result.get("verified_success"): + return "verified" + err = (result.get("error") or "").lower() + runner = result.get("runner_result") or {} + status = runner.get("status") + expl = (runner.get("explanation") or "").lower() + if "timed out" in err or "timeout" in err: + return "timeout" + if status == "failed" and ("suppressed" in expl or "no effect" in expl): + return "dead_tap_escalation" + if result.get("reported_completed") and not result.get("state_achieved"): + return "claimed_without_state" + if status == "failed": + return "runner_failed" + if not result.get("reported_completed"): + return "incomplete" + return "other" + + +rows = [] +for d in sorted(OUT.iterdir()): + rj = d / "result.json" + if not rj.exists() or not d.is_dir(): + continue + try: + r = json.loads(rj.read_text()) + except Exception: + continue + log = "" + lf = d / "run.log" + if lf.exists(): + try: + log = lf.read_text(errors="replace") + except Exception: + pass + turns = [int(m) for m in re.findall(r"Turn (\d+)/\d+", log)] + rows.append({ + "run": d.name, + "task": r.get("task"), + "class": classify(r, log), + "turns": max(turns) if turns else None, + "oracle_source": r.get("oracle_source"), + "oracle_disagreement": bool(r.get("oracle_disagreement")), + "oracle_observed": r.get("oracle_observed"), + }) + +by_task = {} +for row in rows: + t = by_task.setdefault(row["task"], {"runs": 0, "verified": 0, "classes": {}, "turns": []}) + t["runs"] += 1 + t["verified"] += row["class"] == "verified" + t["classes"][row["class"]] = t["classes"].get(row["class"], 0) + 1 + if row["turns"]: + t["turns"].append(row["turns"]) + +for t in by_task.values(): + t["median_turns"] = statistics.median(t["turns"]) if t["turns"] else None + t.pop("turns", None) + +classes = {} +for row in rows: + classes[row["class"]] = classes.get(row["class"], 0) + 1 + +report = { + "runs": len(rows), + "verified": sum(r["class"] == "verified" for r in rows), + "rate": round(sum(r["class"] == "verified" for r in rows) / max(1, len(rows)), 3), + "by_class": classes, + "by_task": by_task, + "oracle_disagreements": sum(r["oracle_disagreement"] for r in rows), + "rows": rows, +} +(OUT / "analysis.json").write_text(json.dumps(report, indent=2) + "\n") + +print(f"{'task':28} {'verified':>9} classes") +for name in sorted(by_task): + t = by_task[name] + cls = " ".join(f"{k}:{v}" for k, v in sorted(t["classes"].items())) + print(f"{name:28} {t['verified']}/{t['runs']:<7} {cls}") +print(f"\nTOTAL {report['verified']}/{report['runs']} = {report['rate']:.1%} oracle_disagreements={report['oracle_disagreements']}") +print("by_class:", classes) diff --git a/tests/integration/flash_live/run_batch.py b/tests/integration/flash_live/run_batch.py new file mode 100644 index 000000000..98b28fcd6 --- /dev/null +++ b/tests/integration/flash_live/run_batch.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""Batch driver: N executions of the complex-UI suite, serial on one emulator. + +Usage (from this directory, inside the project venv): + python spec.py # once per checkout — writes local-config.json + python run_batch.py --out /tmp/suite-out --runs-per-task 6 + +Resumable: skips dirs that already hold result.json. One hung run never +blocks the batch (hard per-run timeout = task timeout + 180s grace). +Seed-major ordering gives every task family coverage in the first pass. +Aggregates report.json incrementally after every run. +""" +import argparse +import json +import subprocess +import sys +import time +import urllib.request +from pathlib import Path + +from spec import ADB, API_BASE, ROOT, SERIAL +from suite_spec import TASKS + +PY = sys.executable +RUNNER = ROOT / "run_suite_task.py" +ENDPOINT = f"{API_BASE}/models" + +PARSER = argparse.ArgumentParser() +PARSER.add_argument("--out", type=Path, required=True) +PARSER.add_argument("--runs-per-task", type=int, default=7) +PARSER.add_argument("--tasks", nargs="*", default=None, help="subset of task names") +ARGS = PARSER.parse_args() + +tasks = ARGS.tasks or list(TASKS) +total = len(tasks) * ARGS.runs_per_task +out = ARGS.out +out.mkdir(parents=True, exist_ok=True) + + +def llm_alive(): + try: + with urllib.request.urlopen(ENDPOINT, timeout=5) as r: + return b"gemma" in r.read().lower() + except Exception: + return False + + +def adb_alive(): + try: + subprocess.run([ADB, "-s", SERIAL, "shell", "echo", "ok"], + capture_output=True, timeout=8, check=True) + return True + except Exception: + return False + + +def write_report(): + rows = [] + for d in sorted(out.iterdir()): + rj = d / "result.json" + if not rj.exists(): + continue + try: + r = json.loads(rj.read_text()) + except Exception: + continue + rows.append({ + "run": d.name, + "task": r.get("task"), + "verified_success": r.get("verified_success"), + "state_achieved": r.get("state_achieved"), + "reported_completed": r.get("reported_completed"), + "runner_status": (r.get("runner_result") or {}).get("status") if isinstance(r.get("runner_result"), dict) else None, + "error": r.get("error"), + "oracle_observed": r.get("oracle_observed"), + }) + by_task = {} + for row in rows: + t = by_task.setdefault(row["task"], {"runs": 0, "verified": 0, "achieved": 0, "completed": 0}) + t["runs"] += 1 + t["verified"] += 1 if row["verified_success"] else 0 + t["achieved"] += 1 if row["state_achieved"] else 0 + t["completed"] += 1 if row["reported_completed"] else 0 + report = { + "total_planned": total, + "runs_finished": len(rows), + "verified_success": sum(1 for r in rows if r["verified_success"]), + "by_task": by_task, + "rows": rows, + } + (out / "report.json").write_text(json.dumps(report, indent=2) + "\n") + return report + + +def wait_healthy(max_wait=300): + """Poll the endpoint+device for up to max_wait; True when both answer.""" + deadline = time.time() + max_wait + while time.time() < deadline: + if llm_alive() and adb_alive(): + return True + time.sleep(20) + return False + + +# Seed-major order: every task family gets one run before the second seed of +# anything starts, so a per-family harness problem surfaces in the first pass +# instead of after hours of one task type. +done = 0 +for seed_i in range(ARGS.runs_per_task): + for task_name in tasks: + run_dir = out / f"{task_name}__s{seed_i}" + if (run_dir / "result.json").exists(): + done += 1 + continue + if not wait_healthy(): + print(f"[{done}/{total}] SKIP {task_name} s{seed_i}: endpoint or device dead", flush=True) + run_dir.mkdir(parents=True, exist_ok=True) + (run_dir / "driver_error.marker").write_text("llm/adb unreachable >5min\n") + write_report() + continue + timeout = TASKS[task_name].get("timeout", 420) + 180 + print(f"[{done}/{total}] RUN {task_name} s{seed_i} (hard cap {timeout}s)", flush=True) + run_dir.mkdir(parents=True, exist_ok=True) + log = (run_dir / "run.log").open("w") + try: + subprocess.run( + [PY, str(RUNNER), "--task", task_name, "--output-dir", str(run_dir)], + timeout=timeout, stdout=log, stderr=subprocess.STDOUT, + ) + except subprocess.TimeoutExpired: + (run_dir / "timeout.marker").write_text(f"killed at {timeout}s\n") + except Exception as e: + (run_dir / "driver_error.marker").write_text(str(e) + "\n") + finally: + log.close() + done += 1 + rep = write_report() + v = rep["verified_success"] + print(f" -> {v}/{rep['runs_finished']} verified so far", flush=True) + time.sleep(4) + +final = write_report() +print(json.dumps({k: final[k] for k in ("runs_finished", "verified_success", "by_task")}, indent=2)) diff --git a/tests/integration/flash_live/run_suite_task.py b/tests/integration/flash_live/run_suite_task.py new file mode 100644 index 000000000..42f9e2f6e --- /dev/null +++ b/tests/integration/flash_live/run_suite_task.py @@ -0,0 +1,424 @@ +#!/usr/bin/env python3 +"""Suite variant of run_invariant_task.py — reads task defs from suite_spec.py. + +Generalized oracles (text_equals / text_present / element_attr / +element_attr_text / night_mode), per-task seed actions, intent variants +(action / action+data / -n component), and the same frozen protocol: +real local Gemma endpoint only, no cloud prewarm, deterministic checks. +""" +import argparse +import asyncio +import json +import os +import re +import subprocess +import time +import traceback +import xml.etree.ElementTree as ET +from pathlib import Path + +from spec import ADB, MODEL, ROOT, SERIAL +from suite_spec import TASKS + +PARSER = argparse.ArgumentParser(description=__doc__) +PARSER.add_argument("--task", choices=TASKS, required=True) +PARSER.add_argument("--output-dir", type=Path, required=True) +ARGS = PARSER.parse_args() +TASK = TASKS[ARGS.task] +assert ARGS.output_dir.parent.is_dir() +ARGS.output_dir.mkdir(exist_ok=True) +os.environ["ARTEMIS_ARTEMIS_JSONC"] = str(ROOT / "local-config.json") +os.environ["ARTEMIS_TRACES_DIR"] = str(ARGS.output_dir / "traces") +os.environ["ARTEMIS_HELPER_AUTO_INSTALL"] = "false" +os.environ["ARTEMIS_CHECKER_ENABLED"] = "false" +os.environ["ARTEMIS_OUTPUTTER_ENABLED"] = "false" +os.environ["ARTEMIS_PLANNER_VALIDATION_ENABLED"] = "false" +os.environ["LANGCHAIN_TRACING_V2"] = "false" +os.environ["HF_HUB_OFFLINE"] = "1" +os.environ["TRANSFORMERS_OFFLINE"] = "1" +if TASK.get("hierarchy_backend"): + os.environ["ARTEMIS_HIERARCHY_BACKEND"] = TASK["hierarchy_backend"] + +from artemis.agents.flash.runner import FlashRunner +from artemis.context import DevicePlatform +from artemis.sdk import Agent +import artemis.sdk.agent as sdk_agent +from third_party.mobile_use.main import new_default_config_builder + +EVENTS = ARGS.output_dir / "events.jsonl" + + +def event(kind, **values): + with EVENTS.open("a") as output: + output.write(json.dumps({"kind": kind, "monotonic": time.perf_counter(), **values}, default=str) + "\n") + + +def adb(*parts, timeout=40, check=True): + return subprocess.run( + [ADB, "-s", SERIAL, *parts], capture_output=True, check=check, timeout=timeout, + ).stdout + + +def launch(task=None): + task = task or TASK + pkg = task.get("package", "com.android.settings") + if task.get("intent") == "main_activity": + adb("shell", "am", "start", "-n", task["main_activity"], check=False) + elif task.get("intent_data"): + adb("shell", "am", "start", "-a", task["intent"], "-d", task["intent_data"], check=False) + else: + adb("shell", "am", "start", "-a", task["intent"], check=False) + + +def dump_xml(remote="/sdcard/suite_seed.xml"): + """Best-effort hierarchy dump; returns (nodes, element_dicts).""" + try: + adb("shell", "uiautomator", "dump", remote, timeout=25, check=False) + raw = adb("exec-out", "cat", remote, timeout=25, check=False) + except Exception: + return [], [] + if b"hierarchy" not in raw: + return [], [] + return list(ET.fromstring(raw).iter("node")), [] + + +def _center(node): + m = re.match(r"\[(\d+),(\d+)\]\[(\d+),(\d+)\]", node.get("bounds") or "") + if not m: + return None + l, t, r, b = (int(g) for g in m.groups()) + return (l + r) // 2, (t + b) // 2 + + +def run_seed(seed_spec): + stype = seed_spec["type"] + if stype == "uimode": + adb("shell", "cmd", "uimode", "night", seed_spec["value"]) + elif stype == "clear_app_data": + adb("shell", "pm", "clear", TASK.get("package", ""), check=False) + elif stype == "stopwatch_reset": + launch_task = {"intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "package": "com.google.android.deskclock"} + launch(launch_task) + time.sleep(2) + nodes, _ = dump_xml() + # Go to the Stopwatch tab — prefer the clickable tab node over the + # action-bar title that carries the same text. + tab = next( + (n for n in nodes + if ((n.get("text") or "") == "Stopwatch" or (n.get("content-desc") or "") == "Stopwatch") + and (n.get("clickable") or "") == "true"), + None, + ) or next( + (n for n in nodes + if (n.get("text") or "") == "Stopwatch" or (n.get("content-desc") or "") == "Stopwatch"), + None, + ) + if tab is not None: + c = _center(tab) + if c: + adb("shell", "input", "tap", str(c[0]), str(c[1]), check=False) + time.sleep(1) + nodes, _ = dump_xml() + for label in ("Stop", "Reset", "Pause"): + for n in nodes: + if (n.get("text") or "") == label or (n.get("content-desc") or "") == label: + c = _center(n) + if c: + adb("shell", "input", "tap", str(c[0]), str(c[1]), check=False) + time.sleep(0.5) + break + elif stype == "ensure_element_attr": + # Drive the target element's attribute to `expected` by tapping it + # when the current value differs. Deterministic seed, same dump + # mechanism the oracle uses. + launch({"intent": seed_spec.get("intent", TASK["intent"]), + "main_activity": seed_spec.get("main_activity") or TASK.get("main_activity", ""), + "package": TASK["package"], + "intent_data": seed_spec.get("intent_data")}) + time.sleep(2) + want = seed_spec["expected"] + for attempt in range(3): + nodes, _ = dump_xml() + hit = None + for n in nodes: + if seed_spec.get("resource_id") and n.get("resource-id") != seed_spec["resource_id"]: + continue + if seed_spec.get("text") and (n.get("text") or "") != seed_spec["text"] and (n.get("content-desc") or "") != seed_spec["text"]: + continue + hit = n + break + if hit is None and "deskclock" in TASK["package"] and seed_spec.get("text"): + # Alarm rows live on the Alarm tab — prefer the clickable + # tab node over the action-bar title carrying the same text. + tab = next( + (n for n in nodes + if ((n.get("text") or "") == "Alarm" or (n.get("content-desc") or "") == "Alarm") + and (n.get("clickable") or "") == "true"), + None, + ) or next( + (n for n in nodes + if (n.get("text") or "") == "Alarm" or (n.get("content-desc") or "") == "Alarm"), + None, + ) + if tab is not None: + c = _center(tab) + if c: + adb("shell", "input", "tap", str(c[0]), str(c[1]), check=False) + time.sleep(1) + continue + if hit is None: + event("seed_warn", seed=stype, detail="element not found") + return + if (hit.get(seed_spec["attribute"]) or "false") == want: + return + c = _center(hit) + if not c: + return + adb("shell", "input", "tap", str(c[0]), str(c[1]), check=False) + time.sleep(1) + event("seed_warn", seed=stype, detail="attribute did not reach expected") + + +# Inject the declared invariants into every ExecutionSetup the SDK builds. +_ORIG_ES = sdk_agent.ExecutionSetup +def _es_with_invariants(**kwargs): + inv = TASK.get("invariants") or [] + if inv: + kwargs["goal_invariants"] = inv + return _ORIG_ES(**kwargs) +sdk_agent.ExecutionSetup = _es_with_invariants + + +async def no_cloud_prewarm(self, api_key=None): + event("prewarm_skipped", reason="Cloud prewarm disabled; all inference uses the real local endpoint.") + + +Agent._prewarm_llm_connections = no_cloud_prewarm +ORIGINAL_RUN = FlashRunner.run +RUNNER_RESULT = None + + +async def measured_run(self, state): + global RUNNER_RESULT + adb("shell", "am", "force-stop", TASK.get("package", "com.android.settings")) + launch() + await asyncio.sleep(1) + event("ready", invariants=TASK.get("invariants") or []) + start = time.perf_counter() + RUNNER_RESULT = await ORIGINAL_RUN(self, state) + event("runner_result", duration_s=time.perf_counter() - start, result=RUNNER_RESULT) + return RUNNER_RESULT + + +FlashRunner.run = measured_run + + +def _fallback_elements(output_dir): + """Newest element list from the run's trace DB (ui_tree stores JSON dicts).""" + import sqlite3 + + db = output_dir / "traces" / "data_engine.db" + if not db.exists(): + return [] + rows = sqlite3.connect(db).execute( + "select ui_tree from images where ui_tree is not null and ui_tree != ''" + " order by timestamp desc limit 1" + ).fetchall() + if not rows: + return [] + try: + elements = json.loads(rows[0][0]) + except Exception: + return [] + return elements if isinstance(elements, list) else [] + + +def _norm_elements(xml_nodes, json_els): + """Yield (rid, class, text, desc, attrs) across XML and JSON-fallback sources.""" + for n in xml_nodes: + yield { + "rid": n.get("resource-id"), "class": n.get("class"), + "text": n.get("text"), "desc": n.get("content-desc"), + "attrs": dict(n.attrib), + } + for e in json_els: + yield { + "rid": e.get("resource-id"), "class": e.get("class"), + "text": e.get("text"), "desc": e.get("content-desc"), + "attrs": e, + } + + +def evaluate_oracle(oracle, xml_nodes, json_els, night_after): + ot = oracle["type"] + els = list(_norm_elements(xml_nodes, json_els)) + if ot == "night_mode": + return night_after == oracle["expected"], night_after + if ot == "system_setting": + # Device-level truth for settings whose widget is filtered out of the + # accessibility tree (e.g. Battery percentage: the Switch is + # clickable=false/focusable=false so neither backend's live tree + # surfaces it — only raw `uiautomator dump` does). The Settings.System + # value is synchronous and unambiguous. + val = adb( + "shell", "settings", "get", + oracle.get("namespace", "system"), oracle["key"], check=False, + ).decode(errors="replace").strip() + return val == str(oracle["expected"]), val + if ot == "text_equals": + hits = [e for e in els if e["rid"] == oracle["resource_id"]] + observed = hits[0]["text"] if hits else None + return bool(hits) and observed == oracle["expected"], observed + if ot == "text_present": + hits = [e for e in els if oracle["text"] in (e["text"] or "") or oracle["text"] in (e["desc"] or "")] + return bool(hits), len(hits) + if ot == "element_attr": + hits = [e for e in els if e["rid"] == oracle["resource_id"]] + observed = hits[0]["attrs"].get(oracle["attribute"]) if hits else None + return bool(hits) and str(observed) == oracle["expected"], observed + if ot == "element_attr_text": + hits = [e for e in els if (e["text"] or "") == oracle["text"] or (e["desc"] or "") == oracle["text"]] + observed = hits[0]["attrs"].get(oracle["attribute"]) if hits else None + return bool(hits) and str(observed) == oracle["expected"], observed + raise ValueError(f"unknown oracle type {ot}") + + +async def main(): + result = {"task": ARGS.task, "mode": "suite", "spec": TASK} + agent = None + try: + for seed_spec in TASK.get("seed") or []: + run_seed(seed_spec) + (ARGS.output_dir / "before.png").write_bytes(adb("exec-out", "screencap", "-p", check=False)) + builder = new_default_config_builder() + builder.for_device(DevicePlatform.ANDROID, SERIAL) + builder.with_video_recording_tools(enabled=False) + builder.with_checker(enabled=False) + builder.with_planner_validation(enabled=False) + builder.with_outputter(enabled=False) + builder.with_flash_step_summarizer(enabled=False) + agent = Agent(config=builder.build()) + await agent.init(retry_count=1, retry_wait_seconds=1) + returned = await asyncio.wait_for( + agent.run_task(goal=TASK["goal"], profile="flash", name=f"suite_{ARGS.task}"), + timeout=TASK.get("timeout", 420), + ) + result["sdk_result"] = returned + result["runner_result"] = RUNNER_RESULT + except BaseException as error: + result["error"] = f"{type(error).__name__}: {error}" + result["traceback"] = traceback.format_exc() + finally: + # Capture final UI state while the app is still alive: clean(force) + # force-stops the target, and DB-backed widgets (DeskClock alarms) + # flush their toggle write asynchronously — a post-clean dump can + # show reverted state the UI never displayed. + # uiautomator itself can return a stale attribute on a rebinding view + # (observed: teardown dump read checked=false while the same-moment + # screenshot and a later dump both showed the switch on), so the + # oracle votes over up to three dumps rather than trusting one. + remote = "/sdcard/artemis_suite_final.xml" + + def _dump_once(): + for _attempt in range(3): + try: + adb("shell", "uiautomator", "dump", remote, timeout=30, check=False) + data = adb("exec-out", "cat", remote, timeout=30, check=False) + if b"hierarchy" in data: + return data + except Exception as capture_err: + event("final_dump_retry", attempt=_attempt, error=str(capture_err)) + time.sleep(2) + return b"" + + def _oracle_on(xml_bytes): + if b"hierarchy" not in xml_bytes: + return evaluate_oracle( + TASK["oracle"], [], _fallback_elements(ARGS.output_dir), night + ) + return evaluate_oracle( + TASK["oracle"], + list(ET.fromstring(xml_bytes).iter("node")), + [], + night, + ) + + night = adb("shell", "cmd", "uimode", "night", check=False).decode(errors="replace").strip() + # Primary evidence: the newest live-run capture. The run's own + # observation path (UIAutomator2/helper, re-observed fresh before every + # invariant decision) proved fresher than a post-run shell `uiautomator + # dump`, which can serve stale node attributes — observed: two dumps + # read checked=false while the same-moment screenshot showed the + # switch on and the in-run captures had been stable true for ~2min. + trace_els = _fallback_elements(ARGS.output_dir) + trace_verdict = None + if trace_els: + try: + trace_verdict = evaluate_oracle(TASK["oracle"], [], trace_els, night) + except Exception as oracle_err: + result["oracle_error"] = str(oracle_err) + dumps = [_dump_once()] + verdicts = [] + try: + verdicts.append(_oracle_on(dumps[0])) + except Exception as oracle_err: + result["oracle_error"] = str(oracle_err) + while len(dumps) < 3 and ( + len(verdicts) < 2 or verdicts[-1][0] != verdicts[-2][0] + ): + time.sleep(2) + dumps.append(_dump_once()) + try: + verdicts.append(_oracle_on(dumps[-1])) + except Exception as oracle_err: + result["oracle_error"] = str(oracle_err) + break + if len(dumps) > 2: + result["oracle_unstable"] = True + event("oracle_unstable", verdicts=[str(v) for v in verdicts]) + xml = dumps[-1] + result["final_xml_source"] = "uiautomator" if b"hierarchy" in xml else "trace_db_fallback" + result["oracle_dump_votes"] = [bool(v[0]) for v in verdicts] + result["oracle_trace"] = ( + {"achieved": bool(trace_verdict[0]), "observed": str(trace_verdict[1])} + if trace_verdict is not None + else None + ) + if TASK["oracle"]["type"] == "system_setting": + ok, obs = evaluate_oracle(TASK["oracle"], [], [], night) + result["oracle_source"] = "system_setting" + result["state_achieved"] = bool(ok) + result["oracle_observed"] = obs + result["oracle_dump_votes"] = [bool(ok)] + elif trace_verdict is not None: + result["oracle_source"] = "trace_last" + result["state_achieved"] = bool(trace_verdict[0]) + result["oracle_observed"] = trace_verdict[1] + if verdicts and verdicts[-1][0] != trace_verdict[0]: + result["oracle_disagreement"] = True + elif verdicts: + result["oracle_source"] = "dump_vote" + result["state_achieved"] = sum(bool(v[0]) for v in verdicts) > len(verdicts) / 2 + for v in reversed(verdicts): + if bool(v[0]) == result["state_achieved"]: + result["oracle_observed"] = v[1] + break + else: + result["state_achieved"] = False + (ARGS.output_dir / "final.xml").write_bytes(xml) + (ARGS.output_dir / "after.png").write_bytes(adb("exec-out", "screencap", "-p", check=False)) + if agent is not None: + await agent.clean(force=True) + result["night_after"] = night + result["reported_completed"] = isinstance(RUNNER_RESULT, dict) and RUNNER_RESULT.get("status") == "completed" + result["verified_success"] = result["state_achieved"] and result["reported_completed"] + with (ARGS.output_dir / "result.json").open("x") as output: + json.dump(result, output, indent=2, default=str) + output.write("\n") + print(json.dumps(result, indent=2, default=str), flush=True) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/tests/integration/flash_live/spec.py b/tests/integration/flash_live/spec.py new file mode 100644 index 000000000..67d949e89 --- /dev/null +++ b/tests/integration/flash_live/spec.py @@ -0,0 +1,73 @@ +"""Live-eval configuration: on-device model, emulator, frozen protocol. + +Everything machine-specific reads from the environment so the suite runs +on any host with the same pieces: + + ARTEMIS_LIVE_ADB adb binary (default: platform-tools adb on PATH) + ARTEMIS_LIVE_SERIAL device serial (default: emulator-5554) + ARTEMIS_LIVE_MODEL served model id (default: local gemma-4-e4b 4-bit) + ARTEMIS_LIVE_API_BASE OpenAI-compatible endpoint (default: 127.0.0.1:8097) + +`prepare()` writes local-config.json next to this file; run it once per +checkout. No cloud inference: every node is pinned to the local endpoint. +""" +import json +import os +import shutil +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +ADB = os.environ.get( + "ARTEMIS_LIVE_ADB", + shutil.which("adb") + or "/Users/james/Library/Android/sdk/platform-tools/adb", +) +SERIAL = os.environ.get("ARTEMIS_LIVE_SERIAL", "emulator-5554") +MODEL = os.environ.get( + "ARTEMIS_LIVE_MODEL", + "/Users/james/.ondevice-agent-platform/models/" + "mlx-community--gemma-4-e4b-it-4bit__475b9088d29754a3379866cf5aeb6b41acd313c2", +) +API_BASE = os.environ.get("ARTEMIS_LIVE_API_BASE", "http://127.0.0.1:8097/v1") + +LLM = { + "provider": "custom", "model": MODEL, "api_base": API_BASE, + "temperature": 0.0, "max_tokens": 512, "timeout": 120, + "coordinate_format": "yx_1000", +} +NODES = ( + "planner", "summarizer", "operator", "operator_summarizer", + "log_reader_sub_agent", "log_analyzer", "diagnoser", "checker", + "planner_avatar", "history_analyzer_expert", "diagnoser_expert", "explorer", + "history_analyzer", "planner_validation", "validator_pixel_safety_net", "output_analyzer", +) + + +def prepare(): + llm = {**LLM, "fallback": dict(LLM)} + config = {name: dict(llm) for name in NODES} + config["utils"] = { + name: dict(llm) for name in ("outputter", "hopper", "video_analyzer", "object_detector") + } + config["agent"] = { + # 18 turns: a 7-digit dial needs 7 action turns + a report, plus + # recovery slack for index corrections and backspace repair on the + # dense pad (observed: two runs burned ~13 turns on mis-dials). + "flash": {"max_turns": 18, "step_summarizer": {"enabled": False}}, + "decision": {"enabled": False, "flash_loop": False}, + "pro": { + "checker": {"enabled": False}, + "planner_validation": {"enabled": False}, + "committee": {"enabled": False}, + }, + "video_analyzer": {"enabled": False, "enable_ledger": False}, + "outputter": {"enabled": False}, + "memory": {"chunking": {"enabled": False}}, + } + with (ROOT / "local-config.json").open("x") as output: + json.dump(config, output, indent=2) + output.write("\n") + + +if __name__ == "__main__": + prepare() diff --git a/tests/integration/flash_live/suite_spec.py b/tests/integration/flash_live/suite_spec.py new file mode 100644 index 000000000..93a3c3941 --- /dev/null +++ b/tests/integration/flash_live/suite_spec.py @@ -0,0 +1,299 @@ +"""Complex-UI suite: task definitions for the ~100-run Artemis Flash +on-device evaluation. Every task is verified by deterministic device +state (XML elements / element attributes / system queries) — never by +model self-report. + +Each task: + goal — natural-language instruction handed to the model + intent — `am start -a` action (or main_activity for pkg launch) + package — force-stopped before each run + oracle — {type, params}: how the final tree/system is checked + invariants — declared goal predicates (Flash gate) + seed — optional pre-run ADB state actions + timeout — wall clock for the run + hierarchy_backend — which a11y backend the run forces +""" + +TASKS = { + # ---- Digit entry on a 50-element lookalike pad ------------------------- + "dial_5550155": { + "goal": "Dial 5550155 using the on-screen dial pad.", + "intent": "android.intent.action.DIAL", + "package": "com.google.android.dialer", + "hierarchy_backend": "uiautomator", + "timeout": 900, + "invariants": [{ + "id": "dialed", + "selector": {"resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText"}, + "attribute": "text", "expected": "555-0155", + }], + "oracle": {"type": "text_equals", "resource_id": "com.google.android.dialer:id/digits", + "expected": "555-0155"}, + "seed": [{"type": "clear_app_data"}], + }, + "dial_8675309": { + "goal": "Dial 8675309 using the on-screen dial pad.", + "intent": "android.intent.action.DIAL", + "package": "com.google.android.dialer", + "hierarchy_backend": "uiautomator", + "timeout": 900, + "invariants": [{ + "id": "dialed", + "selector": {"resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText"}, + "attribute": "text", "expected": "867-5309", + }], + "oracle": {"type": "text_equals", "resource_id": "com.google.android.dialer:id/digits", + "expected": "867-5309"}, + "seed": [{"type": "clear_app_data"}], + }, + "dial_5551234": { + "goal": "Dial 5551234 using the on-screen dial pad.", + "intent": "android.intent.action.DIAL", + "package": "com.google.android.dialer", + "hierarchy_backend": "uiautomator", + "timeout": 900, + "invariants": [{ + "id": "dialed", + "selector": {"resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText"}, + "attribute": "text", "expected": "555-1234", + }], + "oracle": {"type": "text_equals", "resource_id": "com.google.android.dialer:id/digits", + "expected": "555-1234"}, + "seed": [{"type": "clear_app_data"}], + }, + "dial_911": { + # Short sequence: the model must stop at 3 digits, not over-dial. + "goal": "Dial 911 using the on-screen dial pad.", + "intent": "android.intent.action.DIAL", + "package": "com.google.android.dialer", + "hierarchy_backend": "uiautomator", + "timeout": 900, + "invariants": [{ + "id": "dialed", + "selector": {"resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText"}, + "attribute": "text", "expected": "911", + }], + "oracle": {"type": "text_equals", "resource_id": "com.google.android.dialer:id/digits", + "expected": "911"}, + "seed": [{"type": "clear_app_data"}], + }, + + # ---- Text input + dense suggestion list -------------------------------- + "sms_5550155": { + "hierarchy_backend": "uiautomator", + "goal": "Start a new text message conversation to the number 5550155.", + "intent": "android.intent.action.VIEW", + "intent_data": "sms:", + "package": "com.google.android.apps.messaging", + "timeout": 900, + "invariants": [{ + "id": "recipient", + "selector": {"class": "android.widget.TextView", + "text": "Texting with 555-0155 (SMS/MMS)"}, + "attribute": "text", "expected": "Texting with 555-0155 (SMS/MMS)", + }], + "oracle": {"type": "text_present", "text": "Texting with 555-0155"}, + "seed": [{"type": "clear_app_data"}], + }, + "sms_1234567": { + "hierarchy_backend": "uiautomator", + "goal": "Start a new text message conversation to the number 1234567.", + "intent": "android.intent.action.VIEW", + "intent_data": "sms:", + "package": "com.google.android.apps.messaging", + "timeout": 900, + "invariants": [{ + "id": "recipient", + "selector": {"class": "android.widget.TextView", + "text": "Texting with 123-4567 (SMS/MMS)"}, + "attribute": "text", "expected": "Texting with 123-4567 (SMS/MMS)", + }], + "oracle": {"type": "text_present", "text": "Texting with 123-4567"}, + "seed": [{"type": "clear_app_data"}], + }, + + # ---- Dense scrollable navigation --------------------------------------- + "nav_battery": { + "goal": "Open Battery settings.", + "intent": "android.settings.SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "battery_page", + "selector": {"resource_id": "android:id/title", + "class": "android.widget.TextView", + "text": "Battery usage"}, + "attribute": "text", "expected": "Battery usage", + }], + "oracle": {"type": "text_present", "text": "Battery usage"}, + "seed": [], + }, + "nav_sound": { + "goal": "Open Sound & vibration settings.", + "intent": "android.settings.SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "sound_page", + "selector": {"class": "android.widget.TextView", "text": "Media volume"}, + "attribute": "text", "expected": "Media volume", + }], + "oracle": {"type": "text_present", "text": "Media volume"}, + "seed": [], + }, + "nav_storage": { + "goal": "Open Storage settings.", + "intent": "android.settings.SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "storage_page", + "selector": {"class": "android.widget.TextView", "text": "Free up space"}, + "attribute": "text", "expected": "Free up space", + }], + "oracle": {"type": "text_present", "text": "Free up space"}, + "seed": [], + }, + "appinfo_chrome": { + # Settings → Apps → See all apps → Chrome (a long alphabetized list). + "goal": "Open the app info page for Chrome in Settings.", + "intent": "android.settings.SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "chrome_info", + "selector": {"class": "android.widget.Button", "text": "Force stop"}, + "attribute": "text", "expected": "Force stop", + }], + "oracle": {"type": "text_present", "text": "Force stop"}, + "seed": [], + }, + + # ---- State toggles on busy pages ---------------------------------------- + "battery_pct_on": { + "goal": "Turn on 'Battery percentage' so it shows in the status bar.", + "intent": "android.intent.action.POWER_USAGE_SUMMARY", + "package": "com.android.settings", + # The row's Switch is clickable=false/focusable=false (the row owns + # the tap) — the a11y helper drops it from the tree entirely, so the + # invariant and oracle can never see it there (verified live). + "hierarchy_backend": "uiautomator", + "timeout": 900, + "invariants": [{ + "id": "battery_pct_switch", + "selector": {"resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch"}, + "attribute": "checked", "expected": True, + }], + # The Switch node is invisible to every live backend (see above) — + # verify the Settings.System value it writes instead. + "oracle": {"type": "system_setting", "namespace": "system", + "key": "status_bar_show_battery_percent", "expected": "1"}, + "seed": [{"type": "ensure_element_attr", + "intent": "android.intent.action.POWER_USAGE_SUMMARY", + "resource_id": "com.android.settings:id/switchWidget", + "attribute": "checked", "expected": "false"}], + }, + "dark_off": { + "goal": "Turn off Dark theme.", + "intent": "android.settings.DISPLAY_SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "dark_theme_off", + "selector": {"resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch"}, + "attribute": "checked", "expected": False, + }], + "oracle": {"type": "night_mode", "expected": "Night mode: no"}, + "seed": [{"type": "uimode", "value": "yes"}], + }, + "dark_on": { + "goal": "Turn on Dark theme.", + "intent": "android.settings.DISPLAY_SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "dark_theme_on", + "selector": {"resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch"}, + "attribute": "checked", "expected": True, + }], + "oracle": {"type": "night_mode", "expected": "Night mode: yes"}, + "seed": [{"type": "uimode", "value": "no"}], + }, + + # ---- Clock: existing alarms + hard creation ----------------------------- + "alarm_830_on": { + "hierarchy_backend": "uiautomator", + "goal": "Turn on the 8:30 AM alarm in the Clock app.", + "intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "package": "com.google.android.deskclock", + "timeout": 900, + "invariants": [{ + "id": "alarm_830_switch", + "selector": {"class": "android.widget.Switch", "text": "8:30 AM alarm"}, + "attribute": "checked", "expected": True, + }], + "oracle": {"type": "element_attr_text", "text": "8:30 AM alarm", + "attribute": "checked", "expected": "true"}, + "seed": [{"type": "ensure_element_attr", + "intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "text": "8:30 AM alarm", "attribute": "checked", "expected": "false"}], + }, + "alarm_900_off": { + "hierarchy_backend": "uiautomator", + "goal": "Turn off the 9:00 AM alarm in the Clock app.", + "intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "package": "com.google.android.deskclock", + "timeout": 900, + "invariants": [{ + "id": "alarm_900_switch", + "selector": {"class": "android.widget.Switch", "text": "9:00 AM alarm"}, + "attribute": "checked", "expected": False, + }], + "oracle": {"type": "element_attr_text", "text": "9:00 AM alarm", + "attribute": "checked", "expected": "false"}, + "seed": [{"type": "ensure_element_attr", + "intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "text": "9:00 AM alarm", "attribute": "checked", "expected": "true"}], + }, + "stopwatch_start": { + "hierarchy_backend": "uiautomator", + "goal": "Start the stopwatch in the Clock app.", + "intent": "main_activity", + "main_activity": "com.google.android.deskclock/com.android.deskclock.DeskClock", + "package": "com.google.android.deskclock", + "timeout": 900, + "invariants": [{ + "id": "stopwatch_running", + "selector": {"class": "android.widget.Button", "text": "Stop"}, + "attribute": "text", "expected": "Stop", + }], + "oracle": {"type": "text_present", "text": "Stop"}, + "seed": [{"type": "stopwatch_reset"}], + }, + + # ---- Search-driven navigation ------------------------------------------- + "settings_search_wallpaper": { + "goal": "Use the Settings search bar to find and open 'Wallpaper & style' settings.", + "intent": "android.settings.SETTINGS", + "package": "com.android.settings", + "timeout": 900, + "invariants": [{ + "id": "wallpaper_page", + "selector": {"class": "android.widget.TextView", "text": "Clock color & size"}, + "attribute": "text", "expected": "Clock color & size", + }], + "oracle": {"type": "text_present", "text": "Clock color & size"}, + "seed": [], + }, +} diff --git a/tests/unit/admin_console/test_local_model_endpoints.py b/tests/unit/admin_console/test_local_model_endpoints.py new file mode 100644 index 000000000..90675ac0d --- /dev/null +++ b/tests/unit/admin_console/test_local_model_endpoints.py @@ -0,0 +1,233 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for the local-model lifecycle endpoints. + +These guard the click-to-run console path: only resolvable model refs may +spawn ``artemis model`` subprocesses, ``stop`` stays port-scoped, and task +state/logs surface for polling. +""" + +import asyncio + +import pytest +from httpx import ASGITransport, AsyncClient + +from apps.admin_console.routers import system +from apps.admin_console.server import app + + +@pytest.fixture(autouse=True) +def _clear_model_tasks(): + system._MODEL_TASKS.clear() + yield + system._MODEL_TASKS.clear() + + +class _FakeStdout: + def __init__(self, lines): + self._lines = lines + + def __aiter__(self): + return self._iter() + + async def _iter(self): + for line in self._lines: + yield line + + +class _FakeProc: + def __init__(self, rc=0, lines=(b"done\n",)): + self.stdout = _FakeStdout(lines) + self._rc = rc + + async def wait(self): + return self._rc + + +def _patch_subprocess(monkeypatch, captured: list, rc=0, lines=(b"done\n",)): + async def _fake_exec(*cmd, **kwargs): + captured.append(list(cmd)) + return _FakeProc(rc=rc, lines=lines) + + monkeypatch.setattr(asyncio, "create_subprocess_exec", _fake_exec) + + +def _client(**kwargs): + return AsyncClient(transport=ASGITransport(app=app, **kwargs), base_url="http://localhost") + + +@pytest.mark.asyncio +async def test_status_reports_cli_unavailable_without_model_module(): + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://localhost") as ac: + res = await ac.get("/api/system/local-models/status") + assert res.status_code == 200 + data = res.json() + # The `artemis model` CLI ships with the local-models feature; on builds + # without it the console must fall back to copy-to-terminal guidance. + assert data["commands_available"] == system._model_cli_available() + assert data["tasks"] == {} + + +@pytest.mark.asyncio +async def test_pull_returns_501_when_cli_missing(): + async with _client() as ac: + res = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + if system._model_cli_available(): + pytest.skip("model CLI present in this build") + assert res.status_code == 501 + assert "terminal" in res.json()["detail"] + + +@pytest.mark.asyncio +async def test_pull_rejects_unresolvable_alias(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + async with _client() as ac: + for bad in ("not-a-model", "--port", "gemma4-e4b; rm -rf /"): + res = await ac.post("/api/system/local-models/pull", json={"alias": bad}) + assert res.status_code == 400, bad + assert system._MODEL_TASKS == {} + + +@pytest.mark.asyncio +async def test_pull_rejects_remote_client(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + async with _client(client=("192.168.1.20", 42000)) as ac: + res = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + assert res.status_code == 403 + assert "local-only" in res.json()["detail"] + + +@pytest.mark.asyncio +async def test_pull_starts_task_and_completes(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + captured = [] + _patch_subprocess(monkeypatch, captured, rc=0, lines=(b"downloading\n", b"done\n")) + + async with _client() as ac: + res = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + assert res.status_code == 200 + body = res.json() + assert body["status"] == "running" + assert body["key"] == "pull:gemma4-e4b" + + assert captured[0][-3:] == ["model", "pull", "gemma4-e4b"] + + await system._MODEL_TASKS["pull:gemma4-e4b"]["watcher"] + task = system._MODEL_TASKS["pull:gemma4-e4b"] + assert task["status"] == "done" + assert task["returncode"] == 0 + assert "downloading" in "\n".join(task["log"]) + + +@pytest.mark.asyncio +async def test_pull_is_idempotent_while_running(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + started = asyncio.Event() + release = asyncio.Event() + + class _HangingProc(_FakeProc): + async def wait(self): + release.set() + return 0 + + async def _fake_exec(*cmd, **kwargs): + proc = _HangingProc() + started.set() + return proc + + monkeypatch.setattr(asyncio, "create_subprocess_exec", _fake_exec) + + async with _client() as ac: + res1 = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + res2 = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + assert res1.json()["status"] == res2.json()["status"] == "running" + + +@pytest.mark.asyncio +async def test_serve_passes_port_and_serve_cmd(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + captured = [] + _patch_subprocess(monkeypatch, captured) + + async with _client() as ac: + res = await ac.post( + "/api/system/local-models/serve", json={"alias": "gemma4-e4b", "port": 8097} + ) + assert res.status_code == 200 + assert captured[0][-5:] == ["model", "serve", "gemma4-e4b", "--port", "8097"] + + +@pytest.mark.asyncio +async def test_stop_is_port_scoped_without_positional_alias(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + captured = [] + _patch_subprocess(monkeypatch, captured) + + async with _client() as ac: + res = await ac.post( + "/api/system/local-models/stop", json={"alias": "gemma4-e4b", "port": 8080} + ) + assert res.status_code == 200 + cmd = captured[0] + # CLI: `artemis model stop --port N` — the alias must not appear as argv. + assert cmd[-3:] == ["model", "stop", "--port"] or cmd[-4:] == [ + "model", + "stop", + "--port", + "8080", + ] + assert "gemma4-e4b" not in cmd + + +@pytest.mark.asyncio +async def test_failed_process_surfaces_log_and_returncode(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + _patch_subprocess(monkeypatch, [], rc=2, lines=(b"error: no space left\n",)) + + async with _client() as ac: + res = await ac.post("/api/system/local-models/pull", json={"alias": "gemma4-e4b"}) + assert res.status_code == 200 + + await system._MODEL_TASKS["pull:gemma4-e4b"]["watcher"] + async with _client() as ac: + status = (await ac.get("/api/system/local-models/status")).json() + task = status["tasks"]["pull:gemma4-e4b"] + assert task["status"] == "failed" + assert task["returncode"] == 2 + assert any("no space left" in line for line in task["log"]) + + +@pytest.mark.asyncio +async def test_org_repo_alias_is_accepted(monkeypatch): + monkeypatch.setattr(system, "_model_cli_available", lambda: True) + captured = [] + _patch_subprocess(monkeypatch, captured) + + async with _client() as ac: + res = await ac.post( + "/api/system/local-models/pull", json={"alias": "mlx-community/gemma-4-e4b-it-4bit"} + ) + assert res.status_code == 200 + assert captured[0][-1] == "mlx-community/gemma-4-e4b-it-4bit" + + +@pytest.mark.asyncio +async def test_invalid_port_rejected(): + async with _client() as ac: + res = await ac.post( + "/api/system/local-models/serve", json={"alias": "gemma4-e4b", "port": 70000} + ) + assert res.status_code == 422 diff --git a/tests/unit/admin_console/test_model_service.py b/tests/unit/admin_console/test_model_service.py index a06346931..40f6952c7 100644 --- a/tests/unit/admin_console/test_model_service.py +++ b/tests/unit/admin_console/test_model_service.py @@ -25,7 +25,7 @@ def test_get_active_model_info_pro_architecture(): info = ModelService.get_active_model_info("pro") assert info["name"] == "Pro" assert info["architecture"] == "ARTEMIS Pro" - assert info["provider"] == "google" + assert info["provider"] in ("google", "custom") assert "id" in info @@ -34,7 +34,7 @@ def test_get_active_model_info_flash_architecture(): info = ModelService.get_active_model_info("flash") assert info["name"] == "Flash" assert info["architecture"] == "ARTEMIS Flash" - assert info["provider"] == "google" + assert info["provider"] in ("google", "custom") def test_resolve_session_profile_from_device_info(): diff --git a/tests/unit/agents/flash/test_hybrid_loop.py b/tests/unit/agents/flash/test_hybrid_loop.py new file mode 100644 index 000000000..54aa91091 --- /dev/null +++ b/tests/unit/agents/flash/test_hybrid_loop.py @@ -0,0 +1,528 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Hybrid Flash loop (decision.flash_loop): VLM describes, decision model acts. + +The describe call binds no tools, so malformed tool-call envelopes cannot be +emitted; the decision model picks from a fixed menu (element taps + +meta-actions). Any inconclusive verdict — unreachable endpoint, low +confidence, or a key outside the menu — escalates to the standard +tool-calling model turn. +""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pytest +from langchain_core.messages import AIMessage, HumanMessage + +from artemis.agents.flash.runner import FlashRunner, _HYBRID_META_ACTIONS +from artemis.core.decision_client import DecisionResult +from artemis.llm.reliability import Failure, FailureCategory, LLMCallError + + +_FAILURE = Failure(FailureCategory.UNKNOWN, retryable=True, should_fallback=True) + + +def _runner(goal: str = "enable dark theme") -> FlashRunner: + r = FlashRunner.__new__(FlashRunner) + r.goal = goal + r._last_action_note = None + return r + + +def _cfg(threshold: float = 0.85): + return SimpleNamespace( + url="http://127.0.0.1:11434/v1/decide", + model="laya", + timeout_seconds=5.0, + confidence_threshold=threshold, + ) + + +def _state(elements): + return SimpleNamespace(indexed_elements=elements) + + +def _verdict(verdict, confidence=0.9): + return DecisionResult(verdict=verdict, confidence=confidence) + + +# --- action menu -------------------------------------------------------------- + + +def test_menu_lists_element_taps_and_meta_actions(): + state = _state( + [ + {"index": 1, "text": "Settings"}, + {"index": 2, "content_desc": "Search"}, + {"index": 3, "resource_id": "id/more"}, + ] + ) + menu = FlashRunner._hybrid_action_menu(state) + assert menu["tap_1"] == "tap element 1: Settings" + assert menu["tap_2"] == "tap element 2: Search" + assert menu["tap_3"] == "tap element 3: id/more" + for key in _HYBRID_META_ACTIONS: + assert key in menu + + +def test_menu_handles_missing_elements(): + assert "done" in FlashRunner._hybrid_action_menu(_state(None)) + assert "tap_" not in " ".join(FlashRunner._hybrid_action_menu(_state(None))) + + +# --- menu -> tool call mapping ------------------------------------------------- + + +def test_choice_maps_element_tap_to_click(): + call = FlashRunner._menu_choice_to_call("tap_4", "desc") + assert call["name"] == "click" + assert call["args"] == {"target": 4} + assert call["id"] + + +def test_choice_maps_swipe_and_keys(): + assert FlashRunner._menu_choice_to_call("swipe_up", "")["args"] == { + "direction": "up" + } + back = FlashRunner._menu_choice_to_call("press_back", "") + assert back["name"] == "press_key" and back["args"] == {"key": "back"} + + +def test_choice_maps_done_and_fail_to_report(): + done = FlashRunner._menu_choice_to_call("done", "all set") + assert done["name"] == "report_task_status" + assert done["args"]["status"] == "completed" + assert done["args"]["explanation"] == "all set" + fail = FlashRunner._menu_choice_to_call("fail", "blocked") + assert fail["args"]["status"] == "failed" + + +def test_choice_rejects_unknown_key(): + assert FlashRunner._menu_choice_to_call("launch_rocket", "") is None + + +# --- executed call -> menu key mapping ------------------------------------------ + + +def test_menu_key_for_call_maps_device_actions(): + f = FlashRunner._menu_key_for_call + assert f("click", {"target": 7, "target_description": "x"}) == "tap_7" + assert f("swipe", {"direction": "up"}) == "swipe_up" + assert f("swipe", {"gesture": "left"}) == "swipe_left" + assert f("press_key", {"key": "BACK"}) == "press_back" + assert f("press_key", {"key": "home"}) == "press_home" + assert f("wait_for_delay", {"time_in_ms": 500}) == "wait" + assert f("report_task_status", {"status": "completed"}) == "done" + assert f("report_task_status", {"status": "failed"}) == "fail" + + +def test_menu_key_for_call_rejects_non_menu_calls(): + f = FlashRunner._menu_key_for_call + assert f("click", {"target": "not-an-int"}) is None + assert f("swipe", {"start": [0, 0], "end": [1, 1]}) is None + assert f("input_text", {"text": "hi"}) is None + + +# --- _hybrid_decide ------------------------------------------------------------ + + +@pytest.mark.asyncio +async def test_hybrid_decide_returns_call_on_confident_verdict(): + runner = _runner() + state = _state([{"index": 1, "text": "Dark theme"}]) + describe_resp = AIMessage(content="Settings list; [1] Display ~[500,300]") + + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=describe_resp), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock( + return_value=_verdict("tap_1") + ) + out = await runner._hybrid_decide( + AsyncMock(), state, HumanMessage(content="obs"), "plan", _cfg() + ) + + assert out is not None + response, tool_calls = out + assert tool_calls[0]["name"] == "click" + assert tool_calls[0]["args"] == {"target": 1} + # The choice is stamped onto the describe message so ToolMessages keep a + # legitimate parent when the ledger replays them. + assert response.tool_calls[0]["name"] == "click" + # The describe response is pure text — no envelope parsing needed. + assert "Settings list" in response.content + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_on_low_confidence(): + runner = _runner() + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock( + return_value=_verdict("tap_1", confidence=0.10) + ) + out = await runner._hybrid_decide( + AsyncMock(), _state([{"index": 1, "text": "Dark theme"}]), HumanMessage(content="x"), None, _cfg() + ) + assert out is None + + +@pytest.mark.asyncio +async def test_hybrid_decide_gates_on_answer_confidence_not_trust_score(): + """laya-serve returns near-zero `confidence` on menu-shaped spreads; + the actionable signal is `answer_confidence` (top-pick probability).""" + runner = _runner() + verdict = DecisionResult( + verdict="tap_1", + confidence=0.008, # calibrated trust score — near zero + raw={"answer_confidence": 0.62}, + ) + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock(return_value=verdict) + out = await runner._hybrid_decide( + AsyncMock(), _state([{"index": 1, "text": "Dark theme"}]), HumanMessage(content="x"), None, _cfg() + ) + assert out is not None + assert out[1][0]["name"] == "click" + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_when_unanswered(): + runner = _runner() + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock( + return_value=DecisionResult(verdict=None) + ) + out = await runner._hybrid_decide( + AsyncMock(), _state([]), HumanMessage(content="x"), None, _cfg() + ) + assert out is None + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_on_out_of_menu_verdict(): + runner = _runner() + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock( + return_value=_verdict("tap_99", confidence=0.95) + ) + out = await runner._hybrid_decide( + AsyncMock(), _state([]), HumanMessage(content="x"), None, _cfg() + ) + assert out is None + + +@pytest.mark.asyncio +async def test_hybrid_decide_masks_tried_menu_keys(): + """Elements already tapped are removed from the menu by (index, label) + identity, so the decision model cannot re-pick a dead element.""" + runner = _runner(goal="tap Alpha Beta") + state = _state([{"index": 1, "text": "Alpha"}, {"index": 2, "text": "Beta"}]) + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + choose = AsyncMock(return_value=_verdict("tap_2")) + MockClient.return_value.achoose = choose + out = await runner._hybrid_decide( + AsyncMock(), + state, + HumanMessage(content="x"), + None, + _cfg(), + tried_taps={(1, "Alpha")}, + tried_labels=set(), + ) + assert out is not None + menu_sent = choose.call_args.args[2] + assert "tap_1" not in menu_sent + assert "tap_2" in menu_sent + # Terminal meta-actions are never masked — wait/done/fail stay available. + assert "wait" in menu_sent + assert "done" in menu_sent + # The decision state discloses what was masked so the pick has context. + state_sent = choose.call_args.args[0] + assert "tap_1" in state_sent + + +@pytest.mark.asyncio +async def test_hybrid_decide_masks_reindexed_label(): + """A distinctive label stays masked even when the element list re-indexes + (the volatile-text problem that defeats screen signatures).""" + runner = _runner(goal="open Settings Beta") + # Same "Settings" row, now at index 9 instead of 12. + state = _state([{"index": 9, "text": "Settings"}, {"index": 2, "text": "Beta"}]) + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + choose = AsyncMock(return_value=_verdict("tap_2")) + MockClient.return_value.achoose = choose + out = await runner._hybrid_decide( + AsyncMock(), + state, + HumanMessage(content="x"), + None, + _cfg(), + tried_taps={(12, "Settings")}, + tried_labels={"Settings"}, + ) + assert out is not None + menu_sent = choose.call_args.args[2] + assert "tap_9" not in menu_sent + assert "tap_2" in menu_sent + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_when_menu_lacks_goal_target(): + """No element label sharing a goal token → between-screen navigation is + the VLM path's job; the decision model only picks on-screen elements.""" + runner = _runner() # goal: "enable dark theme" + choose = AsyncMock() + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="Gmail setup page")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = choose + out = await runner._hybrid_decide( + AsyncMock(), + _state([{"index": 1, "text": "GOT IT"}, {"index": 2, "text": "Gmail"}]), + HumanMessage(content="x"), + None, + _cfg(), + ) + assert out is None + choose.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_hybrid_decide_proceeds_when_goal_target_present(): + runner = _runner() + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="Display screen")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + choose = AsyncMock(return_value=_verdict("tap_2")) + MockClient.return_value.achoose = choose + out = await runner._hybrid_decide( + AsyncMock(), + _state( + [{"index": 1, "text": "Brightness"}, {"index": 2, "text": "Dark theme"}] + ), + HumanMessage(content="x"), + None, + _cfg(), + ) + assert out is not None + choose.assert_awaited() + + +def test_menu_has_goal_target_matching(): + f = FlashRunner._menu_has_goal_target + menu = {"tap_1": "tap element 1: Gmail", "tap_2": "tap element 2: Dark theme"} + assert f(menu, "enable dark theme") is True + assert f(menu, "open the Settings app") is False + assert f({"swipe_up": "scroll"}, "enable dark theme") is False + # goals with no matchable tokens never gate (all short/stopword tokens) + assert f(menu, "do it") is True + + +@pytest.mark.asyncio +async def test_hybrid_decide_never_executes_terminal_picks(): + """done/fail escalate to the VLM turn — the decision model sees no pixels + and cannot honestly verify goal completion (observed live declaring + 'done' on a home screen where the target app wasn't open).""" + runner = _runner() + for pick in ("done", "fail"): + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + MockClient.return_value.achoose = AsyncMock( + return_value=_verdict(pick, confidence=0.95) + ) + out = await runner._hybrid_decide( + AsyncMock(), _state([{"index": 1, "text": "Dark theme"}]), HumanMessage(content="x"), + None, _cfg(), + ) + assert out is None, pick + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_on_describe_failure(): + runner = _runner() + with patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(side_effect=LLMCallError("server down", failure=_FAILURE)), + ): + out = await runner._hybrid_decide( + AsyncMock(), _state([]), HumanMessage(content="x"), None, _cfg() + ) + assert out is None + + +# --- _hybrid_plan -------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_hybrid_plan_returns_milestone_text(): + runner = _runner() + with patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="- open Settings\n- tap Display")), + ): + plan = await runner._hybrid_plan(AsyncMock()) + assert "open Settings" in plan + + +@pytest.mark.asyncio +async def test_hybrid_plan_failure_is_non_fatal(): + runner = _runner() + with patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(side_effect=LLMCallError("boom", failure=_FAILURE)), + ): + assert await runner._hybrid_plan(AsyncMock()) is None + + +@pytest.mark.asyncio +async def test_hybrid_decide_state_includes_last_action_outcome(): + """The decision model has no transcript view — the one-line last-action + note is its only signal that a tap was a dead end vs. real progress.""" + runner = _runner() + runner._last_action_note = "click → no visible change" + state = _state([{"index": 1, "text": "Dark theme"}]) + captured = {} + + with ( + patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=AIMessage(content="desc")), + ), + patch("artemis.agents.flash.runner.DecisionClient") as MockClient, + ): + async def _achoose(state_text, question, menu): + captured["state"] = state_text + return _verdict("tap_1") + + MockClient.return_value.achoose = _achoose + await runner._hybrid_decide( + AsyncMock(), state, HumanMessage(content="x"), "plan", _cfg() + ) + + assert "click → no visible change" in captured["state"] + + +@pytest.mark.asyncio +async def test_hybrid_decide_escalates_when_describe_says_target_reached(): + """'TARGET: reached' is a perception claim, not a menu pick — the turn + escalates so the VLM can verify and report (the terminal gate audits it).""" + runner = _runner() + state = _state([{"index": 1, "text": "Dark theme"}]) + describe = AIMessage(content="Dark theme is already on.\nTARGET: reached") + + with patch( + "artemis.agents.flash.runner.acomplete", + new=AsyncMock(return_value=describe), + ), patch("artemis.agents.flash.runner.DecisionClient") as MockClient: + out = await runner._hybrid_decide( + AsyncMock(), state, HumanMessage(content="x"), "plan", _cfg() + ) + + assert out is None + MockClient.return_value.achoose.assert_not_called() + + +# --- hybrid prompt rendering --------------------------------------------------- + + +@pytest.mark.asyncio +async def test_hybrid_plan_prompt_renders_goal(): + """The plan prompt is rendered with jinja2, so `{{ goal }}` must be + substituted — a `$goal` placeholder would reach the VLM literally.""" + runner = _runner(goal="enable dark theme") + complete = AsyncMock(return_value=AIMessage(content="- open Settings")) + with patch("artemis.agents.flash.runner.acomplete", new=complete): + await runner._hybrid_plan(AsyncMock()) + + prompt = complete.call_args.args[1][0].content[0]["text"] + assert "GOAL: enable dark theme" in prompt + assert "$goal" not in prompt + + +@pytest.mark.asyncio +async def test_hybrid_describe_prompt_renders_goal_and_plan(): + """The describe prompt must carry both the goal and the plan text — + rendered with jinja2, `$goal`/`$plan` would pass through literally.""" + runner = _runner(goal="enable dark theme") + describe = AIMessage(content="Settings list.\nTARGET: reached") + complete = AsyncMock(return_value=describe) + with patch("artemis.agents.flash.runner.acomplete", new=complete): + await runner._hybrid_decide( + AsyncMock(), + _state([{"index": 1, "text": "Dark theme"}]), + HumanMessage(content="x"), + "- open Settings\n- tap Display", + _cfg(), + ) + + prompt = complete.call_args.args[1][0].content + assert "GOAL: enable dark theme" in prompt + assert "- open Settings\n- tap Display" in prompt + assert "$goal" not in prompt + assert "$plan" not in prompt diff --git a/tests/unit/agents/flash/test_state_predicates.py b/tests/unit/agents/flash/test_state_predicates.py new file mode 100644 index 000000000..ec2abba3c --- /dev/null +++ b/tests/unit/agents/flash/test_state_predicates.py @@ -0,0 +1,430 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic goal-invariant predicates (artemis.agents.flash.state_predicates). + +Verdicts are tri-state: ``unknown`` covers absence, ambiguity and missing +state — never silently treated as failure or success. +""" + +from types import SimpleNamespace + +import pytest + +from artemis.agents.flash.runner import FlashRunner +from artemis.agents.flash.state_predicates import ( + describe_failures, + evaluate_invariants, + evaluate_predicate, +) +from artemis.mcp.observation import _control_only_elements + +_SWITCH = { + "index": 14, + "text": "Dark theme", + "class": "android.widget.Switch", + "resource_id": "com.android.settings:id/switchWidget", + "bounds": [834, 692, 961, 744], + "center": [897, 718], + "checked": "true", + "enabled": "true", + "is_ocr": False, +} + +_PRED_OFF = { + "id": "dark_theme_off", + "selector": { + "resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch", + "text": "Dark theme", + }, + "attribute": "checked", + "expected": False, +} + + +# --- single predicate ---------------------------------------------------------- + + +def test_predicate_unsatisfied_when_state_differs(): + r = evaluate_predicate(_PRED_OFF, [_SWITCH]) + assert r["status"] == "unsatisfied" + assert r["evidence"]["observed"] is True + + +def test_predicate_satisfied_when_state_matches(): + el = {**_SWITCH, "checked": "false"} + r = evaluate_predicate(_PRED_OFF, [el]) + assert r["status"] == "satisfied" + + +def test_predicate_unknown_when_target_absent(): + r = evaluate_predicate(_PRED_OFF, [{"text": "unrelated"}]) + assert r["status"] == "unknown" + assert r["reason"] == "target_absent" + + +def test_predicate_unknown_when_ambiguous(): + loose = {**_PRED_OFF, "selector": {"text": "Dark theme"}} + r = evaluate_predicate(loose, [_SWITCH, dict(_SWITCH)]) + assert r["status"] == "unknown" + assert r["reason"] == "ambiguous_target" + + +def test_predicate_unknown_when_attribute_missing(): + el = {k: v for k, v in _SWITCH.items() if k != "checked"} + r = evaluate_predicate(_PRED_OFF, [el]) + assert r["status"] == "unknown" + assert r["reason"] == "attribute_not_known" + + +# --- text attribute ------------------------------------------------------------ + +_DIGITS = { + "index": 3, + "text": "555-0155", + "class": "android.widget.EditText", + "resource_id": "com.google.android.dialer:id/digits", + "bounds": [147, 1196, 933, 1340], + "center": [540, 1268], + "enabled": "true", + "is_ocr": False, +} + +_PRED_DIGITS = { + "id": "dialed_5550155", + "selector": { + "resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText", + }, + "attribute": "text", + "expected": "555-0155", +} + + +def test_text_predicate_satisfied_on_exact_match(): + r = evaluate_predicate(_PRED_DIGITS, [_DIGITS]) + assert r["status"] == "satisfied" + assert r["evidence"]["observed"] == "555-0155" + + +def test_text_predicate_unsatisfied_on_partial_input(): + r = evaluate_predicate(_PRED_DIGITS, [{**_DIGITS, "text": "555-0"}]) + assert r["status"] == "unsatisfied" + + +def test_text_predicate_unknown_when_text_absent(): + r = evaluate_predicate(_PRED_DIGITS, [{k: v for k, v in _DIGITS.items() if k != "text"}]) + assert r["status"] == "unknown" + assert r["reason"] == "attribute_not_known" + + +def test_text_predicate_rejects_bool_expected(): + bad = {**_PRED_DIGITS, "expected": True} + r = evaluate_predicate(bad, [_DIGITS]) + assert r["status"] == "unknown" + assert r["reason"] == "unsupported_predicate" + + +def test_bool_predicate_rejects_string_expected(): + bad = {**_PRED_OFF, "expected": "false"} + r = evaluate_predicate(bad, [_SWITCH]) + assert r["status"] == "unknown" + assert r["reason"] == "unsupported_predicate" + + +def test_predicate_ignores_ocr_elements(): + ocr = {**_SWITCH, "is_ocr": True, "checked": "false"} + r = evaluate_predicate(_PRED_OFF, [ocr]) + assert r["status"] == "unknown" + assert r["reason"] == "target_absent" + + +@pytest.mark.parametrize( + "bad", + [ + {}, + {"selector": {}}, + {**_PRED_OFF, "selector": {"bounds": "[0,0][1,1]"}}, + {**_PRED_OFF, "attribute": "content-desc"}, + {**_PRED_OFF, "expected": "off"}, + "not-a-dict", + ], +) +def test_predicate_unknown_when_spec_unsupported(bad): + assert evaluate_predicate(bad, [_SWITCH])["status"] == "unknown" + + +# --- invariant aggregation ----------------------------------------------------- + + +def test_invariants_satisfied_only_when_all_hold(): + on_pred = {**_PRED_OFF, "id": "on", "expected": True} + report = evaluate_invariants([_PRED_OFF, on_pred], [_SWITCH]) + assert report["verdict"] == "unsatisfied" # _PRED_OFF fails + report = evaluate_invariants([on_pred], [_SWITCH]) + assert report["verdict"] == "satisfied" + + +def test_invariants_unknown_when_evidence_missing(): + report = evaluate_invariants([_PRED_OFF], []) + assert report["verdict"] == "unknown" + + +def test_invariants_empty_is_unknown_never_auto_satisfied(): + assert evaluate_invariants([], [_SWITCH])["verdict"] == "unknown" + + +def test_describe_failures_names_expected_and_observed(): + report = evaluate_invariants([_PRED_OFF], [_SWITCH]) + text = describe_failures(report) + assert "dark_theme_off" in text + assert "expected checked=False" in text + + +# --- element signature sees state ---------------------------------------------- + + +def _sig(elements): + from artemis.agents.flash.runner import FlashRunner + + return FlashRunner._element_signature(elements) + + +def test_signature_dense_screen_small_change_counts(): + # 30-node dial pad: appending one digit changes exactly one label — + # Jaccard 0.8 called this 'unchanged'; strict equality must not. + pad = [ + {"index": i, "text": str(i % 10), "center": [100 * (i % 3) + 50, 1400 + 190 * (i // 3)], "package": "com.google.android.dialer"} + for i in range(29) + ] + pad.append({**_DIGITS, "package": "com.google.android.dialer"}) + before = _sig(pad) + after = _sig([*pad[:-1], {**_DIGITS, "text": "555-015", "package": "com.google.android.dialer"}]) + assert before != after + + +def test_signature_excludes_status_bar_and_ocr_jitter(): + app = [{"index": 1, "text": "five", "center": [540, 1649], "package": "com.google.android.dialer"}] + noise_a = [ + *app, + {"index": 2, "text": "19:08", "center": [60, 40], "package": "com.android.systemui"}, + {"index": 3, "text": "OCR read A", "center": [500, 200], "is_ocr": True}, + ] + noise_b = [ + *app, + {"index": 2, "text": "19:09", "center": [60, 40], "package": "com.android.systemui"}, + {"index": 3, "text": "OCR read B", "center": [500, 200], "is_ocr": True}, + ] + assert _sig(noise_a) == _sig(noise_b) + + +def test_signature_distinguishes_toggle_states(): + on = [_SWITCH] + off = [{**_SWITCH, "checked": "false"}] + assert not FlashRunner._signatures_similar( + FlashRunner._element_signature(on), + FlashRunner._element_signature(off), + ) + + +def test_signature_ignores_volatile_focus(): + a = [dict(_SWITCH)] + b = [{**_SWITCH, "focused": "true"}] + assert FlashRunner._signatures_similar( + FlashRunner._element_signature(a), + FlashRunner._element_signature(b), + ) + + +# --- control-only elements (unlabeled switches etc.) ---------------------------- + +# Raw fused nodes use the uiautomator shape (hyphenated keys, string attrs). +_RAW_SWITCH = { + "text": "", + "resource-id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch", + "checkable": "true", + "checked": "false", + "enabled": "true", + "clickable": "true", + "bounds": "[901,1661][1038,1787]", +} +_RAW_TITLE = { + "text": "Dark theme", + "resource-id": "android:id/title", + "class": "android.widget.TextView", + "checkable": "false", + "checked": "false", + "bounds": "[63,1663][333,1734]", +} + + +def test_control_elements_expose_unlabeled_switch_state(): + labeled = [ + { + "index": 1, + "text": "Dark theme", + "class": "android.widget.TextView", + "resource_id": "android:id/title", + "bounds": [63, 1663, 333, 1734], + "is_ocr": False, + } + ] + controls = _control_only_elements([_RAW_TITLE, _RAW_SWITCH], labeled) + assert len(controls) == 1 + el = controls[0] + assert el["is_control_only"] and not el["is_ocr"] + assert el["index"] == len(labeled) + 1 # appended after the labeled sequence + assert el["checked"] == "false" + # The predicate now sees the bare switch the compact list never showed + # (unlabeled controls match on resource_id + class — there is no text). + bare_switch_pred = { + **_PRED_OFF, + "selector": { + "resource_id": "com.android.settings:id/switchWidget", + "class": "android.widget.Switch", + }, + } + r = evaluate_predicate(bare_switch_pred, labeled + controls) + assert r["status"] == "satisfied" + + +def test_control_elements_skip_textviews_and_duplicates(): + labeled = [ + { + "index": 1, + "text": "x", + "class": "android.widget.Switch", + "resource_id": "com.android.settings:id/switchWidget", + "bounds": [901, 1661, 1038, 1787], + "is_ocr": False, + } + ] + controls = _control_only_elements([_RAW_TITLE, _RAW_SWITCH], labeled) + assert controls == [] # title isn't stateful; switch already represented + + +# --- completion gate ------------------------------------------------------------ + + +def _runner_with_invariants(invariants, final_check=False): + r = FlashRunner.__new__(FlashRunner) + r._goal_invariants = invariants + r._plan_text = None + r.ctx = SimpleNamespace( + execution_setup=SimpleNamespace( + goal_invariants=invariants, + final_check_enabled=final_check, + ) + ) + return r + + +@pytest.mark.asyncio +async def test_verify_completion_rejects_unsatisfied_invariant(): + r = _runner_with_invariants([_PRED_OFF]) + state = SimpleNamespace(indexed_elements=[_SWITCH]) # still ON + allowed, findings = await r._verify_completion(state) + assert not allowed + assert "dark_theme_off" in findings + + +@pytest.mark.asyncio +async def test_verify_completion_releases_on_satisfied_invariants(): + r = _runner_with_invariants([_PRED_OFF]) + state = SimpleNamespace(indexed_elements=[{**_SWITCH, "checked": "false"}]) + allowed, _ = await r._verify_completion(state) + assert allowed + + +@pytest.mark.asyncio +async def test_verify_completion_falls_through_on_unknown(): + # No invariants armed, no checker → existing fail-open behavior. + r = _runner_with_invariants([], final_check=False) + state = SimpleNamespace(indexed_elements=[]) + allowed, _ = await r._verify_completion(state) + assert allowed + + +# --- pixel-level stale-tree oracle -------------------------------------------- + + +def _png_bytes(width=64, height=64, box=None): + import io + + from PIL import Image, ImageDraw + + img = Image.new("RGB", (width, height), "black") + if box: + ImageDraw.Draw(img).rectangle(box, fill="white") + buf = io.BytesIO() + img.save(buf, format="PNG") + return buf.getvalue() + + +def test_screens_differ_flags_a_small_real_change(): + from artemis.agents.flash.runner import FlashRunner + + before = _png_bytes() + after = _png_bytes(box=(10, 10, 24, 24)) # one appended digit's worth + assert FlashRunner._screens_differ(before, after) + + +def test_screens_differ_stays_quiet_on_identical_frames(): + from artemis.agents.flash.runner import FlashRunner + + a = _png_bytes(box=(5, 5, 30, 30)) + b = _png_bytes(box=(5, 5, 30, 30)) + assert not FlashRunner._screens_differ(a, b) + + +def test_screens_differ_refuses_missing_or_garbage_bytes(): + from artemis.agents.flash.runner import FlashRunner + + assert not FlashRunner._screens_differ(None, b"x") + assert not FlashRunner._screens_differ(b"x", None) + assert not FlashRunner._screens_differ(b"not a png", b"also not") + + +def test_describe_failures_locates_the_digit_divergence(): + """A text invariant on a formatted field: the finding names the shared + digit prefix and both remainders — '(555) 001-0555' vs '555-0155' shares + '5550' then diverges.""" + from artemis.agents.flash.state_predicates import describe_failures, evaluate_invariants + + pred = { + "id": "dialed", + "selector": {"resource_id": "com.google.android.dialer:id/digits"}, + "attribute": "text", + "expected": "555-0155", + } + el = { + "index": 1, "text": "(555) 001-0555", + "resource_id": "com.google.android.dialer:id/digits", + "class": "android.widget.EditText", "is_ocr": False, + } + report = evaluate_invariants([pred], [el]) + text = describe_failures(report) + assert "digits 1-4 '5550' match" in text + assert "expected continues '155'" in text + assert "observed continues '010555'" in text + + +def test_digit_prefix_diff_handles_formatting_noise(): + from artemis.agents.flash.state_predicates import _digit_prefix_diff + + assert "digits 1-4" in _digit_prefix_diff("555-0155", "(555) 001-0555") + assert _digit_prefix_diff("555-0155", "555-0155") is None + assert _digit_prefix_diff("", "123") is None diff --git a/tests/unit/agents/test_explorer_tiers.py b/tests/unit/agents/test_explorer_tiers.py index e8c400d3d..c66e4ce39 100644 --- a/tests/unit/agents/test_explorer_tiers.py +++ b/tests/unit/agents/test_explorer_tiers.py @@ -20,10 +20,12 @@ from google.genai import types from langchain_core.messages import AIMessage, HumanMessage, ToolMessage +from PIL import Image import pytest from artemis.agents.explorer.explorer import Explorer from artemis.agents.explorer.tiers import EXPLORER_TIERS, PERCEPTION_TOOLS, get_tier +from artemis.agents.explorer.universal_runner import UniversalRunnerMixin from artemis.config import settings from artemis.context import ArtemisContext from artemis.graph.state import State @@ -244,11 +246,11 @@ async def test_universal_dispatch_appends_image_message(tmp_path): assert [type(m) for m in messages] == [ToolMessage, ToolMessage, HumanMessage] assert messages[0].content == "perception text" human = messages[2] - assert human.content[0] == { + assert human.content[-1] == { "type": "text", "text": "[Annotated image(s) returned by: ask_perception_tool, inspect_region]", } - image_blocks = human.content[1:] + image_blocks = human.content[:-1] assert len(image_blocks) == 2 # the missing file is skipped assert all(b["type"] == "image_url" for b in image_blocks) assert image_blocks[0]["image_url"]["url"].startswith("data:image/jpeg;base64,") @@ -555,3 +557,21 @@ def test_get_tier_normalizes_or_falls_back(version): normalized = str(version or "").strip().lower() expected = normalized if normalized in EXPLORER_TIERS else "flash" assert tier.name == expected + + +def test_universal_initial_messages_single_image_before_text(tmp_path): + """Initial universal prompt carries the screenshot once, before the request text.""" + path = tmp_path / "screen.png" + Image.new("RGB", (8, 8), (255, 255, 255)).save(path) + messages = UniversalRunnerMixin()._build_universal_messages( + query="target", + context_feedback="feedback", + screenshot_path=str(path), + minimal_list="[1] item", + prompt_template="system", + ) + content = messages[1].content + assert [block["type"] for block in content] == ["image_url", "text"] + assert content[0]["image_url"]["url"].startswith("data:image/png;base64,") + assert "Query: target" in content[1]["text"] + assert "[1] item" in content[1]["text"] diff --git a/tests/unit/agents/test_flash_runner.py b/tests/unit/agents/test_flash_runner.py index f188b5e1b..2633a50de 100644 --- a/tests/unit/agents/test_flash_runner.py +++ b/tests/unit/agents/test_flash_runner.py @@ -556,3 +556,64 @@ async def test_final_report_persists_native_thinking(mock_context): assert kwargs["operator_raw_thinking"] == "final text" assert kwargs["operator_native_thinking"] == "native summary" assert isinstance(messages[-1], ToolMessage) + + +# --- Gemma local-server tool-call envelope repair -------------------------------- + + +def _make_runner(mock_context): + with patch("artemis.controllers.unified_controller.get_driver"): + return FlashRunner(mock_context, goal="test goal") + + +def test_resolve_tool_calls_keeps_native_calls(mock_context): + runner = _make_runner(mock_context) + native = [{"name": "click", "args": {"target": 1}, "id": "n1"}] + resp = SimpleNamespace(tool_calls=native, invalid_tool_calls=[]) + assert runner._resolve_tool_calls(resp, "") == native + + +def test_resolve_tool_calls_repairs_enveloped_invalid_call(mock_context): + """mlx_vlm wraps `call:name{args}` as name="call" + unparseable arguments.""" + runner = _make_runner(mock_context) + resp = SimpleNamespace( + tool_calls=[], + invalid_tool_calls=[ + { + "name": "call", + "args": '{"report_task_status{explanation": "Settings is open."}', + "id": "broken-1", + "error": "json error", + } + ], + ) + calls = runner._resolve_tool_calls(resp, "") + assert calls == [ + { + "name": "report_task_status", + "args": {"explanation": "Settings is open."}, + "id": "broken-1", + } + ] + + +def test_resolve_tool_calls_repairs_call_envelope_in_text(mock_context): + """The `call:` envelope can leak into content instead of tool_calls.""" + runner = _make_runner(mock_context) + resp = SimpleNamespace(tool_calls=[], invalid_tool_calls=[]) + text = 'Opening settings. call:click{target: 34, target_description: "Display & touch"}' + calls = runner._resolve_tool_calls(resp, text) + assert calls[0]["name"] == "click" + assert calls[0]["args"] == {"target": 34, "target_description": "Display & touch"} + + +def test_resolve_tool_calls_unrecoverable_envelope_returns_empty(mock_context): + """Genuinely garbled args still surface as no-call (notice path), not a crash.""" + runner = _make_runner(mock_context) + resp = SimpleNamespace( + tool_calls=[], + invalid_tool_calls=[ + {"name": "call", "args": '{"click{target": "6,broken', "id": "b2", "error": "x"} + ], + ) + assert runner._resolve_tool_calls(resp, "") == [] diff --git a/tests/unit/agents/test_flash_runner_ledger.py b/tests/unit/agents/test_flash_runner_ledger.py index 57ff1ef87..edf1accde 100644 --- a/tests/unit/agents/test_flash_runner_ledger.py +++ b/tests/unit/agents/test_flash_runner_ledger.py @@ -192,9 +192,13 @@ def test_observation_tail_has_pro_shape(mock_context): tail = runner._build_tail(ledger, 1, b"IMG", "[1] Settings") texts = [b["text"] for b in tail.content if b["type"] == "text"] assert OBSERVATION_HEADER_RE.match(texts[0]) - assert texts[1] == "--- Current Screenshot ---" - assert texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] Settings" + assert texts[1] == f"{PRO_UI_LIST_MARKER}\n[1] Settings" + assert texts[2] == "--- Current Screenshot ---" assert len(texts) == 3 + # The screenshot is the final block: intermediate-image pruning then + # removes a trailing span and the surviving text prefix stays + # byte-identical across turns for prefix-cache reuse. + assert tail.content[-1]["type"] == "image_url" assert sum(1 for b in tail.content if b["type"] == "image_url") == 1 joined = "".join(texts) assert "Open Settings" not in joined and "objective" not in joined.lower() @@ -242,16 +246,18 @@ def test_per_turn_notices_are_marked_ephemeral(mock_context): previous_turn_silent=True, ) texts = [b.get("text") for b in tail.content] - # header, screenshot label, image, UI list, then the per-turn notices with - # the persistent instruction body right after its wrapper - assert texts[4:] == [ + # header, UI list, then the per-turn notices with the persistent + # instruction body right after its wrapper; screenshot label + image close + # the message (image last so pruning removes a trailing span). + assert texts[2:7] == [ "nudge", guidance.wrapper, guidance.body, _FINAL_TURN_WARNING, REASONING_REMINDER, ] - assert tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [4, 5, 7, 8] + assert texts[7] == "--- Current Screenshot ---" and texts[8] is None + assert tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [2, 3, 5, 6] def test_injected_instruction_body_outlives_its_wrapper(mock_context): @@ -472,8 +478,8 @@ async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context): images = [b for b in tail2.content if b["type"] == "image_url"] assert len(images) == 1 assert images[0]["image_url"]["url"].endswith(base64.b64encode(b"IMG-tc2").decode()) - assert tail2_texts[1] == "--- Current Screenshot ---" - assert tail2_texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] next" + assert tail2_texts[1] == f"{PRO_UI_LIST_MARKER}\n[1] next" + assert tail2_texts[2] == "--- Current Screenshot ---" assert len(turn2_messages) == 6 # Both actions of the multi-action turn were registered on the committed turn. @@ -684,3 +690,195 @@ def test_turn_record_reports_the_first_failure_of_a_multi_action_turn(): ] ) assert record.result()["error"] == "click rejected: coordinates out of bounds" + + +# --------------------------------------------------------------------------- +# Terminal-boundary completion verification (Pro final check at report time) +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_rejected_completion_feeds_verdicts_back_and_the_run_continues(mock_context): + """A 'completed' report the final check rejects must not end the run: + the tool call is answered with a rejection carrying the failed verdicts, + and the next turn still runs — here the agent then fails honestly.""" + responses = [_report(explanation="dark theme on"), _report(status="failed", explanation="cannot")] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + runner._verify_completion = AsyncMock( + return_value=(False, "[verify failed] 'dark theme enabled': ui_night_mode=1") + ) + result = await runner.run(State(initial_goal="Enable dark theme")) + + assert result == {"status": "failed", "explanation": "cannot"} + assert runner._completion_rejections == 1 + # The rejected tool call got an answering ToolMessage with the verdicts. + turn2_messages = runner.calls[1]["messages"] + rejection = next( + m for m in turn2_messages if isinstance(m, ToolMessage) and "rejected" in str(m.content) + ) + assert "ui_night_mode=1" in rejection.content + assert rejection.status == "error" + + +@pytest.mark.asyncio +async def test_completion_blocked_outcome_when_verify_budget_is_exhausted(mock_context): + """Once the retry budget is spent the report is finalized as failed — + narration cannot outvote the checker's ground truth forever.""" + responses = [_report(explanation="dark theme on")] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + runner._completion_rejections = 3 # == default final_check_max_attempts + runner._verify_completion = AsyncMock(return_value=(False, "goal unmet")) + result = await runner.run(State(initial_goal="Enable dark theme")) + + assert result["status"] == "failed" + assert "blocked" in result["explanation"].lower() + assert "goal unmet" in result["explanation"] + + +@pytest.mark.asyncio +async def test_failed_status_skips_completion_verification(mock_context): + """Only 'completed' claims are audited; reporting failure is honest already.""" + responses = [_report(status="failed", explanation="device offline")] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + runner._verify_completion = AsyncMock(return_value=(True, "")) + result = await runner.run(State(initial_goal="Enable dark theme")) + + assert result["status"] == "failed" + runner._verify_completion.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_verbatim_dead_call_set_is_suppressed_not_reexecuted(mock_context): + """An identical call set that already changed nothing must not be + executed a second time — the tool call is answered with an error and a + navigation hint instead of burning a real tap.""" + dead_call = { + "name": "click", + "args": {"target": [897, 390], "target_description": "Dark theme"}, + "id": "tc-a", + } + repeat_call = dict(dead_call, id="tc-b") + responses = [ + AIMessage(content="tap", tool_calls=[dead_call]), + AIMessage(content="tap again", tool_calls=[repeat_call]), + _report(status="failed", explanation="gave up"), + ] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + result = await runner.run(State(initial_goal="Enable dark theme")) + + assert result["status"] == "failed" + # Only ONE real click executed; the verbatim repeat was suppressed. + executed = [c for c in runner.executor.execute.await_args_list] + assert len(executed) == 1 + # The suppressed call got an error ToolMessage in the next turn's prompt. + turn3_messages = runner.calls[2]["messages"] + err = next( + m + for m in turn3_messages + if isinstance(m, ToolMessage) and "was already executed" in str(m.content) + ) + assert err.status == "error" + + +@pytest.mark.asyncio +async def test_ping_pong_cycle_triggers_alternation_notice(mock_context): + """click → BACK → same click is an A,B,A cycle: screens change every + turn so verbatim suppression cannot see it, but the turn-3 re-issue of + the turn-1 call on a matching screen must raise the alternation + notice.""" + home = [{"index": 1, "text": "home", "center": (100, 100)}] + wifi = [{"index": 1, "text": "wifi page", "center": (200, 200)}] + + def _exec(name, args, tc_id, state, **_): + if state is not None: + state.indexed_elements = list( + wifi if name == "click" else home + ) + return _exec_result(tc_id, name, f"IMG-{tc_id}".encode()) + + enter = {"name": "click", "args": {"target": 11}, "id": "c1"} + back = {"name": "press_key", "args": {"key": "BACK"}, "id": "b1"} + reenter = {"name": "click", "args": {"target": 11}, "id": "c2"} + responses = [ + AIMessage(content="enter", tool_calls=[enter]), + AIMessage(content="back", tool_calls=[back]), + AIMessage(content="enter again", tool_calls=[reenter]), + _report(status="failed", explanation="loop"), + ] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + runner.executor.execute = AsyncMock(side_effect=_exec) + state = State(initial_goal="Check Wi-Fi state") + state.indexed_elements = list(home) + await runner.run(state) + + turn4_messages = runner.calls[3]["messages"] + assert any( + "alternating between two screens" in str(m.content) + for m in turn4_messages + ) + + +@pytest.mark.asyncio +async def test_repeated_suppressed_calls_fail_after_streak(mock_context): + """Suppression prevents re-execution, not repetition — a model that + re-issues the same dead call a third time is stuck; end honestly + instead of burning the turn budget.""" + def _call(i): + return { + "name": "click", + "args": {"target": 12}, + "id": f"tc-{i}", + } + responses = [ + AIMessage(content="t", tool_calls=[_call(i)]) for i in range(5) + ] + with ( + patch("artemis.controllers.unified_controller.get_driver"), + patch( + "artemis.agents.flash.runner.capture_screenshot_and_parse_ui", + AsyncMock(return_value=_INITIAL_OBSERVATION), + ), + ): + runner = _make_runner(mock_context, responses, max_turns=0) + result = await runner.run(State(initial_goal="Toggle a switch")) + + assert result["status"] == "failed" + assert "suppressed" in result["explanation"] + # First call executed once; calls 2 and 3 were suppressed; streak 3 + # ended the run before calls 4-5 were ever consumed. + assert runner.executor.execute.await_count == 1 diff --git a/tests/unit/agents/test_object_detector_coordinates.py b/tests/unit/agents/test_object_detector_coordinates.py new file mode 100644 index 000000000..6033db2f6 --- /dev/null +++ b/tests/unit/agents/test_object_detector_coordinates.py @@ -0,0 +1,298 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Coordinate normalization for multi-provider object detection. + +The detector contract is ``[x, y]`` on a 0-1000 grid over the sent image. +Each endpoint declares the convention its model actually emits via +``coordinate_format``; clearly off-contract batches are rescaled. +""" + +from types import SimpleNamespace + +import pytest + +from artemis.agents.object_detector.object_detector import ( + _detected_batch_scale, + _endpoint_coordinate_format, + _image_dimensions, + _normalize_detected_box, + _normalize_detected_point, + _qwen_resized_dimensions, +) + + +def _item(a, b): + return {"point": [a, b], "label": "x"} + + +# --------------------------------------------------------------------------- +# Declared formats +# --------------------------------------------------------------------------- + + +def test_yx_1000_swaps_axes_and_keeps_scale(): + item = _item(800, 200) + _normalize_detected_point(item, "yx_1000", (1000, 2000), None) + assert item["point"] == [200.0, 800.0] + + +def test_xy_1000_keeps_axes_and_scale(): + item = _item(200, 800) + _normalize_detected_point(item, "xy_1000", (1000, 2000), None) + assert item["point"] == [200.0, 800.0] + + +def test_xy_norm_scales_zero_to_one(): + item = _item(0.2, 0.8) + _normalize_detected_point(item, "xy_norm", (1000, 2000), None) + assert item["point"] == [200.0, 800.0] + + +def test_yx_norm_swaps_and_scales(): + item = _item(0.8, 0.2) + _normalize_detected_point(item, "yx_norm", (1000, 2000), None) + assert item["point"] == [200.0, 800.0] + + +def test_xy_px_scales_by_qwen_resize_grid(): + # 1000x2000 snaps to 992x1984 under the factor-32 smart resize. + item = _item(496, 992) # center of the resized grid + _normalize_detected_point(item, "xy_px", (1000, 2000), None) + assert item["point"] == [500.0, 500.0] + + +def test_yx_px_swaps_axes_and_scales(): + item = _item(992, 496) + _normalize_detected_point(item, "yx_px", (1000, 2000), None) + assert item["point"] == [500.0, 500.0] + + +def test_px_without_image_size_leaves_point_unchanged(): + item = _item(1500, 700) + _normalize_detected_point(item, "xy_px", None, None) + assert item["point"] == [1500.0, 700.0] + + +# --------------------------------------------------------------------------- +# Batch-level scale detection +# --------------------------------------------------------------------------- + + +def test_batch_scale_detects_norm_when_all_below_one(): + items = [_item(0.5, 0.2), _item(0.9, 0.1)] + assert _detected_batch_scale(items) == "norm" + + +def test_batch_scale_detects_px_when_any_above_1000(): + items = [_item(500, 200), _item(1500, 300)] + assert _detected_batch_scale(items) == "px" + + +def test_batch_scale_mixed_grid_stays_undeclared(): + # A legit 0-1000 corner point coexisting with mid-grid values must not + # trip the norm heuristic — the batch still reads as the declared grid. + items = [_item(0.5, 0.3), _item(500, 400)] + assert _detected_batch_scale(items) is None + + +def test_batch_scale_all_zero_is_undeclared(): + assert _detected_batch_scale([_item(0, 0)]) is None + + +def test_batch_scale_scans_box_2d_coords(): + boxes = [ + {"box_2d": [0.1, 0.2, 0.3, 0.4], "label": "a"}, + {"box_2d": [0.5, 0.6, 0.7, 0.8], "label": "b"}, + ] + assert _detected_batch_scale(boxes) == "norm" + assert _detected_batch_scale([{"box_2d": [100, 200, 300, 1400]}]) == "px" + + +# --------------------------------------------------------------------------- +# box_2d detection output (Gemma 4 native [ymin, xmin, ymax, xmax] / 1000) +# --------------------------------------------------------------------------- + + +def test_box_2d_yx_1000_derives_center_point(): + # Gemma-style [ymin, xmin, ymax, xmax] -> contract [x1, y1, x2, y2]. + item = {"box_2d": [200, 100, 400, 300], "label": "icon"} + assert _normalize_detected_box(item, "yx_1000", (1000, 2000), None) is True + assert item["box_2d"] == [100.0, 200.0, 300.0, 400.0] + assert item["point"] == [200.0, 300.0] + + +def test_box_2d_xy_1000_keeps_axes(): + item = {"box_2d": [100, 200, 300, 400]} + assert _normalize_detected_box(item, "xy_1000", (1000, 2000), None) is True + assert item["box_2d"] == [100.0, 200.0, 300.0, 400.0] + assert item["point"] == [200.0, 300.0] + + +def test_box_2d_norm_scale_scales_and_swaps(): + item = {"box_2d": [0.2, 0.1, 0.4, 0.3]} + assert _normalize_detected_box(item, "yx_norm", (1000, 2000), None) is True + assert item["box_2d"] == [100.0, 200.0, 300.0, 400.0] + assert item["point"] == [200.0, 300.0] + + +def test_box_2d_keeps_valid_point(): + item = {"point": [800, 200], "box_2d": [200, 100, 400, 300]} + assert _normalize_detected_box(item, "yx_1000", (1000, 2000), None) is False + _normalize_detected_point(item, "yx_1000", (1000, 2000), None) + assert item["point"] == [200.0, 800.0] + assert item["box_2d"] == [100.0, 200.0, 300.0, 400.0] + + +def test_box_2d_replaces_unusable_point(): + item = {"point": ["a", "b"], "box_2d": [200, 100, 400, 300]} + assert _normalize_detected_box(item, "yx_1000", (1000, 2000), None) is True + assert item["point"] == [200.0, 300.0] + + +def test_invalid_box_2d_is_untouched(): + for bad in ({"box_2d": [1, 2]}, {"box_2d": "x"}, {"box_2d": [1, 2, "c", 4]}): + item = dict(bad) + assert _normalize_detected_box(item, "yx_1000", (100, 100), None) is False + assert item == bad + + +def test_batch_scale_override_fixes_norm_batch_under_1000_contract(): + item = _item(0.5, 0.2) + _normalize_detected_point(item, "yx_1000", (1000, 2000), "norm") + # yx order + norm scale: x=0.2*1000, y=0.5*1000 + assert item["point"] == [200.0, 500.0] + + +def test_batch_scale_override_fixes_px_batch_under_1000_contract(): + item = _item(992, 496) # [y_px, x_px] on a 992x1984 grid + _normalize_detected_point(item, "yx_1000", (1000, 2000), "px") + assert item["point"] == [500.0, 500.0] + + +# --------------------------------------------------------------------------- +# Endpoint format resolution + image dims +# --------------------------------------------------------------------------- + + +def test_endpoint_coordinate_format_defaults_to_gemini(): + assert _endpoint_coordinate_format(SimpleNamespace(endpoint=None)) == "yx_1000" + wrapper = SimpleNamespace(endpoint=SimpleNamespace(coordinate_format="xy_px")) + assert _endpoint_coordinate_format(wrapper) == "xy_px" + bogus = SimpleNamespace(endpoint=SimpleNamespace(coordinate_format="bogus")) + assert _endpoint_coordinate_format(bogus) == "yx_1000" + + +def test_qwen_resize_snaps_to_factor_32(): + assert _qwen_resized_dimensions(1000, 2000) == (992, 1984) + assert _qwen_resized_dimensions(2940, 1912) == (2944, 1920) + + +def test_image_dimensions_reads_png_header(): + import io + + from PIL import Image + + buf = io.BytesIO() + Image.new("RGB", (640, 480)).save(buf, format="PNG") + assert _image_dimensions(buf.getvalue()) == (640, 480) + assert _image_dimensions(b"not an image") is None + + +def test_invalid_points_are_untouched(): + for bad in ({}, {"point": "x"}, {"point": [1]}, {"point": ["a", "b"]}): + item = dict(bad) + _normalize_detected_point(item, "yx_1000", (100, 100), None) + assert item == bad + + +@pytest.mark.asyncio +async def test_run_object_detection_uses_endpoint_coordinate_format(): + """End-to-end: an xy_px endpoint turns pixel output into the 0-1000 contract.""" + import io + from unittest.mock import AsyncMock, patch + + from PIL import Image + + from artemis.agents.object_detector.object_detector import _run_object_detection + + buf = io.BytesIO() + Image.new("RGB", (1000, 2000)).save(buf, format="PNG") + image_bytes = buf.getvalue() + + fake_llm = SimpleNamespace(endpoint=SimpleNamespace(coordinate_format="xy_px")) + # Model reports grid-center in pixels of the 992x1984 resized image. + vlm_output = [{"point": [496, 992], "label": "home button"}] + + with ( + patch( + "artemis.agents.object_detector.object_detector.get_llm", + return_value=fake_llm, + ), + patch( + "artemis.agents.object_detector.object_detector._detect_single_label", + new_callable=AsyncMock, + return_value=vlm_output, + ), + ): + ctx = SimpleNamespace(llm_config=SimpleNamespace()) + result = await _run_object_detection( + ctx=ctx, + image_bytes=image_bytes, + queries=["home button"], + ) + + point = result["detected"][0]["point"] + # 496px/992w -> 500x, 992px/1984h -> 500y on the 0-1000 grid. + assert point == [pytest.approx(500.0), pytest.approx(500.0)] + + +@pytest.mark.asyncio +async def test_run_object_detection_accepts_box_2d_output(): + """End-to-end: a box-only response (Gemma-style) still yields a contract point.""" + import io + from unittest.mock import AsyncMock, patch + + from PIL import Image + + from artemis.agents.object_detector.object_detector import _run_object_detection + + buf = io.BytesIO() + Image.new("RGB", (1000, 2000)).save(buf, format="PNG") + image_bytes = buf.getvalue() + + fake_llm = SimpleNamespace(endpoint=SimpleNamespace(coordinate_format=None)) + vlm_output = [{"box_2d": [200, 100, 400, 300], "label": "home button"}] + + with ( + patch( + "artemis.agents.object_detector.object_detector.get_llm", + return_value=fake_llm, + ), + patch( + "artemis.agents.object_detector.object_detector._detect_single_label", + new_callable=AsyncMock, + return_value=vlm_output, + ), + ): + ctx = SimpleNamespace(llm_config=SimpleNamespace()) + result = await _run_object_detection( + ctx=ctx, + image_bytes=image_bytes, + queries=["home button"], + ) + + detected = result["detected"][0] + assert detected["point"] == [pytest.approx(200.0), pytest.approx(300.0)] + assert detected["box_2d"] == [100.0, 200.0, 300.0, 400.0] diff --git a/tests/unit/agents/test_outputter.py b/tests/unit/agents/test_outputter.py index 8def19fa9..259cf0b41 100644 --- a/tests/unit/agents/test_outputter.py +++ b/tests/unit/agents/test_outputter.py @@ -147,11 +147,11 @@ def __init__(self, messages, initial_goal, operator_raw_data): human_content = messages[1].content assert isinstance(human_content, list) - assert human_content[0]["type"] == "text" - assert "Step 1: Opened browser" in human_content[0]["text"] - assert "Plan: Find capital" in human_content[0]["text"] - assert human_content[1]["type"] == "image_url" - assert human_content[1]["image_url"]["url"] == "data:image/jpeg;base64,mock_b64_data" + assert human_content[0]["type"] == "image_url" + assert human_content[0]["image_url"]["url"] == "data:image/jpeg;base64,mock_b64_data" + assert human_content[1]["type"] == "text" + assert "Step 1: Opened browser" in human_content[1]["text"] + assert "Plan: Find capital" in human_content[1]["text"] @patch("artemis.agents.outputter.outputter.get_llm") diff --git a/tests/unit/agents/test_planner_node.py b/tests/unit/agents/test_planner_node.py index 9fabb1143..9aef9a384 100644 --- a/tests/unit/agents/test_planner_node.py +++ b/tests/unit/agents/test_planner_node.py @@ -82,8 +82,8 @@ async def mock_astream(*args, **kwargs): human_msg_content = messages[1].content assert isinstance(human_msg_content, list) - assert human_msg_content[0]["type"] == "text" - assert "Goal: Test Goal" in human_msg_content[0]["text"] + assert human_msg_content[0]["type"] == "image_url" + assert "Goal: Test Goal" in human_msg_content[1]["text"] def test_validate_plan_format_single_sourced_status_alphabet(): diff --git a/tests/unit/agents/test_precondition_pixel_target.py b/tests/unit/agents/test_precondition_pixel_target.py index 99ddfa20a..a3a5851ae 100644 --- a/tests/unit/agents/test_precondition_pixel_target.py +++ b/tests/unit/agents/test_precondition_pixel_target.py @@ -28,7 +28,7 @@ def _texts(messages: list) -> list[str]: return [part["text"] for part in human.content if part["type"] == "text"] -def test_named_control_target_block_precedes_images(): +def test_named_control_target_block_follows_images(): item = { "action": "tap", "coordinates": [1001, 718], @@ -45,14 +45,18 @@ def test_named_control_target_block_precedes_images(): assert isinstance(messages[0], SystemMessage) assert isinstance(messages[1], HumanMessage) - first_text = messages[1].content[0]["text"] - assert first_text.startswith("[Target]") - assert "Kind: specific UI control" in first_text - assert "Label: 进入全屏模式" in first_text - assert "Resource ID: com.google.android.youtube:id/fullscreen_button" in first_text - assert "Class: android.widget.ImageView" in first_text - assert "[922, 655, 1080, 781]" in first_text - assert messages[1].content[1]["text"] == "[Image 1 (Reference)]" + blocks = messages[1].content + assert blocks[0]["type"] == "image_url" + assert blocks[1]["text"] == "[Image 1 (Reference)]" + assert blocks[2]["type"] == "image_url" + assert blocks[3]["text"] == "[Image 2 (Current State)]" + target_text = blocks[4]["text"] + assert target_text.startswith("[Target]") + assert "Kind: specific UI control" in target_text + assert "Label: 进入全屏模式" in target_text + assert "Resource ID: com.google.android.youtube:id/fullscreen_button" in target_text + assert "Class: android.widget.ImageView" in target_text + assert "[922, 655, 1080, 781]" in target_text assert any("Original Thinking" in t for t in _texts(messages)) assert any("Action: tap" in t for t in _texts(messages)) @@ -62,7 +66,8 @@ def test_coordinates_only_target_is_labelled_as_surface(): messages = pp._build_messages("SYSTEM", _PNG, _PNG, item, state=None) - first_text = messages[1].content[0]["text"] + blocks = messages[1].content + first_text = next(b["text"] for b in blocks if b.get("text", "").startswith("[Target]")) assert "Kind: coordinates only" in first_text # No hit test runs any more: the block states that nothing was recorded, it # does not claim the point was looked up in the UI hierarchy. diff --git a/tests/unit/config/test_local_models.py b/tests/unit/config/test_local_models.py new file mode 100644 index 000000000..740d2cba7 --- /dev/null +++ b/tests/unit/config/test_local_models.py @@ -0,0 +1,109 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the local model catalog and the `artemis model` CLI surface.""" + +from typer.testing import CliRunner + +from artemis.config.local_models import ( + DEFAULT_LOCAL_MODEL, + LOCAL_MODELS, + resolve_model_ref, +) +from artemis.interfaces.cli.commands.model import model_app + +runner = CliRunner() + + +class TestResolveModelRef: + def test_catalog_alias(self): + assert resolve_model_ref("gemma4-e4b") == ( + "gemma4-e4b", + LOCAL_MODELS["gemma4-e4b"].repo, + ) + + def test_default_alias_is_cataloged(self): + assert DEFAULT_LOCAL_MODEL in LOCAL_MODELS + + def test_repo_id_passthrough(self): + assert resolve_model_ref("org/some-model") == ( + "org/some-model", + "org/some-model", + ) + + def test_unknown_bare_name_rejected(self): + assert resolve_model_ref("not-a-model") is None + + +class TestModelCli: + def test_list_shows_catalog(self): + result = runner.invoke(model_app, ["list"]) + assert result.exit_code == 0 + assert "gemma4-e4b" in result.output + + def test_pull_rejects_unknown_alias(self): + result = runner.invoke(model_app, ["pull", "bogus-name"]) + assert result.exit_code == 1 + assert "Unknown model" in result.output + + def test_serve_rejects_unknown_alias(self): + result = runner.invoke(model_app, ["serve", "bogus-name"]) + assert result.exit_code == 1 + assert "Unknown model" in result.output + + def test_stop_without_state_is_clean(self, tmp_path, monkeypatch): + import artemis.interfaces.cli.commands.model as model_mod + + monkeypatch.setattr(model_mod, "_STATE_DIR", tmp_path) + result = runner.invoke(model_app, ["stop", "--port", "19191"]) + assert result.exit_code == 0 + assert "No managed server" in result.output + + +class TestVisionSoftTokens: + def test_catalog_default(self): + from artemis.config.local_models import LOCAL_MODELS + + assert LOCAL_MODELS["gemma4-e4b"].vision_soft_tokens == 1120 + + def test_apply_writes_processor_config(self, tmp_path, monkeypatch): + import json as _json + + import artemis.interfaces.cli.commands.model as model_mod + + snap = tmp_path / "snap" + snap.mkdir() + (snap / "processor_config.json").write_text( + _json.dumps({"image_processor": {"max_soft_tokens": 280}}) + ) + monkeypatch.setattr(model_mod, "_snapshot_path", lambda repo: snap) + assert model_mod._apply_vision_soft_tokens("any/repo", 1120) + cfg = _json.loads((snap / "processor_config.json").read_text()) + assert cfg["image_processor"]["max_soft_tokens"] == 1120 + + def test_apply_missing_key_is_noop(self, tmp_path, monkeypatch): + import json as _json + + import artemis.interfaces.cli.commands.model as model_mod + + snap = tmp_path / "snap" + snap.mkdir() + (snap / "processor_config.json").write_text(_json.dumps({"other": {}})) + monkeypatch.setattr(model_mod, "_snapshot_path", lambda repo: snap) + assert not model_mod._apply_vision_soft_tokens("any/repo", 1120) + + def test_serve_rejects_bad_token_value(self): + result = runner.invoke(model_app, ["serve", "--vision-tokens", "999"]) + assert result.exit_code == 1 + assert "vision-tokens" in result.output diff --git a/tests/unit/core/test_decision_client.py b/tests/unit/core/test_decision_client.py new file mode 100644 index 000000000..e137a07b0 --- /dev/null +++ b/tests/unit/core/test_decision_client.py @@ -0,0 +1,118 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for the System-One decision client.""" + +import io +import json +from unittest.mock import patch +import urllib.error + +import pytest + +from artemis.core.decision_client import DecisionClient + + +def _mock_response(body: dict): + return io.BytesIO(json.dumps(body).encode("utf-8")) + + +class _Closing(io.BytesIO): + """BytesIO that survives use as a urlopen context manager.""" + + def __enter__(self): + return self + + def __exit__(self, *args): + self.close() + + +def _mock_ctx(body: dict): + return _Closing(json.dumps(body).encode("utf-8")) + + +def test_noul_yes_verdict(): + client = DecisionClient(url="http://x/v1/decide", model="laya") + body = {"answers": {"q": {"noul": 0.92, "confidence": 0.9}}} + with patch("urllib.request.urlopen", return_value=_mock_ctx(body)) as mock: + result = client.noul("state text", "is it ok?") + assert result.answered + assert result.verdict is True + assert result.confidence == 0.9 + assert result.probabilities["yes"] == pytest.approx(0.92) + payload = json.loads(mock.call_args.args[0].data.decode()) + assert payload["questions"]["q"]["type"] == "noul" + assert payload["model"] == "laya" + + +def test_noul_no_verdict(): + client = DecisionClient(url="http://x/v1/decide") + body = {"answers": {"q": {"noul": 0.1, "confidence": 0.88}}} + with patch("urllib.request.urlopen", return_value=_mock_ctx(body)): + result = client.noul("s", "q?") + assert result.verdict is False + assert result.probabilities["no"] == pytest.approx(0.9) + + +def test_choice_verdict(): + client = DecisionClient(url="http://x/v1/decide") + body = { + "answers": { + "q": { + "choice": "settings", + "probabilities": {"settings": 0.8, "chrome": 0.2}, + "confidence": 0.8, + } + } + } + with patch("urllib.request.urlopen", return_value=_mock_ctx(body)): + result = client.choose("s", "pick", {"settings": "a", "chrome": "b"}) + assert result.verdict == "settings" + assert result.probabilities["settings"] == 0.8 + + +def test_network_error_fails_closed(): + client = DecisionClient(url="http://x/v1/decide") + with patch( + "urllib.request.urlopen", side_effect=urllib.error.URLError("down") + ): + result = client.noul("s", "q?") + assert not result.answered + assert result.verdict is None + assert result.confidence == 0.0 + + +def test_malformed_body_fails_closed(): + client = DecisionClient(url="http://x/v1/decide") + with patch("urllib.request.urlopen", return_value=_Closing(b"not json")): + result = client.noul("s", "q?") + assert not result.answered + + +def test_missing_answer_fields_fail_closed(): + client = DecisionClient(url="http://x/v1/decide") + body = {"answers": {"q": {}}} + with patch("urllib.request.urlopen", return_value=_mock_ctx(body)): + result = client.noul("s", "q?") + assert result.verdict is None + + +@pytest.mark.asyncio +async def test_anoul_async_wrapper(): + client = DecisionClient(url="http://x/v1/decide") + body = {"answers": {"q": {"noul": 0.7, "confidence": 0.95}}} + with patch("urllib.request.urlopen", return_value=_mock_ctx(body)): + result = await client.anoul("s", "q?") + assert result.verdict is True + assert result.confidence == 0.95 diff --git a/tests/unit/core/test_local_model_probe.py b/tests/unit/core/test_local_model_probe.py new file mode 100644 index 000000000..5120ea7fe --- /dev/null +++ b/tests/unit/core/test_local_model_probe.py @@ -0,0 +1,131 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Local model endpoint probe: guidance must be model-aware. + +Artemis catalog aliases get ``artemis model pull`` / ``artemis model serve`` +commands; BYO models on third-party OpenAI-compatible servers get neutral +hints — the probe must never emit a command Artemis cannot run on that +endpoint (or reference a foreign binary). +""" + +import sys +import types +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from artemis.core.diagnostics.probes.local_model_probe import ( + LocalModelEndpointProbe, + _artemis_managed, + _serve_port, +) + + +class _FakeLLM: + def __init__(self, model, api_base="http://127.0.0.1:8097/v1", provider="custom"): + self.provider = provider + self.model = model + self.api_base = api_base + + +@pytest.fixture +def catalog(): + """Inject a stand-in catalog module; restore the import miss afterwards.""" + fake = types.ModuleType("artemis.config.local_models") + fake.LOCAL_MODELS = {"gemma4-e4b": object(), "qwen-vl": object()} + sys.modules["artemis.config.local_models"] = fake + yield fake + del sys.modules["artemis.config.local_models"] + + +def _llms(*models, base="http://127.0.0.1:8097/v1"): + return [_FakeLLM(m, api_base=base) for m in models] + + +def _run(llms): + with patch( + "artemis.core.diagnostics.probes.local_model_probe._iter_configured_llms", + side_effect=lambda: llms, + ): + return __import__("asyncio").run(LocalModelEndpointProbe().probe()) + + +def _commands(result): + return [a.payload for a in result.actions if a.action_type == "command"] + + +def _hints(result): + return [a.payload for a in result.actions if a.action_type == "hint"] + + +def test_catalog_detection_and_port_parsing(catalog): + assert _artemis_managed("gemma4-e4b") is True + assert _artemis_managed("llama3.2-vision") is False + assert _serve_port("http://127.0.0.1:8097/v1") == "8097" + assert _serve_port("https://api.example.com/v1") is None + + +def test_unreachable_catalog_model_suggests_artemis_serve(catalog): + result = _run(_llms("gemma4-e4b", base="http://127.0.0.1:9/v1")) + assert _commands(result) == ["artemis model serve gemma4-e4b --port 9"] + assert result.status.name == "FAIL" + + +def test_unreachable_byo_model_suggests_neutral_hint(catalog): + result = _run(_llms("llama3.2-vision", base="http://127.0.0.1:9/v1")) + assert _commands(result) == [] + assert any("OpenAI-compatible" in h for h in _hints(result)) + assert "ondevice-agent-platform" not in " ".join(_hints(result)) + + +def _serve_models(*ids): + """Patch httpx so ``GET /models`` returns the given ids.""" + resp = MagicMock() + resp.raise_for_status = MagicMock() + resp.json = MagicMock(return_value={"data": [{"id": i} for i in ids]}) + client = MagicMock() + client.get = AsyncMock(return_value=resp) + cm = MagicMock() + cm.__aenter__ = AsyncMock(return_value=client) + cm.__aexit__ = AsyncMock(return_value=False) + return patch( + "artemis.core.diagnostics.probes.local_model_probe.httpx.AsyncClient", + return_value=cm, + ) + + +def test_missing_catalog_model_suggests_artemis_pull(catalog): + """A reachable endpoint lacking a catalog alias gets `artemis model pull`.""" + with _serve_models("something-else"): + result = _run(_llms("qwen-vl")) + assert _commands(result) == ["artemis model pull qwen-vl"] + + +def test_missing_byo_model_suggests_own_tooling(catalog): + with _serve_models("gemma4-e4b"): + result = _run(_llms("llama3.2-vision")) + assert _commands(result) == [] + assert any("own tooling" in h for h in _hints(result)) + + +def test_cloud_only_config_is_not_used_pass(catalog): + class _Cloud: + provider = "google" + model = "gemini-3.8-flash" + api_base = None + + result = _run([_Cloud()]) + assert result.status.name == "PASS" + assert result.summary == "Not used" diff --git a/tests/unit/mcp/test_action_executor_semantics.py b/tests/unit/mcp/test_action_executor_semantics.py index 1efa09adf..d8ceb397c 100644 --- a/tests/unit/mcp/test_action_executor_semantics.py +++ b/tests/unit/mcp/test_action_executor_semantics.py @@ -35,6 +35,7 @@ def _make_executor(width=1080, height=2400): ctx.device.device_height = height actuator = Mock() actuator.controller = Mock() + actuator.extensions.return_value = [] return McpActionExecutor(ctx, actuator=actuator) @@ -347,3 +348,408 @@ def test_click_sequence_accepts_a_serialized_pair_list(): _state(), ) assert wire_args["sequence"] == [[500, 600], [10, 10]] + + +# --- Resolved-target feedback: the description↔element disagreement note --- + + +def _dialer_state(): + """Three entries for one dial-pad key: the parent frame, the digit child and + the letter child all land on the same physical key.""" + state = _state() + state.indexed_elements = [ + { + "index": 1, + "center": [180, 1400], + "text": "5,JKL", + "bounds": [40, 1300, 320, 1500], + "class": "android.widget.FrameLayout", + "resource_id": "com.google.android.dialer:id/five", + "is_ocr": False, + }, + { + "index": 2, + "center": [180, 1360], + "text": "5", + "bounds": [150, 1320, 210, 1400], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/five_digit", + "is_ocr": False, + }, + { + "index": 3, + "center": [180, 1440], + "text": "JKL", + "bounds": [150, 1400, 210, 1480], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/five_letters", + "is_ocr": False, + }, + ] + state.indexed_points = [el["center"] for el in state.indexed_elements] + return state + + +def test_mismatch_note_fires_when_the_description_names_another_key(): + """The observed failure on dense keypads: 'the 1 key' narrated while the + index actually resolves to the JKL letter cell of the 5 key.""" + executor = _make_executor() + _, _, _, semantics = executor._translate( + "click", {"target": 3, "target_description": "the 1 key"}, _dialer_state() + ) + note = executor._description_mismatch_note( + {"target_description": "the 1 key"}, semantics + ) + assert note is not None + assert "'JKL'" in note["text"] + assert "five_letters" in note["text"] + + +def test_mismatch_note_names_the_index_the_description_meant(): + """'1 key' mis-resolved onto 'JKL': the element list carries a '1' digit + cell — the note must point at it, not just flag the error.""" + executor = _make_executor() + state = _dialer_state() + state.indexed_elements = [ + *state.indexed_elements, + { + "index": 4, + "center": [180, 960], + "text": "1", + "bounds": [150, 920, 210, 1000], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/dialpad_key_number", + "is_ocr": False, + }, + ] + _, _, _, semantics = executor._translate( + "click", {"target": 3, "target_description": "the 1 key"}, state + ) + note = executor._description_mismatch_note( + {"target_description": "the 1 key"}, semantics, state.indexed_elements + ) + assert note["suggestion"] == "[4] '1'" + + +def test_mismatch_hint_does_not_substring_match_single_digits_into_values(): + """'0 key' must not 'match' the digits field showing '(555) 004' — a + single-character token requires exact text equality.""" + executor = _make_executor() + elements = [ + { + "index": 1, + "center": [540, 400], + "text": "(555) 004", + "bounds": [40, 350, 1040, 450], + "class": "android.widget.EditText", + "resource_id": "com.google.android.dialer:id/digits", + "is_ocr": False, + }, + { + "index": 2, + "center": [540, 1900], + "text": "0", + "bounds": [500, 1850, 580, 1950], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/dialpad_key_number", + "is_ocr": False, + }, + ] + hint = executor._matching_element_hint({"0"}, elements) + assert hint == "[2] '0'" + + +def test_mismatch_hint_returns_none_without_the_element_list(): + executor = _make_executor() + assert executor._matching_element_hint({"5"}, None) is None + assert executor._matching_element_hint(set(), [{"text": "5"}]) is None + + +@pytest.mark.parametrize( + "description", + [ + "the JKL cell", # letter token matches the resolved text + None, # absent — the field is optional on index targets + "", # blank + ], +) +def test_mismatch_note_stays_quiet_when_the_description_agrees_or_is_absent(description): + executor = _make_executor() + _, _, _, semantics = executor._translate( + "click", + {"target": 3, **({"target_description": description} if description else {})}, + _dialer_state(), + ) + note = executor._description_mismatch_note( + {"target_description": description}, semantics + ) + assert note is None + + +def test_mismatch_note_stays_quiet_for_the_element_the_description_names(): + """'the 5 key' against the digit child '5' — description and resolved + element agree, no note.""" + executor = _make_executor() + _, _, _, semantics = executor._translate( + "click", {"target": 2, "target_description": "the 5 key"}, _dialer_state() + ) + assert executor._description_mismatch_note( + {"target_description": "the 5 key"}, semantics + ) is None + + +@pytest.mark.parametrize("description", ["the 1 key", "the five key"]) +def test_mismatch_note_fires_on_digit_and_spelled_digit_disagreement(description): + """Both the numeral and its spelled-out form are checked against the real + element text — 'five' naming the JKL cell is still a disagreement.""" + executor = _make_executor() + _, _, _, semantics = executor._translate( + "click", {"target": 3}, _dialer_state() + ) + note = executor._description_mismatch_note( + {"target_description": description}, semantics + ) + assert note is not None + assert "'JKL'" in note["text"] + + +def test_mismatch_note_never_fires_for_coordinate_targets(): + """A coordinate target's semantics carry only the model's own description — + there is nothing observed to disagree with.""" + executor = _make_executor() + _, _, _, semantics = executor._translate( + "click", + {"target": [500, 600], "target_description": "1 key"}, + _state(), + ) + note = executor._description_mismatch_note( + {"target_description": "1 key"}, semantics + ) + assert note is None + + +def _alarm_state(): + """The expanded DeskClock alarm card observed live: the switch sits inside + a CardView that fills the list row — taps on the card body hit the + container, not the control.""" + state = _state() + state.indexed_elements = [ + { + "index": 1, + "center": [540, 923], + "text": "8:30 AM Alarm", + "bounds": [42, 317, 1038, 1529], + "class": "androidx.cardview.widget.CardView", + "resource_id": None, + "is_ocr": False, + }, + { + "index": 2, + "center": [943, 380], + "text": "Collapse 8:30 AM alarm", + "bounds": [891, 317, 996, 443], + "class": "android.widget.ImageButton", + "resource_id": None, + "is_ocr": False, + }, + { + "index": 3, + "center": [923, 689], + "text": "8:30 AM alarm", + "bounds": [850, 626, 996, 752], + "class": "android.widget.Switch", + "resource_id": None, + "is_ocr": False, + }, + { + "index": 4, + "center": [923, 1838], + "text": "9:00 AM alarm", + "bounds": [850, 1775, 996, 1901], + "class": "android.widget.Switch", + "resource_id": None, + "is_ocr": False, + }, + ] + state.indexed_points = [el["center"] for el in state.indexed_elements] + return state + + +def test_coordinate_tap_on_a_leaf_checked_against_its_real_text(): + """A point inside the 'JKL' cell is the same resolution as clicking its + index — 'the 1 key' narrated over it is a mismatch naming '1'.""" + executor = _make_executor() + state = _dialer_state() + state.indexed_elements.append( + { + "index": 4, + "center": [500, 960], + "text": "1", + "bounds": [450, 920, 550, 1000], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/dialpad_key_number", + "is_ocr": False, + } + ) + state.indexed_points.append([500, 960]) + note = executor._coordinate_target_note( + {"target_description": "the 1 key"}, [166, 600], state.indexed_elements + ) + assert note is not None + assert "'JKL'" in note["text"] + assert "[4] '1'" in note["text"] + + +def test_coordinate_tap_inside_the_described_element_stays_quiet(): + """The point lands inside the switch the description names — nothing to + correct.""" + executor = _make_executor() + state = _alarm_state() + # [923,689]px is the switch center → normalized [855,287] on 1080x2400. + assert executor._coordinate_target_note( + {"target_description": "Toggle switch for 8:30 AM alarm"}, + [855, 287], + state.indexed_elements, + ) is None + + +def test_coordinate_tap_on_the_card_body_rejects_with_the_switch_index(): + """The live alarm failure: [873,351] normalized lands on the expanded + CardView, not the switch — 'toggle switch' picks the Switch class out of + three identically-labelled candidates.""" + executor = _make_executor() + note = executor._coordinate_target_note( + {"target_description": "Toggle switch for 8:30 AM alarm"}, + [873, 351], + _alarm_state().indexed_elements, + ) + assert note is not None + assert "container" in note["text"] + assert note["suggestion"] == "[3] '8:30 AM alarm'" + + +def test_coordinate_tap_on_dead_space_rejects_with_the_named_element(): + """Nothing listed under the point and the description identifies exactly + one element — refuse the tap and name it.""" + executor = _make_executor() + state = _dialer_state() + state.indexed_elements.append( + { + "index": 4, + "center": [500, 960], + "text": "1", + "bounds": [450, 920, 550, 1000], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/dialpad_key_number", + "is_ocr": False, + } + ) + state.indexed_points.append([500, 960]) + note = executor._coordinate_target_note( + {"target_description": "the 1 key"}, [500, 100], state.indexed_elements + ) + assert note is not None + assert note["suggestion"] == "[4] '1'" + + +def test_coordinate_tap_on_dead_space_without_a_confident_match_only_notes(): + """A vague description can't prove the point wrong — soft note, no + suggestion, so the tap still proceeds.""" + executor = _make_executor() + note = executor._coordinate_target_note( + {"target_description": "the green button"}, + [500, 100], + _dialer_state().indexed_elements, + ) + assert note is not None + assert note["suggestion"] is None + assert "no listed element" in note["text"] + + +def test_coordinate_tap_on_a_container_matching_its_description_stays_quiet(): + """Dial-pad keys are clickable FrameLayouts — a tap inside '5,JKL' + described as 'the 5 key' is a container hit that agrees with the + description and must not be flagged.""" + executor = _make_executor() + # [60,1480]px is inside the FrameLayout but outside both child labels. + assert executor._coordinate_target_note( + {"target_description": "the 5 key"}, [55, 616], _dialer_state().indexed_elements + ) is None + + +@pytest.mark.asyncio +async def test_execute_rejects_a_coordinate_tap_landing_on_a_container(): + """End to end: the described switch is findable while the point lands on + the alarm card — the tap must not reach the device.""" + from unittest.mock import AsyncMock, patch + + executor = _make_executor() + state = _alarm_state() + state.latest_screenshot = None + session = AsyncMock() + with patch.object(executor, "_session_or_start", return_value=session): + result = await executor.execute( + "click", + {"target": [873, 351], "target_description": "Toggle switch for 8:30 AM alarm"}, + "tc", + state, + ) + session.call.assert_not_called() + assert result.status == "error" + assert "[3] '8:30 AM alarm'" in result.text_summary + + +@pytest.mark.asyncio +async def test_execute_rejects_the_tap_when_a_confident_alternative_exists(): + """'1 key' narrated over an index resolving to 'JKL' while a '1' element is + listed: the tap must not reach the device — the result refuses it and names + the right index (a wrong tap on a dense pad corrupts the digits field).""" + from unittest.mock import AsyncMock, patch + + executor = _make_executor() + state = _dialer_state() + state.indexed_elements.append( + { + "index": 4, + "center": [180, 960], + "text": "1", + "bounds": [150, 920, 210, 1000], + "class": "android.widget.TextView", + "resource_id": "com.google.android.dialer:id/dialpad_key_number", + "is_ocr": False, + } + ) + state.indexed_points.append([180, 960]) + state.latest_screenshot = None + session = AsyncMock() + with patch.object(executor, "_session_or_start", return_value=session): + result = await executor.execute( + "click", {"target": 3, "target_description": "the 1 key"}, "tc", state + ) + session.call.assert_not_called() + assert result.status == "error" + assert "resolves to 'JKL'" in result.text_summary + assert "[4] '1'" in result.text_summary + + +@pytest.mark.asyncio +async def test_execute_still_taps_when_the_mismatch_has_no_alternative(): + """A loose description that matches nothing can't prove the index wrong — + the tap proceeds (with the note appended on success).""" + from unittest.mock import AsyncMock, Mock, patch + + executor = _make_executor() + state = _dialer_state() + state.latest_screenshot = None + session = AsyncMock() + res = Mock(ok=True, message="Tapped.", code=Mock(value="OK"), normalized_coordinates=None) + session.call = AsyncMock(return_value=res) + obs = Mock(ok=True, screenshot_path=None, elements_text="L", hierarchy_ok=False, elements=[]) + session.observe = AsyncMock(return_value=obs) + with patch.object(executor, "_session_or_start", return_value=session): + result = await executor.execute( + "click", {"target": 3, "target_description": "the green button"}, "tc", state + ) + session.call.assert_called_once() + assert result.status == "success" + assert "resolves to 'JKL'" in result.text_summary diff --git a/tests/unit/test_config_package.py b/tests/unit/test_config_package.py index 5b8c56ad8..06c233880 100644 --- a/tests/unit/test_config_package.py +++ b/tests/unit/test_config_package.py @@ -152,7 +152,9 @@ def test_llm_config_parsing_and_merging(): """Test LLMConfig parsing, agent querying, and deep merging.""" llm_cfg = get_default_llm_config() assert isinstance(llm_cfg, LLMConfig) - assert llm_cfg.planner.provider in ("google", "openai", "openrouter", "xai", "vertexai") + assert llm_cfg.planner.provider in ( + "google", "openai", "openrouter", "xai", "vertexai", "custom", + ) assert llm_cfg.get_agent("planner") is not None assert llm_cfg.get_utils("hopper") is not None diff --git a/tests/unit/test_llm_local_endpoint.py b/tests/unit/test_llm_local_endpoint.py new file mode 100644 index 000000000..0355cb54c --- /dev/null +++ b/tests/unit/test_llm_local_endpoint.py @@ -0,0 +1,188 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Endpoint plumbing for local OpenAI-compatible providers (e.g. a local OpenAI-compatible server).""" + +from types import SimpleNamespace + +from artemis.config.llm import LLMConfig +from artemis.llm.router import ModelEndpoint, ModelFactory, ModelProvider +from artemis.services.llm import _resolve_endpoint +from third_party.mobile_use.utils.file import strip_json_comments + + +def _ctx(**node_overrides) -> SimpleNamespace: + utils = { + "outputter": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "hopper": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "video_analyzer": None, + "object_detector": None, + } + utils.update(node_overrides.pop("utils", {})) + config = LLMConfig.model_validate( + { + "planner": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "summarizer": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "operator": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "operator_summarizer": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "log_reader_sub_agent": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "log_analyzer": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "diagnoser": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "checker": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "planner_avatar": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "history_analyzer_expert": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "diagnoser_expert": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "explorer": { + "provider": "google", + "model": "g", + "fallback": {"provider": "google", "model": "g"}, + }, + "utils": utils, + **node_overrides, + } + ) + return SimpleNamespace(llm_config=config) + + +_LOCAL_NODE = { + "provider": "custom", + "model": "qwen-vl", + "api_base": "http://127.0.0.1:8080/v1", + "coordinate_format": "xy_px", + "max_tokens": 1024, + "fallback": { + "provider": "custom", + "model": "qwen-vl", + "api_base": "http://127.0.0.1:8080/v1", + "coordinate_format": "xy_px", + }, +} + + +def test_resolve_endpoint_forwards_api_base_and_coordinate_format(): + ctx = _ctx(utils={"object_detector": _LOCAL_NODE}) + ep = _resolve_endpoint(ctx, "object_detector", is_utils=True) + assert ep.provider == ModelProvider.CUSTOM + assert ep.model_name == "qwen-vl" + assert ep.api_base == "http://127.0.0.1:8080/v1" + assert ep.coordinate_format == "xy_px" + assert ep.max_tokens == 1024 + + +def test_resolve_endpoint_fallback_carries_local_fields(): + ctx = _ctx(utils={"object_detector": _LOCAL_NODE}) + ep = _resolve_endpoint(ctx, "object_detector", is_utils=True, use_fallback=True) + assert ep.api_base == "http://127.0.0.1:8080/v1" + assert ep.coordinate_format == "xy_px" + + +def test_resolve_endpoint_defaults_keep_google_shape(): + ctx = _ctx() + ep = _resolve_endpoint(ctx, "operator") + assert ep.provider == ModelProvider.GOOGLE + assert ep.api_base is None + assert ep.coordinate_format is None + + +def test_custom_endpoint_builds_chat_openai_with_local_base(monkeypatch): + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + ep = ModelEndpoint( + provider=ModelProvider.CUSTOM, + model_name="qwen-vl", + api_base="http://127.0.0.1:8080/v1", + ) + model = ModelFactory.create_model(ep) + assert type(model).__name__ == "ChatOpenAI" + assert str(model.openai_api_base) == "http://127.0.0.1:8080/v1" + + +def test_empty_openai_env_key_falls_back_to_placeholder(monkeypatch): + """An exported-but-empty OPENAI_API_KEY must not reach ChatOpenAI.""" + monkeypatch.setenv("OPENAI_API_KEY", "") + ep = ModelEndpoint( + provider=ModelProvider.CUSTOM, + model_name="qwen3.8-9b", + api_base="http://127.0.0.1:8080/v1", + ) + model = ModelFactory.create_model(ep) + assert type(model).__name__ == "ChatOpenAI" + assert model.openai_api_key.get_secret_value() == "EMPTY" + + +def test_strip_json_comments_preserves_urls_and_strings(): + text = ( + "{ // header\n" + ' "api_base": "http://127.0.0.1:8080/v1", // trailing\n' + ' "note": "x /* not a comment */ y",\n' + ' "escaped": "a \\"//\\" b"\n' + "}" + ) + import json + + parsed = json.loads(strip_json_comments(text)) + assert parsed["api_base"] == "http://127.0.0.1:8080/v1" + assert parsed["note"] == "x /* not a comment */ y" + assert parsed["escaped"] == 'a "//" b' diff --git a/tests/unit/tools/history/test_screenshot.py b/tests/unit/tools/history/test_screenshot.py index 244f59f91..ad4968486 100644 --- a/tests/unit/tools/history/test_screenshot.py +++ b/tests/unit/tools/history/test_screenshot.py @@ -49,9 +49,9 @@ def test_pre_and_post_screenshots_are_attached_as_content_blocks(tmp_path): assert shot.image_bytes == pre.read_bytes() assert shot.description == "Screenshot of step 2 (pre-action) is attached." blocks = shot.to_content_blocks() - assert blocks[0] == {"type": "text", "text": shot.description} - assert blocks[1]["type"] == "image_url" - url = blocks[1]["image_url"]["url"] + assert blocks[1] == {"type": "text", "text": shot.description} + assert blocks[0]["type"] == "image_url" + url = blocks[0]["image_url"]["url"] assert url.startswith("data:image/jpeg;base64,") assert base64.b64decode(url.split(",", 1)[1]) == pre.read_bytes() @@ -151,7 +151,7 @@ async def test_tool_execute_returns_blocks_or_text(tmp_path): pre = _jpeg_file(tmp_path) ctx = SimpleNamespace(data_engine=_reader(pre=pre)) out = await get_step_screenshot.execute(ctx=ctx, step_number=1, which="pre") - assert isinstance(out, list) and out[1]["type"] == "image_url" + assert isinstance(out, list) and out[0]["type"] == "image_url" out = await get_step_screenshot.execute(ctx=ctx, step_number=1, which="post") assert out == "No post-action screenshot recorded for step 1." diff --git a/tests/unit/tools/test_explorer_tool.py b/tests/unit/tools/test_explorer_tool.py index 5e363b3fa..2b56f52ec 100644 --- a/tests/unit/tools/test_explorer_tool.py +++ b/tests/unit/tools/test_explorer_tool.py @@ -349,13 +349,13 @@ def fake_draw_dots(screenshot_path, points, labels, output_path, **kwargs): assert kwargs["color"] == "magenta" assert isinstance(result, list) and len(result) == 2 - assert result[0]["type"] == "text" - assert "Explorer located 1 candidate(s)" in result[0]["text"] + assert result[1]["type"] == "text" + assert "Explorer located 1 candidate(s)" in result[1]["text"] # The Operator's click takes a bare integer index. - assert "target=3, a bare integer" in result[0]["text"] - assert "target=[x, y]" not in result[0]["text"] - assert result[1]["type"] == "image_url" - assert result[1]["image_url"]["url"].startswith("data:image/jpeg;base64,") + assert "target=3, a bare integer" in result[1]["text"] + assert "target=[x, y]" not in result[1]["text"] + assert result[0]["type"] == "image_url" + assert result[0]["image_url"]["url"].startswith("data:image/jpeg;base64,") def test_render_operator_blocks_numbers_annotations_sequentially(tmp_path): @@ -547,8 +547,8 @@ def fake_draw_dots(screenshot_path, points, labels, output_path, **kwargs): ) assert tool.name == "ask_explorer" - assert isinstance(result, list) and result[1]["type"] == "image_url" - assert "- [1] 'B' at normalized [500, 500]" in result[0]["text"] + assert isinstance(result, list) and result[0]["type"] == "image_url" + assert "- [1] 'B' at normalized [500, 500]" in result[1]["text"] @pytest.mark.asyncio @@ -582,7 +582,7 @@ def fake_draw_dots(screenshot_path, points, labels, output_path, **kwargs): assert instance.run.await_args.kwargs["version"] == "ultra" assert isinstance(result, list) and len(result) == 2 - assert result[0]["type"] == "text" and result[1]["type"] == "image_url" + assert result[0]["type"] == "image_url" and result[1]["type"] == "text" assert state.indexed_points == [[540, 1200]] diff --git a/tests/unit/utils/test_optional_ocr.py b/tests/unit/utils/test_optional_ocr.py index d8aa0dcd7..e7733e5b3 100644 --- a/tests/unit/utils/test_optional_ocr.py +++ b/tests/unit/utils/test_optional_ocr.py @@ -39,15 +39,19 @@ async def test_is_ocr_configured_false_when_unset(monkeypatch): monkeypatch.delenv("API_KEY", raising=False) monkeypatch.delenv("GOOGLE_API_KEY", raising=False) - with patch.object(Settings, "get_api_key", return_value=None): + with ( + patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=False), + ): assert not is_ocr_configured() @pytest.mark.asyncio async def test_perform_ocr_returns_empty_list_when_unconfigured(monkeypatch): - """Verify perform_ocr returns [] when no OCR key is present.""" + """Verify perform_ocr returns [] when no OCR provider is available.""" with ( patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=False), patch.dict("os.environ", {}, clear=True), ): result = await perform_ocr(screenshot_b64="dummy_b64") @@ -62,6 +66,7 @@ async def test_perform_ocr_executes_api_call_when_configured(): with ( patch.object(Settings, "get_api_key", return_value=mock_key), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=False), patch("artemis.utils.ocr_api.get_http_client") as mock_client, ): mock_resp = MagicMock() @@ -90,6 +95,158 @@ async def test_perform_ocr_executes_api_call_when_configured(): assert result[0]["text"] == "Login" +def test_is_ocr_configured_true_with_apple_vision(monkeypatch): + """Apple Vision alone (no Google key) makes OCR available on macOS.""" + monkeypatch.delenv("OCR_API_KEY", raising=False) + monkeypatch.delenv("VISION_API_KEY", raising=False) + with ( + patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + ): + assert is_ocr_configured() + + +@pytest.mark.asyncio +async def test_perform_ocr_uses_apple_vision_first(monkeypatch): + """Under auto provider, on-device Apple Vision answers without any HTTP call.""" + monkeypatch.delenv("OCR_API_KEY", raising=False) + monkeypatch.delenv("VISION_API_KEY", raising=False) + apple_result = [{"text": "Login", "position": [{"x": 1, "y": 2}]}] + with ( + patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + patch( + "artemis.utils.ocr_api._run_apple_vision", + new_callable=AsyncMock, + return_value=apple_result, + ) as mock_apple, + ): + result = await perform_ocr(screenshot_b64="aGVsbG8=") + assert result == apple_result + mock_apple.assert_awaited_once_with(b"hello") + + +@pytest.mark.asyncio +async def test_perform_ocr_falls_back_to_google_on_apple_failure(): + """An Apple Vision error under auto falls back to a keyed Google Vision.""" + mock_key = MagicMock() + mock_key.get_secret_value.return_value = "test_key" + + with ( + patch.object(Settings, "get_api_key", return_value=mock_key), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + patch( + "artemis.utils.ocr_api._run_apple_vision", + new_callable=AsyncMock, + side_effect=RuntimeError("boom"), + ), + patch("artemis.utils.ocr_api.get_http_client") as mock_client, + ): + mock_resp = MagicMock() + mock_resp.status_code = 200 + mock_resp.json.return_value = { + "responses": [ + { + "textAnnotations": [ + {"description": "Full", "boundingPoly": {}}, + { + "description": "Login", + "boundingPoly": {"vertices": [{"x": 1, "y": 2}]}, + }, + ] + } + ] + } + mock_http = AsyncMock() + mock_http.post.return_value = mock_resp + mock_client.return_value = mock_http + + result = await perform_ocr(screenshot_b64="aGVsbG8=") + assert result[0]["text"] == "Login" + + +@pytest.mark.asyncio +async def test_perform_ocr_apple_failure_without_key_returns_empty(monkeypatch): + """Apple failure with no Google key degrades to [] instead of raising.""" + monkeypatch.delenv("OCR_API_KEY", raising=False) + monkeypatch.delenv("VISION_API_KEY", raising=False) + with ( + patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + patch( + "artemis.utils.ocr_api._run_apple_vision", + new_callable=AsyncMock, + side_effect=RuntimeError("boom"), + ), + ): + assert await perform_ocr(screenshot_b64="aGVsbG8=") == [] + + +@pytest.mark.asyncio +async def test_perform_ocr_forced_apple_never_calls_google(monkeypatch): + """ARTEMIS_OCR_PROVIDER=apple must not fall through to the cloud provider.""" + monkeypatch.setenv("ARTEMIS_OCR_PROVIDER", "apple") + mock_key = MagicMock() + mock_key.get_secret_value.return_value = "test_key" + with ( + patch.object(Settings, "get_api_key", return_value=mock_key), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + patch( + "artemis.utils.ocr_api._run_apple_vision", + new_callable=AsyncMock, + side_effect=RuntimeError("boom"), + ), + patch("artemis.utils.ocr_api.get_http_client") as mock_client, + ): + assert await perform_ocr(screenshot_b64="aGVsbG8=") == [] + mock_client.assert_not_called() + + +@pytest.mark.asyncio +async def test_perform_ocr_apple_breaker_stops_dispatch_after_failures(monkeypatch): + """Consecutive Apple subprocess failures disable the provider; a third + call must not spawn another worker.""" + import artemis.utils.ocr_api as ocr_api + + monkeypatch.delenv("OCR_API_KEY", raising=False) + monkeypatch.delenv("VISION_API_KEY", raising=False) + monkeypatch.setattr(ocr_api, "_apple_vision_consecutive_failures", 0) + with ( + patch.object(Settings, "get_api_key", return_value=None), + patch("artemis.utils.ocr_api._apple_vision_supported", return_value=True), + patch( + "artemis.utils.ocr_api._apple_vision_subprocess", + new_callable=AsyncMock, + side_effect=RuntimeError("wedged"), + ) as mock_worker, + ): + assert await perform_ocr(screenshot_b64="aGVsbG8=") == [] + assert await perform_ocr(screenshot_b64="aGVsbG8=") == [] + assert mock_worker.await_count == 4 + assert await perform_ocr(screenshot_b64="aGVsbG8=") == [] + assert not ocr_api.is_ocr_configured() + assert mock_worker.await_count == 4 + + +def test_vision_box_to_vertices_converts_bottom_left_origin(): + """Vision's normalized bottom-left box maps to TL/TR/BR/BL pixel vertices.""" + from types import SimpleNamespace + + from artemis.utils.apple_vision_ocr import _vision_box_to_vertices + + bb = SimpleNamespace( + origin=SimpleNamespace(x=0.1, y=0.8), + size=SimpleNamespace(width=0.2, height=0.05), + ) + vertices = _vision_box_to_vertices(bb, width=1000, height=2000) + assert vertices == [ + {"x": 100, "y": 300}, + {"x": 300, "y": 300}, + {"x": 300, "y": 400}, + {"x": 100, "y": 400}, + ] + + def test_fuse_ocr_with_xml_empty_ocr_retains_xml_pruning(): """Verify fuse_ocr_with_xml preserves full XML structure and allows downstream pruning.""" sample_xml = [ diff --git a/third_party/mobile_use/config/llm.py b/third_party/mobile_use/config/llm.py index be9f509a9..3f764c1b4 100644 --- a/third_party/mobile_use/config/llm.py +++ b/third_party/mobile_use/config/llm.py @@ -58,11 +58,17 @@ class LLM(BaseModel): provider: LLMProvider model: str temperature: float | None = None + api_base: str | None = None + max_tokens: int | None = None + coordinate_format: ( + Literal["yx_1000", "xy_1000", "yx_px", "xy_px", "yx_norm", "xy_norm"] | None + ) = None thinking_budget: int | None = None thinking_level: Literal["minimal", "low", "medium", "high"] | None = None reasoning_effort: Literal["none", "low", "medium", "high"] | None = None include_thoughts: bool | None = None enable_grounding: bool | None = None + timeout: float | None = None def validate_provider(self, name: str) -> None: """Ensure the required API key or credentials exist in settings for this provider.""" diff --git a/third_party/mobile_use/utils/file.py b/third_party/mobile_use/utils/file.py index e7930014e..055adc105 100644 --- a/third_party/mobile_use/utils/file.py +++ b/third_party/mobile_use/utils/file.py @@ -17,14 +17,51 @@ # See third_party/mobile_use/METADATA for the upstream source and local modifications. import json -import re from typing import IO def strip_json_comments(text: str) -> str: - text = re.sub(r"//.*?$", "", text, flags=re.MULTILINE) - text = re.sub(r"/\*.*?\*/", "", text, flags=re.DOTALL) - return text + """Remove ``//`` and ``/* */`` comments while preserving string literals. + + The naive regex version stripped ``//`` sequences inside quoted strings, + which made URL values (e.g. ``"api_base": "http://127.0.0.1:8080/v1"``) + unparseable. This scan tracks string/escape state and only treats the + comment markers that appear outside strings as comments. + """ + out: list[str] = [] + i, n = 0, len(text) + in_string = False + while i < n: + char = text[i] + if in_string: + out.append(char) + if char == "\\" and i + 1 < n: + out.append(text[i + 1]) + i += 2 + continue + if char == '"': + in_string = False + i += 1 + continue + if char == '"': + in_string = True + out.append(char) + i += 1 + continue + if char == "/" and i + 1 < n: + if text[i + 1] == "/": + while i < n and text[i] != "\n": + i += 1 + continue + if text[i + 1] == "*": + i += 2 + while i + 1 < n and not (text[i] == "*" and text[i + 1] == "/"): + i += 1 + i += 2 + continue + out.append(char) + i += 1 + return "".join(out) def load_jsonc(file: IO) -> dict: