Skip to content

Fix reasoning manager response validation#377

Draft
RasulOs wants to merge 3 commits into
mainfrom
fix/reasoning-manager-response-contract
Draft

Fix reasoning manager response validation#377
RasulOs wants to merge 3 commits into
mainfrom
fix/reasoning-manager-response-contract

Conversation

@RasulOs

@RasulOs RasulOs commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Harden manager response parsing so final success is bound to the actual <request_accomplished>/<answer> tag and missing/invalid success no longer defaults to true.
  • Add shared manager response validation used by both stateful and stateless managers.
  • Prevent invalid manager output from being returned after retry exhaustion or retry-call failures.
  • Preserve the approved fallback for a valid single plan plus stray final tags by stripping final tags and continuing with the plan.
  • Convert non-recoverable manager validation failures into FinalizeEvent(success=False) and make success=None final answers fail closed.
  • Clarify manager prompt contracts to require exactly one of plan or final answer.

Tests

  • UV_CACHE_DIR=/private/tmp/uv-cache uv sync --python 3.13 --group dev --locked
  • UV_CACHE_DIR=/private/tmp/uv-cache uv run ruff check .
  • UV_CACHE_DIR=/private/tmp/uv-cache uv run pytest -q -> 239 passed
  • UV_CACHE_DIR=/private/tmp/uv-cache uv run python -m compileall -q mobilerun tests
  • UV_CACHE_DIR=/private/tmp/uv-cache uv lock --check
  • UV_CACHE_DIR=/private/tmp/uv-cache uv run --with black black --check mobilerun/agent/manager/prompts.py mobilerun/agent/manager/manager_agent.py mobilerun/agent/manager/stateless_manager_agent.py mobilerun/agent/droid/droid_agent.py tests/test_manager_response_validation.py

Live regression

  • Local Android: emulator-5554; mobilerun ping --device emulator-5554 passed.
  • Anthropic OAuth reasoning smoke passed with Android version prompt; no manager validation retry; reported Android 16.
  • Gemini OAuth reasoning smoke passed with Android version prompt; no manager validation retry; reported Android 16.
  • OpenAI OAuth reasoning smoke was blocked by saved OAuth refresh failure: 401 Unauthorized from https://auth.openai.com/oauth/token. Classified as credential/login blocker, not code regression.
  • Cloud command regression passed against ready device 5a4d2cd6-0ba6-49ac-bcdc-d4c709446017: devices --cloud, device ui --cloud, device screenshot --cloud, and device press home --cloud.

Independent reviews

  • Claude Opus 4.8 max reviewed the final diff after one temporary 529 Overloaded retry. It found the implementation met the requirements and suggested extra tests for plan+empty/duplicate final tag salvage and run_manager failure wiring; those were added.
  • Codex 5.5 xhigh reviewed the implementation. Its empty-plan validation finding was fixed. Its concern about strip-and-continue for plan+final conflicts with the approved plan and Claude accepted that behavior as intentional.

@RasulOs RasulOs marked this pull request as draft June 24, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant