Add poll-loop liveness watchdog to auto-restart wedged workers

[manan164]

Problem

Workers occasionally stop polling and stay stopped until manually restarted (reported by a customer running async workers, HTTP/1.1, thread_count=8, fast ~10ms tasks; server metrics clean during the stall).

The poll loop runs poll → execute → update on a single thread (one event loop for async workers). If an SDK→server HTTP call never returns — e.g. a stale keep-alive connection silently dropped by a proxy/LB/NAT (no RST), which never reaches the server so server metrics stay clean — or a blocking call freezes the loop, polling halts with no error and no crash.

TaskHandler already supervises and restarts worker processes, but only when process.is_alive()==False. A wedged-but-alive process is invisible to it, so the worker stays dead until an operator restarts it.

What this changes

Adds a poll-loop liveness watchdog to both TaskRunner and AsyncTaskRunner:

• run_once() records a monotonic heartbeat each iteration. A healthy loop reaches it within milliseconds even at full capacity (a busy loop still spins), so this does not false-positive on legitimately busy workers.
• A daemon thread (so a frozen event loop can't block it) exits the process via os._exit(70) once the loop has been silent past CONDUCTOR_WORKER_POLL_STALL_TIMEOUT_SECONDS (default 300s, 0 disables).
• The existing supervisor (restart_on_failure) then restarts the worker — turning a permanent stall into a few-second blip.

Scope / honesty

This is a defense-in-depth backstop for the symptom, not a root-cause fix. It auto-recovers a stalled worker regardless of why it wedged. The critical log it emits points operators at capturing a stack dump (py-spy dump --pid <pid>) to diagnose the underlying blocked call.

Empirically, the httpx transport the SDK uses does time out and self-heal on a half-open connection (verified locally), so the bounded-stall case already recovers; this watchdog covers the permanent-wedge case that the supervisor's is_alive()-only check cannot.

Config

Env	Default	Meaning
CONDUCTOR_WORKER_POLL_STALL_TIMEOUT_SECONDS	300	Max poll-loop silence before restart; 0 disables

Tests

tests/unit/automator/test_poll_stall_watchdog.py — env parsing, stall/fresh/disabled/shutdown decision logic for both runners. All 13 pass; existing 52 runner tests still pass.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 73.23944% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rc/conductor/client/automator/async_task_runner.py	61.11%	14 Missing ⚠️
src/conductor/client/automator/task_runner.py	85.71%	5 Missing ⚠️

Files with missing lines	Coverage Δ
src/conductor/client/automator/task_runner.py	79.01% <85.71%> (+0.42%)	⬆️
...rc/conductor/client/automator/async_task_runner.py	57.90% <61.11%> (+0.23%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[manan164]

Closing — the watchdog is a band-aid, not a root-cause fix. Reverting to diagnose the actual cause first.