fix: idle-timeout for agent-loop engine calls, infra-failure escalation - #104
Merged
Conversation
… infra-failure escalation The flat 15min wall-clock timeout added yesterday (PR #102) killed the Implement step mid-exploration on a wide-scope issue (#52) that was actively producing output, not hung. It couldn't tell "genuinely stuck" (the 84min zero-output hang on issue #85 that motivated PR #102) from "slow but working". run-engine.sh now kills a call only after N seconds with NO NEW output, with a much larger hard ceiling as backstop; Implement gets a generous idle/ceiling budget, Review (where the real hang happened) stays tight, and the one-round fix-call gets its own inline override. Also, while auditing the loop for other robustness gaps: - Infra failures (engine crash, timeout, no result marker) never had a counter or escalation path — an issue that reliably infra-fails would retry forever, 3x/day, with no human ever finding out. They now get their own counter and escalate to a new `agent:infra-stuck` label after 3, excluded from future picks like `agent:blocked` already is. - A DONE-but-empty-diff guard failure (known opencode silent-exit bug, anomalyco/opencode#28605) was writing `result=DONE` before the guard could invalidate it, so Cleanup miscounted it as a real attempt instead of the infra failure it actually is. - Every failure path now writes a short reason (implement-idle-timeout, verify-backend, review-fix-exhausted, ...) into the issue comment instead of a generic "attempt N failed" — no more opening the Actions log just to see which stage broke. - Verify — landing now installs Chromium before building, matching ci.yml's landing-build job (agent-loop's Verify could previously diverge from what real CI would say). - Added a Verify — pip-audit (CVE gate) step: confirmed via branch protection that pip-audit isn't a required context and agent-loop never checked it either, so a CVE-flagged dependency bump could previously auto-merge unblocked. Verified main is currently clean before wiring this in as a blocking step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KpuHkuMadFm8SGgw7nuZki
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run-engine.sh: replace the flat 15min wall-clocktimeout(PR fix: bound engine calls with a per-call timeout in run-engine.sh #102) with an idle-timeout — a call is only killed after N seconds with zero new stdout/stderr, with a much larger hard ceiling as backstop. Fixes today's incident where the Implement step was killed mid-exploration on issue [data-fidelity] Sprint 1 — metadata trust: status enum, 4KB frontmatter, article identity #52 while still actively producing output.anomalyco/opencode#28605) was wrongly counted as a real attempt instead of the infra failure it is.agent:infra-stucklabel after 3 — previously they could retry forever, 3x/day, with no escalation.implement-idle-timeout,verify-backend,review-fix-exhausted, ...) instead of a generic message — no more opening the Actions log to find out which stage broke.Verify — landingnow installs Chromium before building, matchingci.yml'slanding-buildjob.Verify — pip-audit (CVE gate)step: confirmed viagh api .../branches/main/protectionthat pip-audit isn't a required branch-protection context, and agent-loop never checked it either — a CVE-flagged dependency could previously auto-merge unblocked. Verifiedmainis currently clean (uv run --with pip-audit pip-audit --skip-editable→ "No known vulnerabilities found") before wiring this in as blocking.pick-issue.sh: excludes the newagent:infra-stucklabel from future picks.README.mdandscripts/setup-github.shupdated to match (label already created on the repo).Validation done locally
bash -non all three shell scripts, YAML-parsed the workflow (js-yaml) — all clean.pip-auditrun locally againstmain: no known vulnerabilities.Known residual risk (documented, not blocking)
run-engine.sh's idle-kill usespkill -TERM -P "$runner_pid", which only reaps direct children, not grandchildren — if opencode/cursor-agent spawn multi-level subprocess trees, an idle-kill could theoretically leave an orphan running. Plan is to validate via aworkflow_dispatchtest run before trusting the 3x/day cron on this, per the README's own existing practice for engine/model changes, and watch the job log for orphan processes.Test plan
workflow_dispatchon a low-risk issue to confirm: idle-timeout doesn't kill active work, hard ceiling still works as backstop, no orphan processes, failure-reason appears correctly in issue comments if a failure is forced.Claude-Session: https://claude.ai/code/session_01KpuHkuMadFm8SGgw7nuZki