Skip to content

fix: idle-timeout for agent-loop engine calls, infra-failure escalation - #104

Merged
Santisoutoo merged 1 commit into
mainfrom
fix/agent-loop-idle-timeout
Aug 25, 2026
Merged

fix: idle-timeout for agent-loop engine calls, infra-failure escalation#104
Santisoutoo merged 1 commit into
mainfrom
fix/agent-loop-idle-timeout

Conversation

@Santisoutoo

Copy link
Copy Markdown
Owner

Summary

  • run-engine.sh: replace the flat 15min wall-clock timeout (PR fix: bound engine calls with a per-call timeout in run-engine.sh #102) with an idle-timeout — a call is only killed after N seconds with zero new stdout/stderr, with a much larger hard ceiling as backstop. Fixes today's incident where the Implement step was killed mid-exploration on issue [data-fidelity] Sprint 1 — metadata trust: status enum, 4KB frontmatter, article identity #52 while still actively producing output.
  • Per-step budgets: Implement idle=6min/ceiling=40min (needs exploration room), Review idle=3min/ceiling=15min (this is where the real 84min zero-output hang happened, issue [frontend-perf] Sprint 4 — bundle hygiene: mock out of entry, lazy wizard, drop dead dep #85), fix-round idle=5min/ceiling=20min.
  • Fix a misclassification: a DONE-but-empty-diff guard failure (known opencode silent-exit bug, anomalyco/opencode#28605) was wrongly counted as a real attempt instead of the infra failure it is.
  • Infra failures (engine crash, timeout, no result marker) now get their own counter and escalate to a new agent:infra-stuck label after 3 — previously they could retry forever, 3x/day, with no escalation.
  • Every failure path now writes a short reason into the issue comment (implement-idle-timeout, verify-backend, review-fix-exhausted, ...) instead of a generic message — no more opening the Actions log to find out which stage broke.
  • Verify — landing now installs Chromium before building, matching ci.yml's landing-build job.
  • New Verify — pip-audit (CVE gate) step: confirmed via gh api .../branches/main/protection that pip-audit isn't a required branch-protection context, and agent-loop never checked it either — a CVE-flagged dependency could previously auto-merge unblocked. Verified main is currently clean (uv run --with pip-audit pip-audit --skip-editable → "No known vulnerabilities found") before wiring this in as blocking.
  • pick-issue.sh: excludes the new agent:infra-stuck label from future picks. README.md and scripts/setup-github.sh updated to match (label already created on the repo).

Validation done locally

  • bash -n on all three shell scripts, YAML-parsed the workflow (js-yaml) — all clean.
  • Standalone smoke test of the idle-detection loop shape: a simulated hang (output stops, then 60s of silence) gets killed after the idle threshold (~7s in test config), not the full hang duration. A simulated slow-but-steady process (output every 2s) runs to completion untouched. Confirms the core fix for today's false-positive.
  • pip-audit run locally against main: no known vulnerabilities.

Known residual risk (documented, not blocking)

run-engine.sh's idle-kill uses pkill -TERM -P "$runner_pid", which only reaps direct children, not grandchildren — if opencode/cursor-agent spawn multi-level subprocess trees, an idle-kill could theoretically leave an orphan running. Plan is to validate via a workflow_dispatch test run before trusting the 3x/day cron on this, per the README's own existing practice for engine/model changes, and watch the job log for orphan processes.

Test plan

Claude-Session: https://claude.ai/code/session_01KpuHkuMadFm8SGgw7nuZki

… infra-failure escalation

The flat 15min wall-clock timeout added yesterday (PR #102) killed the
Implement step mid-exploration on a wide-scope issue (#52) that was
actively producing output, not hung. It couldn't tell "genuinely stuck"
(the 84min zero-output hang on issue #85 that motivated PR #102) from
"slow but working". run-engine.sh now kills a call only after N seconds
with NO NEW output, with a much larger hard ceiling as backstop; Implement
gets a generous idle/ceiling budget, Review (where the real hang happened)
stays tight, and the one-round fix-call gets its own inline override.

Also, while auditing the loop for other robustness gaps:

- Infra failures (engine crash, timeout, no result marker) never had a
  counter or escalation path — an issue that reliably infra-fails would
  retry forever, 3x/day, with no human ever finding out. They now get
  their own counter and escalate to a new `agent:infra-stuck` label after
  3, excluded from future picks like `agent:blocked` already is.
- A DONE-but-empty-diff guard failure (known opencode silent-exit bug,
  anomalyco/opencode#28605) was writing `result=DONE` before the guard
  could invalidate it, so Cleanup miscounted it as a real attempt instead
  of the infra failure it actually is.
- Every failure path now writes a short reason (implement-idle-timeout,
  verify-backend, review-fix-exhausted, ...) into the issue comment
  instead of a generic "attempt N failed" — no more opening the Actions
  log just to see which stage broke.
- Verify — landing now installs Chromium before building, matching
  ci.yml's landing-build job (agent-loop's Verify could previously diverge
  from what real CI would say).
- Added a Verify — pip-audit (CVE gate) step: confirmed via branch
  protection that pip-audit isn't a required context and agent-loop never
  checked it either, so a CVE-flagged dependency bump could previously
  auto-merge unblocked. Verified main is currently clean before wiring
  this in as a blocking step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KpuHkuMadFm8SGgw7nuZki
@github-actions github-actions Bot added area: codebase Touches src/, scripts/ or main.py area: ci-cd GitHub Actions / workflows / dependabot area: docs Documentation only labels Aug 25, 2026
@Santisoutoo
Santisoutoo merged commit bc4052d into main Aug 25, 2026
15 of 16 checks passed
@Santisoutoo
Santisoutoo deleted the fix/agent-loop-idle-timeout branch August 25, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci-cd GitHub Actions / workflows / dependabot area: codebase Touches src/, scripts/ or main.py area: docs Documentation only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant