Skip to content

[compass] Fix: the "execution jobs" quick check fails on a slow machine at its first job ("exit code or streams missing") - #25

Merged
skulitom merged 2 commits into
mainfrom
compass/anode-fix-2026-10-07-jobs-check-slow-start
Oct 8, 2026
Merged

skulitom merged 2 commits into
mainfrom
compass/anode-fix-2026-10-07-jobs-check-slow-start

Conversation

@skulitom

@skulitom skulitom commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Report

B-024 in Compass's ANODE-BUGS.md (filed by compass-merge on 2026-10-07): the CI run of main at 2dbf078 (run 37551098000, attempt 1) failed one quick check, [FAIL] execution jobs InvalidOperationException: exit code or streams missing, with the job summary running, exit code pending after about 10 s. The same tree passed locally, and CI passed on the next two main commits.

Root cause

DevelopmentChecks.Jobs starts each worker, a Windows PowerShell 5.1 process, with waitMs: 10000 (the most seat_exec and seat_job allow) and required the job to be finished in that first reply. A cold windows-latest runner can take longer than 10 s to start Windows PowerShell, so the job was still running when the check judged it. Three steps had this shape: the exit-code job, the UTF-8 job and the recovered job. #17 fixed the same family (B-012) for the descendant-PID step only.

Fix

A local ReadUntilFinished helper reads the job again, 10 s at a time, until it reports finished, for up to 30 s, and the three steps judge the result only then. A job that never finishes still fails with its whole reply, as before. Reading without a cursor returns the output from the start, so the stdout/stderr equality checks are unchanged. Test code only; no product change, so no CHANGELOG entry (as with #17).

How it was tested

  • FOCUS test command (scripts\build.ps1 -QuickTest -OutputDirectory artifacts\pkg-build): pass, 93/93, docs consistent. execution jobs took 3.8 s.
  • Slow-start experiments (Debug build, selftest --quick, file restored byte for byte after each), delaying a worker's start by 11 s with [Threading.Thread]::Sleep(11000):
    Experiment Result
    first worker 11 s late, this fix ok (check 14.2 s)
    first worker 11 s late, old single wait FAIL exit code or streams missing (the CI message)
    recovered worker 11 s late, this fix ok (14.2 s)
    recovered worker 11 s late, old single wait FAIL client cancellation killed an already started command
  • The experiments ran on the first commit (c85acad); the second commit only renames the helper and raises the PID step's budget.
  • A first try used Start-Sleep, which failed even with the fix: Windows PowerShell started with -EncodedCommand writes Start-Sleep's progress record to stderr as CLIXML, so stderr wasn't problem. That is the experiment's artefact, not this check's; the experiments above use a delay that writes nothing.

Second review

A fresh read-only Claude subagent (Codex is at its usage limit until Fri 10-09) found no P1 or P2: the loop always ends (about 40 s at worst), the wait stays inside seat_job's 0 to 10000 range, re-reading neither loses nor repeats output, agentId is always the job's owner, a failed or killed job still fails the check, and no other quick check uses a PowerShell job this way. Its P3s:

  • the descendant-PID step allowed 15 s against these steps' 30 s: raised to 30 s, so the check is consistent;
  • the helper's name read like a yes/no test: renamed ReadUntilFinished;
  • ExecutionJobs.ReadAsync reads the output before finished, so a job that ends in between can be reported finished without the end of its output. That is product code with a narrow window, so it's filed as B-025 for its own PR rather than folded into this test-only one.

Manual check in a seat

None: this changes a quick check only.

🤖 Generated with Claude Code

skulitom and others added 2 commits October 7, 2026 23:19
… one 10 s wait (B-024)

The "execution jobs" check started Windows PowerShell workers with waitMs 10000 and required the
job to be finished in that first reply. On a cold runner PowerShell can take longer than 10 s to
start, which failed CI on main at 2dbf078 with "exit code or streams missing" and the summary
"running, exit code pending". The three steps that wait for a worker to finish (the exit-code
job, the UTF-8 job and the recovered job) now read it again, 10 s at a time, for up to 30 s, and
judge the result only then. A job that never finishes still fails with its summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p 30 s too (B-024 review)

The second review's P3s: the descendant-PID step allowed 15 s against the other steps' 30 s, and
the helper's name read like a yes/no test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T22:35:22.806813Z 04b2d64 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@skulitom

skulitom commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Compass review: ready to merge

Tested by compass-merge on 2026-10-08 on head 04b2d64, with the current main merged in locally (5539834, after tonight's merges of #23 and #24; local merge tree 850c3a3, never pushed).

  • scripts\build.ps1 -QuickTest: Release build, 95/95 quick checks ("execution jobs" ok in 3.9 s), documentation links, anchors and tool names consistent.
  • CI: build green on 04b2d64.
  • Review of the whole diff (one file, src/Anode/Cli/DevelopmentChecks.cs): ReadUntilFinished re-reads with the reply's own jobId and agentId (both are in every ExecutionJobs.ReadAsync reply), is bounded at 30 s with each wait clamped to 1–10 s, and only adds waiting at the three steps that need a finished job; the PID step's limit goes from 15 s to 30 s. No product code changes, and no check is weakened: every assertion is the same.
  • Second opinion: the Codex CLI is at its usage limit until Fri 10-09 22:15, so no codex-review.sh run. The fixer's read-only Claude subagent reviewed it (no P1/P2), the fixer's slow-start experiment fails the old code and passes the new, and the Codex GitHub review of this head finished with no findings.

@skulitom
skulitom merged commit 0d637d0 into main Oct 8, 2026
1 check passed
@skulitom
skulitom deleted the compass/anode-fix-2026-10-07-jobs-check-slow-start branch October 8, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant