Evict failing merge-queue entries ~35 min sooner

[Mikola Lysenko (mikolalysenko)]

Problem

ci-ok (the merge queue's required check) needs: every CI job, so a queue entry whose CI is already doomed only reports failure after the slowest leg ends. Today's queue shows the cost:

• PR Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009 entered the queue at 15:13Z. Its group run 37799050567 hit a compile error at about 15:16Z: Make the vendored-to-hosted takeover atomic #1039 dropped an argument from hosted::engine::rewrite, and Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009 adds a test that still passes it (error[E0061]). coverage, test (macos/windows) and test-release were all red by 15:20Z. ci-ok still sat pending behind the Gradle e2e legs for another ~40 min.
• The three entries stacked on it (Remove the SOCKET_FORCE env binding; --force is flag-only (#615) #1021 37799056730, Use tests/common's binary() and git_sha256 in CLI tests (#824) #1124 37799132112, Label the setup-php pin in ci.yml with its real tag #1118 37799140116) contain Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009, so they failed the same four jobs by about 15:20Z. Use tests/common's binary() and git_sha256 in CLI tests (#824) #1124's ci-ok only reported at 15:51Z, ~35 min after its first failure. The queue can't evict and rebuild until then, and each of these runs (~190 jobs, macOS and Windows included) keeps burning runners.

Last 60 merge-group CI runs (2026-10-07 22:07Z to 2026-10-08 15:33Z): 12 failed runs got a `ci-ok` verdict, and 5 more were still pending at the time of writing. In those 12, the first failed job finished a median of ~2 min into the run: `clippy` at 0.6–1.7 min, or the compile error in `coverage`/`test`/`test-release` at ~1.7 min. `ci-ok` then took another 31–43 min (434 min total, ~36 min per run) to report:

run	entry	first failure (min)	verdict (min)	wasted (min)	failed jobs
37741022380	#1057	0.6	44.0	43.4	clippy
37738827925	#1050	1.7	42.5	40.9	clippy, e2e
37739124591	#1029	1.7	38.7	37.0	clippy
37730827414	#724	2.1	44.5	42.4	coverage, test, test-release
37730570198	#1038	1.7	41.9	40.2	coverage, test, test-release
37729900304	#1083	1.7	40.0	38.2	coverage, e2e, test, test-release
37730608490	#1035	1.6	36.2	34.6	coverage, test, test-release
37713148083	#1046	11.2	53.8	42.6	coverage, test, test-release
37713149707	#1003	14.8	52.0	37.2	coverage, test, test-release
37759566368	#768	11.2	42.6	31.4	e2e
37726258998	#724	18.0	49.0	31.0	e2e
37799137861	#1121	3.1	18.2	15.1	coverage, test, test-release

These runs also ran ~490 macOS job-minutes, counting every macOS job that finished after the run's first failure.

Root cause

A job with needs: can't start until all of its needs finish, and the Gradle e2e legs take 28–35 min. Nothing in the run stops the other jobs once a failure makes ci-ok certain to fail.

Fix

A new merge_group-only workflow, merge-queue-fail-fast.yml, runs scripts/merge-queue-fail-fast.py. The script polls the entry's CI run every 30 s and cancels it as soon as any job concludes failure or timed_out.

• Safe: no CI job sets continue-on-error, so a failed job already means ci-ok fails. Cancelling only changes when it fails.
• The verdict still arrives: ci-ok is if: always(), so it runs on a cancelled run and fails within seconds. This repo's own cancelled PR runs show it: 37800029383, 37799319868 and 37782495575 each got a ci-ok failure 3–7 s after the cancel.
• Guarded: scripts/tests/test_merge_queue_fail_fast.py pins both of those ci.yml properties. It fails if anyone adds a job-level continue-on-error or drops if: always() from ci-ok.
• Never in the way: it isn't a required check and never fails its job. An API error is a warning, and the run then finishes as before.
• cancelled jobs don't trigger a cancel, because someone (the queue, a person, Cut merge-group CI from ~46 to ~20 min: shard Gradle e2e and test legs, skip test-release in queue, cancel orphaned runs #1133's orphan janitor) is already cancelling.

No ci.yml change, so this doesn't conflict with #1133 or #1139 and keeps every required-check name. All tests still run exactly where they did before. The only difference is that a doomed merge-group run stops early.

Proof

• Dry run against the live doomed run (POST stubbed out): it found 37799050567 by the group's head SHA and would have cancelled it for coverage, test (macos-latest), test (windows-latest) and test-release.
• python3 -B -m unittest discover -s scripts/tests: 259 tests OK, 5 of them new (this suite runs in lint-ecosystems).
• actionlint clean; zizmor reports no findings.

Expected effect: a failing entry leaves the queue ~35 min sooner (about 7 queue-hours saved in the window above), and the entries behind it rebuild that much sooner. Each failed group run stops using macOS and Windows runners minutes after its first failure instead of running all ~190 jobs. The cost is one ubuntu job per queue entry that polls for the length of the run.

Note: like any merge_group workflow, this takes effect only after it lands on main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CtHdJ7b6ncYXfs8LSJMsPy

---

Note

Medium Risk
Introduces automated workflow-run cancellation with actions:write, gated to merge queue and base-branch script; mis-detection could cancel runs that could still pass if CI invariants change without updating the contract tests.

Overview
Adds a merge-queue-only workflow that watches each entry’s ci.yml run and cancels it as soon as any job finishes with failure or timed_out, so the required ci-ok gate (which runs if: always()) fails in seconds instead of waiting on slow e2e legs.

The watcher checks out base_sha only (sparse scripts/merge-queue-fail-fast.py) so queued PR code never runs with an actions:write token; if the script isn’t on main yet, the job exits cleanly. The Python poller finds the merge-group run by head SHA, posts cancel via the Actions API, logs to the step summary, and never fails the workflow on API errors.

scripts/tests/test_merge_queue_fail_fast.py covers fatal_jobs logic and asserts ci.yml keeps no job-level continue-on-error and ci-ok keeps if: always(), so early cancel stays safe.

^{Reviewed by Cursor Bugbot for commit 66aa864. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 66aa864. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review at 66aa864.

• CI: 189/189 non-skipped checks green (ci-ok success).
• Bugbot: the MEDIUM security finding (the actions:write watch job ran the queued PR's copy of scripts/merge-queue-fail-fast.py) is fixed in 66aa864. The canceller is now checked out at merge_group.base_sha, and the step skips while the script is not on base yet. Bugbot re-reviewed 66aa864 with no findings.
• Reviewer note: on this PR's own queue entry the watcher is a no-op, because base has no script yet. It starts working from the next queue entry after this lands.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Tanmay Singla (@Tanmay182003) — final reviewer: not enqueueing yet. One non-merge commit landed after your approval at 5bf4852:

• 66aa864 Run the merge-queue canceller from the base branch, not the queue head — fixes Bugbot's MEDIUM security finding: the actions:write watch job in .github/workflows/merge-queue-fail-fast.yml now checks out github.event.merge_group.base_sha before running scripts/merge-queue-fail-fast.py, so a queued PR can't run its own copy of the script with that token.

CI is green on 66aa864 (ci-ok, clippy). Could you take another look at the workflow diff since your approval? Once you re-approve, it can go straight to the merge queue.

---

Generated by Claude Code