Skip to content

fix(ci): cancel a destroyed merge group's queued Check run - #6105

Merged
kixelated merged 5 commits into
mainfrom
fix/queue-stale-runs
Oct 21, 2026
Merged

kixelated merged 5 commits into
mainfrom
fix/queue-stale-runs

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

The merge queue stalled for hours. GitHub never cancels a merge group's workflow run when it destroys the group. That happens whenever an entry ahead leaves the queue and the groups behind it are rebuilt on new gh-readonly-queue/main/pr-<N>-<base> refs. Each ref is unique, so check-${{ github.ref }} never matches the old run. All queue jobs run on one self-hosted runner, so the dead runs stay queued ahead of the live groups.

Example: #6082's group was created at 21:56Z, but its Check job only started at 22:58Z (run). That is past the 60-minute check_response_timeout_minutes, so the queue removed it with checks_timed_out even though the run passed. Each removal rebuilds the groups behind it and leaves another set of stale runs, so the queue livelocks. At 23:13Z five 22:59Z runs were again queued for branches GitHub had already deleted.

Approach

Also trigger Check on the merge_group destroyed activity. Actions delivers it, although its trigger docs list only checks_requested. The webhook docs do define it. The destroyed run uses the same queue ref, so the existing check-${{ github.ref }} concurrency group cancels the stale run. The job then skips itself, like the closed PR run already does, so it never needs a runner.

  • Concurrency key, PR-side behavior, permissions, and jobs are unchanged.
  • A live group can only be cancelled by an event on its own ref. Another PR's group has a different ref.
  • When a group merges, destroyed also fires (reason merged), but the run is already complete, so nothing is cancelled. The only effect is a skipped Check on main's new HEAD SHA.
  • .github/actionlint.yaml ignores that one message for check.yml only, until rhysd/actionlint#726 lands.
  • sh/gh/queue.test.ts and sh/gh/setup.test.ts now cover the trigger, the skip, and the shared concurrency key.

Prior art: chapel-lang/chapel#29274 uses the same pattern. Its merge_group runs show each rebuilt group's old run going cancelled within about two minutes of a skipped destroyed run on the same branch. See also the community discussion.

Built on #6086, which is set to land first. Once it merges, this diff reduces to the merge_group trigger, the job if, the actionlint config, and the tests.

Impact

  • No public API or wire changes. CI only.

Alternatives

  • One concurrency key per PR across its groups: expressions cannot extract the PR number from the ref. merge_group.head_commit.message works with squash titles, but group names are capped at 400 characters. It is also fragile when a title is edited, and it only cancels if the PR is re-enqueued, not when the PR leaves the queue.
  • Early-exit step on the self-hosted runner (skip when the queue branch is gone): each stale job still waits its turn and holds the runner for its setup. It also needs if: on every step, or actions: write to cancel itself.
  • API reaper job (cancel incomplete runs for the destroyed SHA): needs actions: write and more code, and was reported to cancel unrelated push runs on the same SHA.
  • One global queue concurrency group: a new pending run replaces the older pending one, which would cancel live groups.

Follow-ups

  • Ruleset recommendation, not changed here: with one queue runner, groups build one job at a time. max_entries_to_build 5 makes the last group wait behind four others, which can exceed the 60-minute timeout even without stale runs. Speculative builds add no throughput on one runner and are wasted on any failure. Set max_entries_to_build to the runner count (1 today), or raise check_response_timeout_minutes to cover the queue depth.
  • bun sh/gh/queue.test.ts, which runs nightly, already fails on main: smoke.yml must skip main PRs, since #6046 moved smoke into the repo. This PR does not touch that.

Validation: actionlint and just gh check pass, and the bun test sh/gh/setup.test.ts tests pass. bun sh/gh/queue.test.ts passes its new assertions before reaching the pre-existing smoke.yml failure. just check passed everything except bench/relay.test.py, which timed out at 15s under load. Run alone, it passes in 1.2s.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 4 commits October 20, 2026 12:19
Co-Authored-By: GPT-6.1 <noreply@openai.com>
Keep one Check job and one dispatch; move feature permutations and optimized WASM builds to nightly and release PRs. Use dev-profile binding checks while preserving optimized Dart benchmarks.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub never cancels a merge group's workflow run when the group is
destroyed, for example when an entry ahead leaves and the groups behind it
are rebuilt on new refs. On the single queue runner those stale runs stay
queued ahead of the live groups, which then pass the 60-minute check
timeout and get removed, rebuilding the groups behind them again.

Trigger Check on the merge_group `destroyed` activity too. Its run shares
the group's queue ref, so the existing `check-${{ github.ref }}`
concurrency group cancels the stale run, and the job skips itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T01:04:11.311034Z b938a29 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a67e242e-6da3-4d83-82ac-1a1f7c1c64a9
📥 Commits

Reviewing files that changed from the base of the PR and between 10623a8 and b938a29.

📒 Files selected for processing: 4
  • .github/actionlint.yaml
  • .github/workflows/check.yml
  • sh/gh/queue.test.ts
  • sh/gh/setup.test.ts

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 16e887d8-d34c-4eb6-9cde-ec3729791125
📥 Commits

Reviewing files that changed from the base of the PR and between eecb86c and 10623a8.

📒 Files selected for processing: 34
  • .github/actionlint.yaml
  • .github/justfile
  • .github/workflows/check.yml
  • .github/workflows/features.yml
  • .github/workflows/nightly.yml
  • .github/workflows/platform.yml
  • dart/README.md
  • dart/justfile
  • dart/moq_ffi/hook/build.dart
  • js/justfile
  • justfile
  • nix/overlay.nix
  • quest/m0/ci/README.md
  • quest/m0/ci/audit.md
  • quest/m0/ci/mbx-r2-verify.md
  • quest/m0/ci/one-pass.md
  • quest/m0/ci/release-gates.md
  • quest/m0/ci/runners.md
  • sh/dart/bench.sh
  • sh/dart/build.test.py
  • sh/dart/check.sh
  • sh/dart/generate.sh
  • sh/dispatch-query.test.py
  • sh/dispatch.sh
  • sh/gh/queue.test.ts
  • sh/gh/setup.test.ts
  • sh/go/stage.sh
  • sh/js/README.md
  • sh/js/workspace.py
  • sh/js/workspace.test.py
  • sh/kt/generate.sh
  • sh/rs/select.sh
  • sh/rs/select.test.py
  • sh/swift/check.sh
💤 Files with no reviewable changes: 3
  • quest/m0/ci/README.md
  • quest/m0/ci/runners.md
  • quest/m0/ci/one-pass.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

Walkthrough

The pull request combines CI lint and test dispatch in one Check job, adds a reusable workflow for Rust feature and JavaScript/WebAssembly checks, and removes feature permutations from Rust change selection. It also adds configurable WebAssembly profiles and changes Dart checkout builds to use the development profile while benchmarks use release-profile bindings. Related workflow tests and CI documentation are updated.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 10623

The change appears mergeable after normal checks. Its queue-cancellation behavior is supported by observed GitHub Actions behavior, though that behavior is not documented as a supported workflow trigger.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 3.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 31 functions across 15 files. (16 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly describes the main merge-queue fix: canceling queued Check runs when GitHub destroys a merge group.
Description check Passed The description directly explains the stale merge-group run problem, the destroyed-event solution, affected files, tests, validation, and known unrelated failures.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 3.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 31 functions across 15 files. (16 skipped: 16 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kixelated
kixelated added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit f2c27fa Oct 21, 2026
9 checks passed
@kixelated
kixelated deleted the fix/queue-stale-runs branch October 21, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant