Skip to content

Stop nightly scheduled runs from cancelling in-flight push CI on main - #21691

Open
shoumikhin wants to merge 1 commit into
mainfrom
shoumikhin/concurrency-schedule-discriminator
Open

Stop nightly scheduled runs from cancelling in-flight push CI on main#21691
shoumikhin wants to merge 1 commit into
mainfrom
shoumikhin/concurrency-schedule-discriminator

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What is broken

Every so often a commit on main shows a cancelled ARM or CoreML job on HUD even though nobody cancelled anything and the commit is fine. It looks like a real failure, so the oncall investigates a phantom.

Why it is broken

GitHub Actions cancels an older run when a newer run lands in the same concurrency group. These workflows use a group key built like this:

group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true

The key already tells apart pull requests, branches, commits and manual runs. It does not tell apart a scheduled run.

These same workflows also have a nightly schedule trigger. When the nightly cron fires, github.ref_name is main and github.sha is the current head of main, which are exactly the same values the push run for that commit used. Same key, cancel-in-progress: true, so the cron run kills the push run that was still going.

The result is a commit whose CI reads as cancelled with no explanation.

The fix

Add -${{ github.event_name == 'schedule' }} to the concurrency group, so a scheduled run and a push run on the same commit sit in different groups and no longer cancel each other.

This is the same pattern already used by other workflows in this repo. It is applied here to every workflow that has both a schedule and a push trigger:

  • test-backend-arm.yml
  • test-backend-coreml.yml
  • test-backend-openvino.yml
  • test-backend-qnn.yml
  • test-backend-vulkan.yml
  • test-backend-webgpu.yml
  • test-backend-xnnpack.yml
  • test-webgpu-native.yml
  • build-cmsis-pack.yml
  • docker-builds.yml

doc-build.yml also has both triggers but is deliberately left alone: it sets cancel-in-progress: ${{ github.event_name == 'pull_request' }}, so outside a pull request it never cancels anything and the collision cannot happen there.

This only separates schedule from push. It does not change how two scheduled runs
of the same workflow behave: github.sha is still in the key, so two scheduled
runs share a group only while main has not moved between them, and that is
unchanged by this PR.

How this was verified

Every workflow under .github/workflows with a schedule: trigger was listed
together with its concurrency group, to make sure the change is both correct and
complete:

  • apple.yml, periodic.yml and riscv64.yml already carry exactly the
    -${{ github.event_name == 'schedule' }} suffix this PR adds, so this is not a
    new idea, it is existing practice in this repo. periodic.yml goes one step
    further and also appends ${{ github.event.schedule }} to separate its
    individual crons.
  • The 10 workflows changed here are precisely the ones still on the older key
    that also have a push trigger, so none were missed.
  • build-cadence-runner.yml and test-pico2-build.yml already put
    ${{ github.event_name }} in the key, which covers the same collision.
  • nightly.yml has both triggers but only pushes on ciflow/nightly/* tags, so
    its github.ref_name is a tag and can never match the scheduled run on main.
  • doc-build.yml is the case described above.
  • The other scheduled workflows have no concurrency block at all.

Note that the fix only takes effect once it is on main, because a scheduled run
always uses the workflow file from the default branch.

CI on this pull request: 213 checks reported, 195 success, 15 skipped, 2
cancelled, 0 failures. Both cancellations were manual, and in one of them every
step had already finished successfully before the cancel landed.

Overlap with other pull requests

Two of the files here are also touched by other open pull requests, in both cases
far away from the concurrency: block this PR edits:

The hunks do not touch and the changes merge cleanly in any order.

Copilot AI lite review requested due to automatic review settings August 8, 2026 23:01
@pytorch-bot

pytorch-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21691

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a0cade3 with merge base 9cd0c12 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 8, 2026 23:02
@shoumikhin
shoumikhin force-pushed the shoumikhin/concurrency-schedule-discriminator branch from e748c7e to abe17c5 Compare August 8, 2026 23:02
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 8, 2026 23:33
@shoumikhin
shoumikhin force-pushed the shoumikhin/concurrency-schedule-discriminator branch from abe17c5 to 5ead287 Compare August 8, 2026 23:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 9, 2026 02:30
@shoumikhin
shoumikhin force-pushed the shoumikhin/concurrency-schedule-discriminator branch from 5ead287 to b56ba24 Compare August 9, 2026 02:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 10, 2026 05:51
@shoumikhin
shoumikhin force-pushed the shoumikhin/concurrency-schedule-discriminator branch from b56ba24 to a67917f Compare August 10, 2026 05:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 10, 2026 18:08
@shoumikhin
shoumikhin force-pushed the shoumikhin/concurrency-schedule-discriminator branch from a67917f to a0cade3 Compare August 10, 2026 18:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants