Stop nightly scheduled runs from cancelling in-flight push CI on main - #21691
Open
shoumikhin wants to merge 1 commit into
Open
Stop nightly scheduled runs from cancelling in-flight push CI on main#21691shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21691
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit a0cade3 with merge base 9cd0c12 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
shoumikhin
force-pushed
the
shoumikhin/concurrency-schedule-discriminator
branch
from
August 8, 2026 23:02
e748c7e to
abe17c5
Compare
This PR needs a
|
shoumikhin
force-pushed
the
shoumikhin/concurrency-schedule-discriminator
branch
from
August 8, 2026 23:33
abe17c5 to
5ead287
Compare
shoumikhin
force-pushed
the
shoumikhin/concurrency-schedule-discriminator
branch
from
August 9, 2026 02:30
5ead287 to
b56ba24
Compare
shoumikhin
force-pushed
the
shoumikhin/concurrency-schedule-discriminator
branch
from
August 10, 2026 05:51
b56ba24 to
a67917f
Compare
shoumikhin
force-pushed
the
shoumikhin/concurrency-schedule-discriminator
branch
from
August 10, 2026 18:08
a67917f to
a0cade3
Compare
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is broken
Every so often a commit on main shows a cancelled ARM or CoreML job on HUD even though nobody cancelled anything and the commit is fine. It looks like a real failure, so the oncall investigates a phantom.
Why it is broken
GitHub Actions cancels an older run when a newer run lands in the same concurrency group. These workflows use a group key built like this:
The key already tells apart pull requests, branches, commits and manual runs. It does not tell apart a scheduled run.
These same workflows also have a nightly
scheduletrigger. When the nightly cron fires,github.ref_nameismainandgithub.shais the current head of main, which are exactly the same values the push run for that commit used. Same key,cancel-in-progress: true, so the cron run kills the push run that was still going.The result is a commit whose CI reads as cancelled with no explanation.
The fix
Add
-${{ github.event_name == 'schedule' }}to the concurrency group, so a scheduled run and a push run on the same commit sit in different groups and no longer cancel each other.This is the same pattern already used by other workflows in this repo. It is applied here to every workflow that has both a
scheduleand apushtrigger:test-backend-arm.ymltest-backend-coreml.ymltest-backend-openvino.ymltest-backend-qnn.ymltest-backend-vulkan.ymltest-backend-webgpu.ymltest-backend-xnnpack.ymltest-webgpu-native.ymlbuild-cmsis-pack.ymldocker-builds.ymldoc-build.ymlalso has both triggers but is deliberately left alone: it setscancel-in-progress: ${{ github.event_name == 'pull_request' }}, so outside a pull request it never cancels anything and the collision cannot happen there.This only separates schedule from push. It does not change how two scheduled runs
of the same workflow behave:
github.shais still in the key, so two scheduledruns share a group only while
mainhas not moved between them, and that isunchanged by this PR.
How this was verified
Every workflow under
.github/workflowswith aschedule:trigger was listedtogether with its concurrency group, to make sure the change is both correct and
complete:
apple.yml,periodic.ymlandriscv64.ymlalready carry exactly the-${{ github.event_name == 'schedule' }}suffix this PR adds, so this is not anew idea, it is existing practice in this repo.
periodic.ymlgoes one stepfurther and also appends
${{ github.event.schedule }}to separate itsindividual crons.
that also have a
pushtrigger, so none were missed.build-cadence-runner.ymlandtest-pico2-build.ymlalready put${{ github.event_name }}in the key, which covers the same collision.nightly.ymlhas both triggers but only pushes onciflow/nightly/*tags, soits
github.ref_nameis a tag and can never match the scheduled run onmain.doc-build.ymlis the case described above.concurrencyblock at all.Note that the fix only takes effect once it is on
main, because a scheduled runalways uses the workflow file from the default branch.
CI on this pull request: 213 checks reported, 195 success, 15 skipped, 2
cancelled, 0 failures. Both cancellations were manual, and in one of them every
step had already finished successfully before the cancel landed.
Overlap with other pull requests
Two of the files here are also touched by other open pull requests, in both cases
far away from the
concurrency:block this PR edits:.github/workflows/test-webgpu-native.ymlalso gets atimeout:change inFix test-webgpu-native: export the missing rope fixture, raise the job timeout #21690, in the job block further down.
.github/workflows/test-backend-coreml.ymlalso gets a matrixexclude:entryin Drop the coreml_static_int8 operators job, which has never once finished #21695, likewise further down.
The hunks do not touch and the changes merge cleanly in any order.