Stop waiting for a pre-build run when one of its jobs fails - #122
Merged
Conversation
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as off-topic.
This comment was marked as off-topic.
sscarduzio
changed the title
sscarduzio
force-pushed
the
ci/prebuild-fail-fast
branch
from
September 10, 2026 07:23
e464424 to
c5468ab
Compare
This was referenced Sep 10, 2026
coutoPL
requested changes
Sep 10, 2026
_prebuild_run_state polled the RUN's status and conclusion, so a run whose only meaningful job had already failed still read as "running" until the whole run wrapped up. The waiter holds a paid runner while it polls - three ubicloud-standard-8 legs per readonlyrest_kbn pipeline run - so that time is billed for nothing. Neither publish-pre-builds workflow uses continue-on-error (checked in both repos: the ES one has a single `publish` job, the KBN one has prepare-docker-env and publish-pre-builds), so the first job to end in failure, cancelled or timed_out decides the run. Report that conclusion at once and let the existing terminal branch stop the wait. The error now names the job that decided it and how long we waited, instead of saying the run "finished with failure" when it had not finished at all. `skipped` is deliberately not terminal - a skipped job is normal and says nothing about the run's outcome. Verified the jq against every run shape, including real API output from run 32949023073: running, no jobs done -> running running, one job FAILED -> failure (early bail) running, one job cancelled -> cancelled (early bail) running, a job merely skipped -> running queued, no jobs listed yet -> running completed success -> success completed failure -> failure completed, conclusion missing -> unknown This does not help the case where the run sits QUEUED with no jobs started - nothing has failed yet there. That was the 78-minute wait on 2026-09-04, and sscarduzio/elasticsearch-readonlyrest-plugin#1372 is what fixes it, by taking the workflow off a two-slot pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sscarduzio
force-pushed
the
ci/prebuild-fail-fast
branch
from
September 10, 2026 09:22
c5468ab to
4a6b867
Compare
docs/dev/code-style.md asks a comment to explain the current state, not the change or the event that led to it. The comment carried the date and the cost of the run that showed the problem, plus the runner sizes of another repository's pipeline. That is the PR description's job, and the numbers are there now. What stays is the constraint a reader cannot see from the code: the pre-build workflows do not set continue-on-error, so the first job that ends badly decides the run, and the waiter pays for the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The poller read only the run status. A run with a failed job still read as
runninguntil the whole run finished, so the waiter kept polling something that could no longer succeed.That waiter holds a paid runner. A
readonlyrest_kbnpipeline run has threeubicloud-standard-8legs waiting on the same pre-build. On 4 September that cost 78 minutes of idling for a pre-build that had already lost a job.The change
Neither pre-build workflow sets
continue-on-error, so the first job that ends infailure,cancelledortimed_outdecides the run. The poller now asks forjobsas well asstatus, reports that job's conclusion at once, and names the job in the error:Limit
This does not help a run that is still queued — nothing has failed there yet. sscarduzio/elasticsearch-readonlyrest-plugin#1372 covers the queue.
Review notes
master.docs/dev/branching.mdin [RORDEV-2220] add branching, code style and writing style docs #125, case 4: a pipeline, workflow or runner change targetsmaster. Rebased ontomaster, so this PR carries only its own commit.docs/dev/code-style.mdasks a comment to explain the current state, not the event that produced it. The date, the idle cost and the other repository's runner sizes are above instead. The comment keeps only the constraint a reader cannot see from the code: the pre-build workflows do not setcontinue-on-error, so the first job that ends badly decides the run, and the waiter pays for the rest.🤖 Generated with Claude Code