Skip to content

fix(checks): bound each download-heavy step with timeout-minutes - #577

Merged
kyle-sexton merged 2 commits into
mainfrom
ci/checks-step-timeouts
Sep 8, 2026
Merged

fix(checks): bound each download-heavy step with timeout-minutes#577
kyle-sexton merged 2 commits into
mainfrom
ci/checks-step-timeouts

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

No related issue: runner-host download stall consumed the full checks budget on 2026-09-08 (dotfiles run 34190855785)

Summary

On 2026-09-08 the self-hosted runner host's network path collapsed to roughly
12 KB/s for about fifteen minutes. dotfiles' checks / checks job — which
calls this repository's checks.yml reusable at v0.22.2 with
timeout-minutes: 15 — stalled inside the check-jsonschema composite, on
astral-sh/setup-uv's download. No step had its own budget, so the stall
consumed the whole job budget and the run ended cancelled.

cancelled is the worst possible verdict for this lane. The outcome join is
gated on !cancelled(), so it never ran and named nothing; a reusable-workflow
job that does not complete publishes no outputs; and the caller's ci-status
aggregate read the lane as a failure with no cause attached. Three reruns were
needed to get green. A healthy run of that job takes 1 to 4.5 minutes.

Fix

Every step in the checks job that downloads something now carries
timeout-minutes: 8 — comfortably above the 4.5-minute healthy ceiling for the
whole job, well under the 15-minute default job budget:

Step What it downloads
typos crate-ci/typos release tarball behind actions/cache
gitleaks gitleaks/gitleaks release tarball behind actions/cache
editorconfig editorconfig-checker release tarball behind actions/cache
markdown actions/setup-node, then markdownlint-cli2
shellcheck koalaman/shellcheck release tarball behind actions/cache
actionlint rhysd/actionlint and koalaman/shellcheck tarballs
lychee_offline lycheeverse/lychee release tarball behind actions/cache
check_jsonschema astral-sh/setup-uv, then uvx check-jsonschema — the step that stalled

Untouched, because they download nothing: configuration, detect
(actions/github-script, API only), exec_bit, machine_specific_paths,
eol_renormalize, comment_hygiene, and the outcome join. No composite
action file is modified.

Why the key goes on the workflow step and not inside the composite

A composite action's runs.steps[*] accepts run, shell, if, name,
id, env, working-directory, uses, with and continue-on-error — and
no timeout-minutes
(Metadata syntax for GitHub Actions).
The feature request to add it is
actions/runner#1979, still
open. The workflow step's own timeout-minutes is honoured for the whole
composite invocation:
actions/runner#599 (merged
2020-07-22) wired the step's timeout-minutes into the cancellation token of
the composite action step's execution context.

Why this restores the signal rather than just shortening the stall

A step killed by its own timeout-minutes is marked Failed, not
CanceledStepsRunner.cs
distinguishes the two by whether the job token also fired, and sets
TaskResult.Failed for the step-timeout branch. So continue-on-error: true
still holds, steps.<id>.outcome is failure, the outcome join runs and
emits ::error::check-jsonschema failed (outcome=failure)., and the caller's
aggregate gets a named cause instead of a bare cancelled.

Known ceiling

This bounds one stalled step. In the 2026-09-08 incident the host's whole
network path was degraded, so several downloads would have stalled in turn:
the first fails at 8:00, the next stalls, and the job still hits its 15-minute
budget and ends cancelled. Narrowing the per-step budget, or deriving it from
inputs.timeout-minutes, is a separate change and is deliberately not made
here.

Verification

Run from the worktree D:/worktrees/ciw-checks-step-timeout on
ci/checks-step-timeouts:

  • actionlint --version1.7.12 — the exact version
    .github/actions/actionlint/action.yml pins as its default.
  • actionlint .github/workflows/checks.yml → no output, exit 0. actionlint
    parses the file as YAML before linting it, so this covers the parse too.
  • git diff --stat.github/workflows/checks.yml | 18 ++++++++++++++++++,
    1 file changed, 18 insertions, 0 deletions.
  • git diff → exactly eight + timeout-minutes: 8 lines and a
    ten-line # Stalls: paragraph appended to the file header. No other key,
    value, pin, or SHA changed. No file outside .github/workflows/checks.yml
    is touched.

Change-contract check: this repository has no CHANGELOG.md (grep -rli changelog --include='*.md' . returns nothing) and no version file — README.md
§Versioning states the version lives only in the git tag, cut by release.yml's
manual workflow_dispatch. No bump or changelog entry is owed for this PR.

This PR's own CI executes the modified file: ci.yml dogfoods it through
uses: ./.github/workflows/checks.yml at the commit under test, so the runner
accepts the new keys on a real run. Not exercised: the timeout firing.
Reproducing a 12 KB/s network path on a runner is out of reach here, so that
half rests on the two runner citations above.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01L5ZJnwTRCgc31bmfrdugyU

A stalled dependency download in the shared `checks` lane consumed the
calling job's entire `timeout-minutes` budget, so the run ended
`cancelled` rather than `failure`. The `outcome` join never ran, the
reusable workflow published no outputs, and the caller's `ci-status`
aggregate read the lane as a failure with no named cause.

Every step in the job that downloads something now carries
`timeout-minutes: 8`: the four `actions/cache`-backed tarball installers
(typos, gitleaks, editorconfig, lychee-offline), the two that also fetch
ShellCheck (shellcheck, actionlint), the `actions/setup-node` one
(markdown), and the `astral-sh/setup-uv` one (check-jsonschema). The
pure-git composites (exec-bit, machine-specific-paths, eol-renormalize,
comment-hygiene) and the API-only `change-detection` are unchanged.

The key goes on the workflow step, not inside the composite: a composite
action's `runs.steps[*]` accept no `timeout-minutes`. The runner sets a
timed-out step's result to `Failed`, not `Canceled`, so
`continue-on-error: true` still holds and the join names the stalled
tool instead of the run dying silently.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L5ZJnwTRCgc31bmfrdugyU
@kyle-sexton
kyle-sexton marked this pull request as ready for review September 8, 2026 14:24
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T14:27:22.297020Z e287209 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Sep 8, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 43s —— View job


Code review

  • Gather context (PR body, diff, repo rules)
  • Review .github/workflows/checks.yml diff for correctness/maintainability
  • Post findings (inline + summary)

Reviewed the diff (git diff origin/main...HEAD), the full checks.yml job, the 12 composite action.yml files (to check the PR's download-audit table against reality), and the existing workflow-YAML test suite (checks-outcome.test.cjs, ci-fanout-consolidation.test.cjs) for coupling. CI on this PR (run 34238029840) is green, and the 8 flagged steps do match the composites that actually perform actions/cache/setup-node/astral-sh/setup-uv downloads — the audit table checks out.

Finding: the Check out step downloads the repository over the network and is not bounded

.github/workflows/checks.yml#L215-L218

      - name: Check out
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

The PR's own "Fix" section frames the change as bounding "every step in the checks job that downloads something," and its "Untouched" list accounts for every other step by name (configuration, detect, exec_bit, machine_specific_paths, eol_renormalize, comment_hygiene, the outcome join) with a reason each downloads nothing. Check out is the one step in the job that appears in neither list — it's not in the 8-row table, and it's not in the "downloads nothing" accounting either.

But actions/checkout performs exactly the kind of network operation this PR is defending against: it clones the repository over HTTPS. On the described incident's network path (~12 KB/s for ~15 minutes), a git clone stalls the same way a release-tarball download does — TCP throughput, not payload size, is what stalled astral-sh/setup-uv in the original incident. Because this step has no timeout-minutes and runs first (before continue-on-error even matters — a checkout failure should fail the job), a stall here reproduces the exact bug this PR fixes: the step consumes the whole 15-minute job budget and the run ends cancelled, before the outcome join's if: ${{ !cancelled() }} ever runs.

Unlike the 8 composite steps, Check out isn't wrapped in continue-on-error, so it wouldn't get a step-timeout-minutes bound in quite the same shape (a checkout timeout should still fail the job, not join with the composite outcomes) — but leaving it with no bound at all means the job-level cancelled verdict this PR is written to eliminate is still reachable through this step. Worth a timeout-minutes: (a few minutes is generous for a checkout of this repo) so a checkout stall fails fast and visibly instead of quietly consuming the job's whole budget.

Finding: no test pins the new timeout-minutes: 8 values

Left as an inline comment on checks.yml#L230 (typos step, representative of all 8). Summary: this repo's convention is to assert literal workflow invariants like this in script tests (go-quality.test.cjs, claude-review-superseded-guard.test.cjs), and checks-outcome.test.cjs already parses this exact file's steps — but nothing asserts these 8 steps carry timeout-minutes: 8, so a future drop of the key (or a new download-heavy composite added without one) won't be caught until another stall incident.

Everything else — the per-step placement, the 8 value's headroom versus the 4.5-minute healthy ceiling and the 15-minute default job budget, the composite audit table, and the new header comment's claims about runner behavior (Failed vs Cancelled on step timeout) — checks out against the cited runner source and this repo's own composite files.
· branch

@claude

claude Bot commented Sep 8, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 42s —— View job


Security review — no security-relevant surface

  • Gather context (PR diff, changed files)
  • Run skip gate
  • Review diff for security-relevant surface
  • Post findings (or "no issues" statement)

Skip gate hit: no security-relevant surface. The diff (.github/workflows/checks.yml, +18/-0) is purely additive:

  • Adds timeout-minutes: 8 to the 8 download-heavy steps (typos, gitleaks, editorconfig, markdown, shellcheck, actionlint, lychee_offline, check_jsonschema).
  • Adds a 10-line # Stalls: comment block to the file header.

No triggers, permissions:, secrets handling, if:/expression contexts, inputs/outputs, or action pins are touched — every uses: reference in the diff is unchanged, already SHA-pinned. timeout-minutes only bounds step wall-clock time; it introduces no new trust boundary, no injection surface, and no way to widen access or leak data. This is squarely a reliability fix (turning a silent cancelled verdict into a named failure), not a security-relevant change.

No inline findings to report.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Last security-reviewed head: e287209f6806eb930b535a791c9ec93411500e65. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e287209f68

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/checks.yml
Comment thread .github/workflows/checks.yml
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

The eight `timeout-minutes: 8` values were unasserted, so a future edit
that dropped one, or a new download-heavy composite added without one,
would not surface until the next stall incident.

`checks-outcome.test.cjs` already parses `checks.yml` structurally and
enumerates the twelve `continue-on-error` composites, so the assertion
reuses that enumeration: every composite except the four shell-only ones
must carry `timeout-minutes: 8`. A new composite is required to declare a
budget or be named as shell-only, rather than defaulting to unbounded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L5ZJnwTRCgc31bmfrdugyU
@kyle-sexton
kyle-sexton merged commit 2c1de45 into main Sep 8, 2026
30 checks passed
@kyle-sexton
kyle-sexton deleted the ci/checks-step-timeouts branch September 8, 2026 14:43
kyle-sexton added a commit that referenced this pull request Sep 8, 2026
…umers (#578)

No related issue: drift found while cutting the v0.24.0 release for PR
#577; the correction is small enough to carry on its own PR rather than
open an issue first.

## Summary

The `## Versioning` section justified the tag-per-change policy with a
claim
that is not true of this repository's consumers:

> cutting one after every meaningful change keeps `main` at the latest
tag so
> Dependabot's `github-actions` group bumps consumers to a tagged SHA
instead
> of tracking `main` HEAD by drift.

Nothing bumps consumers automatically. Consumers pin `ci-workflows` by
full
40-character commit SHA with a `# vX.Y.Z` trailing comment, and that pin
moves
only through a reviewed pull request in the consumer's own repository.
At
least one consumer's Dependabot configuration explicitly ignores
`melodic-software/ci-workflows/*`, so the `github-actions` group never
opens a
bump PR for this repository at all.

The claim mattered in practice: PR #577 added `timeout-minutes` to eight
steps
in `.github/workflows/checks.yml` and merged at `2c1de45`, but because
consumers pin by SHA and no release existed at that commit, the change
reached
no consumer. Believing the README meant believing the change had already
propagated.

`SECURITY.md` stated the same two now-corrected claims, that consumers
pick up
fixes via an advancing pinned SHA Dependabot bumps and that they should
keep
the `github-actions` Dependabot ecosystem enabled, and is rewritten here
to the
same reviewed-repin model.

## Fix

Replaced the Dependabot clause with what actually happens, and left the
rest of
the paragraph's reasoning intact:

- The stated reason to cut a release is now that it makes a tagged SHA
  available to pin to, which is the true benefit.
- Added one sentence stating the real propagation path: SHA pin plus
version
  comment, moved by a reviewed PR in the consumer's repository.
- The following `release-gap-check` sentence was already correct and is
preserved verbatim; it is only rewrapped because the replacement changed
the
  line boundaries.
- `SECURITY.md`: the `## Supported versions` paragraph now says a
security fix
reaches a consumer only when that consumer repins by reviewed pull
request,
names the `release-gap-check` advisory issue, and tells consumers to
watch
this repository's Releases; the **Pin by full commit SHA** bullet now
says to
keep this repository out of Dependabot's `github-actions` updates with
an
  `ignore` entry and move the pin by reviewed pull request.

The diff is one README paragraph plus two SECURITY.md sentences.

## Verification

- `npx markdownlint-cli2@0.23.2 README.md` against the repo's own
  `.markdownlint-cli2.jsonc`:
  `Summary: 0 issues in 0 files`, exit 0.
- `npx markdownlint-cli2 --config .markdownlint-cli2.jsonc SECURITY.md`:
  `Summary: 0 issues in 0 files`, exit 0.
- `git diff --stat` for the README commits: `README.md | 15
+++++++-------`,
  `1 file changed, 8 insertions(+), 7 deletions(-)`.
- `git diff --stat` for the SECURITY.md commit: `SECURITY.md | 14
+++++++-----`,
  `1 file changed, 9 insertions(+), 5 deletions(-)`.
- `git diff | grep '^+.*[ \t]$'` returns no matches, so no trailing
whitespace
  was introduced.
- Read back the rendered paragraph to confirm the `release-gap-check`
sentence
  survives word-for-word and only its line wrapping changed.
- Confirmed the underlying facts before writing: `release.yml` computes
the
  next version from published Releases and creates a tag plus Release at
`GITHUB_SHA`; nothing in it or in this repository opens a bump PR
against a
  consumer.

## Related

- Follows from #577, whose
`timeout-minutes`
change sat unreleased and therefore unreachable by SHA-pinned consumers.
- Release `v0.24.0`
(https://github.com/melodic-software/ci-workflows/releases/tag/v0.24.0)
was cut at `2c1de45aa0e1b1489afb8edfebc12cb3a4fa6ac3` immediately before
this
  PR, which is what actually delivers #577 to consumers once they repin.
- The same incorrect Dependabot claim also appears in the header comment
of
`.github/workflows/release.yml`. It is deliberately left alone here to
keep
this PR to documentation files; it is reported separately as follow-up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01L5ZJnwTRCgc31bmfrdugyU

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant