Skip to content

ci: ci-success does not fail on skipped jobs, so a skipped required check reads as green #1688

Description

@cristim

Noticed during the adversarial review of PR #1682. Generic CI-gating hole, not specific to that PR.

What

ci-success (.github/workflows/ci.yml) is the summary job every other job feeds into. It fails the build on failure and on cancelled:

if: always()
steps:
  - run: |
      if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then ... exit 1; fi
      if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then ... exit 1; fi
      echo "All CI checks passed!"

It does not test for skipped. A needed job that is skipped therefore reports as an overall pass.

Why it is latent rather than live

No job in needs currently carries an if: or a paths: filter, so nothing can be skipped today, and ci-success is accurate as things stand. The hole opens the first time any gating job gains a condition: from then on the gate silently stops covering it, and the failure mode is a green ci-success with a check that never ran. That is the same defect class as a guard asserting a property it does not inspect.

Fix direction

Add the third branch:

if [[ "${{ contains(needs.*.result, 'skipped') }}" == "true" ]]; then
  echo "One or more required CI jobs were skipped"
  exit 1
fi

If a job ever legitimately needs to be skippable, it should be excluded from needs deliberately, or use if: always() internally and report a neutral success, rather than relying on ci-success not looking.

Worth checking the other workflows for the same summary-job pattern while in there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions