Skip to content

Run tests in CI, and gate releases on them - #10

Merged
agarzon merged 2 commits into
mainfrom
ci/run-tests
Aug 6, 2026
Merged

Run tests in CI, and gate releases on them#10
agarzon merged 2 commits into
mainfrom
ci/run-tests

Conversation

@agarzon

@agarzon agarzon commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Nothing has ever run the tests in CI. .github/workflows/ held only publish.yml and release.yml, both triggered by a tag — so a red commit could reach the ComfyUI Registry, where it lands on real installs. The 293 backend tests only ever ran on a maintainer machine.

What this does

  • test.ymlpytest -q plus node --test tests/test_upload.mjs, on pull_request and pushes to main. Also exposes workflow_call.
  • release.yml / publish.yml — each gains a test job (uses: ./.github/workflows/test.yml) and needs: test. A red tag now ships nothing, to the Registry or to GitHub Releases.
  • CLAUDE.md — the commands block still said "there is no JS test harness"; there is one as of Folder upload: drag a folder in, or pick one (#5) #9.

Choices

  • Python 3.10, the floor in requires-python. One version, no matrix — add one when a version-specific bug actually bites.
  • The node step names its file: node --test <dir> treats a directory it finds no tests in as a missing module and fails outright.
  • No coverage gates, no lint job.

Known corner, accepted

The manual workflow_dispatch changelog-backfill path tests the dispatch ref (main) rather than the old tag being backfilled. Costs a minute, and can block a backfill while main is red — safe-fail, and not worth conditional gymnastics to avoid.

Verification

Both suites pass on this branch (293 passed / 1 skipped; 3 JS tests), and all three workflow files parse with the expected job graphs — test, test + publish, test + release. The gating itself only proves out on the next tag push.

Nothing ran the tests: .github/workflows held only publish.yml and
release.yml, both firing on a tag, so a red commit could reach the ComfyUI
Registry — where it lands on real installs.

test.yml runs pytest and the node check on pull_request and on pushes to
main, and exposes workflow_call so release.yml and publish.yml can each
depend on it. A tag whose tests fail now ships nothing, to the Registry or
to GitHub Releases.

Python pinned to 3.10, the floor in requires-python; the node check names
its file because node --test on a bare directory fails when it discovers
nothing.
Copilot AI lite review requested due to automatic review settings August 6, 2026 02:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a dedicated CI test workflow and wires release/publish workflows to depend on it, preventing tags from publishing or releasing when tests fail.

Changes:

  • Introduces .github/workflows/test.yml to run Python (pytest) and Node (node --test) test suites on PRs and pushes to main, and as a reusable workflow.
  • Updates .github/workflows/release.yml to run the test workflow first and gate the release job on it.
  • Updates .github/workflows/publish.yml to run the test workflow first and gate the publish job on it.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
.github/workflows/test.yml Adds a reusable CI workflow that runs backend (pytest) and frontend (node test runner) tests.
.github/workflows/release.yml Adds a test job and gates the release job on it.
.github/workflows/publish.yml Adds a test job and gates the publish job on it.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/test.yml Outdated

- name: Backend tests
run: |
pip install -e ".[test]"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — python -m pip and python -m pytest, so both target the interpreter setup-python provisioned.

Comment on lines +16 to +17
test:
uses: ./.github/workflows/test.yml

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, in both directions.

test.yml now declares permissions: contents: read at the workflow level, so every caller gets the reduced scope rather than each one remembering to pass it — this job runs repo code via pip install -e . and never writes anything back.

release.yml also had contents: write at the workflow level; that has moved onto the release job, which is the only thing that needs it to create the release.

Comment on lines +16 to +20
test:
uses: ./.github/workflows/test.yml

release:
needs: test

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and it is a pre-existing bug rather than something this PR introduced — release.yml has always checked out the dispatch ref. publish.yml already pinned ref: ${{ github.event.inputs.tag || github.ref_name }}; release.yml now matches it.

Outside this PR's original scope, but it is two lines, the fix already existed in the sibling file, and leaving a known-wrong release-notes path in a file I am editing anyway is the worse call. Flagged in the PR description.

Note it does not extend to the reusable test job: a workflow_call job runs at the caller's ref, so a manual backfill still tests main. That corner is called out in the description as accepted.

…on manual release

Review follow-ups on #10. test.yml declares contents: read at workflow level
so every caller gets it — release.yml granted contents: write workflow-wide,
which the reusable job inherited despite only running repo code. That write
scope now sits on the release job alone.

Also fixes a pre-existing bug the review surfaced: release.yml checked out
the dispatch ref, so re-releasing an old tag read main's CHANGELOG rather
than the tag's. publish.yml already had the ref pin; release.yml now matches.
@agarzon
agarzon merged commit 38f1cb7 into main Aug 6, 2026
1 check passed
@agarzon
agarzon deleted the ci/run-tests branch August 6, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants