Skip to content

ci: make validate.sh run the consumer suite #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codeforester merged 2 commits into from
Sep 19, 2026
Merged

ci: make validate.sh run the consumer suite #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ec180fa
ci(validation): run consumer tests from the validation entry point
codeforester Sep 18, 2026
81d5181
ci: make validation fail fast and authoritative
codeforester Sep 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/compatibility.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@ jobs:
run: python -c 'import base_cli_demo, pathlib; root = pathlib.Path.cwd().resolve(); assert root not in pathlib.Path(base_cli_demo.__file__).resolve().parents'
- name: Validate the repository baseline
run: ./tests/validate.sh
- name: Run installed-wheel compatibility tests
run: python -m pytest -q

upcoming:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.upcoming_base_cli != '' }}
Expand All @@ -73,4 +71,4 @@ jobs:
- name: Install the demo wheel without source dependencies
run: python -m pip install --no-deps dist/*.whl
- name: Run the non-blocking upcoming compatibility check
run: python -m pytest -q
run: ./tests/validate.sh
6 changes: 2 additions & 4 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,14 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Validate repository baseline
run: ./tests/validate.sh
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.13"
- name: Install the reference consumer
run: python -m pip install ".[dev]"
- name: Run consumer tests
run: python -m pytest
- name: Run authoritative consumer validation
run: ./tests/validate.sh

@codeforester codeforester Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: lost fail-fast behavior for the baseline check.

Previously 'Validate repository baseline' ran as the very first step, before 'Set up Python' — a missing required file (e.g. LICENSE, CHANGELOG.md) failed within seconds. Now ./tests/validate.sh only runs after Python setup and pip install ".[dev]" complete, and (per the change in tests/validate.sh) it now also runs the entire pytest suite at the end. A trivial baseline failure that used to surface almost instantly now waits through the full environment setup first.


optional-integrations:
runs-on: macos-latest
Expand Down
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,12 @@ profile boundary.

## Development

Install the development extra and run the focused suite:
Install the development extra and run the authoritative consumer gate (which
checks the installed environment and runs the complete suite, including the
documented-command smoke tests):

```bash
python -m pip install ".[dev]"
python -m pytest
./tests/validate.sh
```

Expand Down
23 changes: 22 additions & 1 deletion tests/validate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail

required_files=(
README.md
Expand All @@ -21,4 +22,24 @@ for file in "${required_files[@]}"; do
}
done

printf 'Repository baseline is present.\n'
command -v python >/dev/null || {
printf 'Python is required; install the project development extra first.\n' >&2
exit 1
}

python - <<'PY'

@codeforester codeforester Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: the new distribution-version gate never actually blocks the script.

tests/validate.sh has no set -e, and nothing checks the exit status of the python - <<'PY' ... PY block. If a required distribution (base-cli-demo, base-cli, or pytest) is missing, the heredoc raises SystemExit and prints the intended diagnostic to stderr, but the script falls straight through to printf 'Running the complete consumer and documentation-command suite.\n' and exec python -m pytest -q anyway — i.e. the 'authoritative' gate is advisory only.

I verified this pattern in isolation:

before
boom
after (should NOT print if set -e worked, but WILL print without set -e)
exit code: 0

Suggest adding set -euo pipefail near the top of the script (or an explicit || exit 1 after the heredoc) so a missing distribution actually stops the run with the clear message, instead of silently falling through to pytest.

from importlib.metadata import PackageNotFoundError, version

for distribution in ("base-cli-demo", "base-cli", "pytest"):
try:
installed_version = version(distribution)
except PackageNotFoundError as exc:
raise SystemExit(
f"Missing installed distribution {distribution!r}; "
'run `python -m pip install ".[dev]"` first.'
) from exc
print(f"Found {distribution} {installed_version}.")
PY

printf 'Running the complete consumer and documentation-command suite.\n'
exec python -m pytest -q

@codeforester codeforester Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-file impact: folding pytest into validate.sh makes two other workflows run the full suite twice.

This PR only updates .github/workflows/tests.yml, but ./tests/validate.sh is also invoked (unchanged) as a preliminary 'baseline' step in:

  • .github/workflows/compatibility.yml (supported job): 'Validate the repository baseline' (./tests/validate.sh) is immediately followed by 'Run installed-wheel compatibility tests' (python -m pytest -q).
  • .github/workflows/release-package.yml (verify job): 'Validate repository baseline' (./tests/validate.sh) runs before ./tests/package.sh and the wheel build, and is followed later by 'Run the installed package tests' (python -m pytest -q) against the built wheel.

With this exec python -m pytest -q now inside validate.sh, both jobs run the entire pytest suite twice per invocation (4x across the compatibility matrix). In release-package.yml it's worse than just wasted time: the full suite now also runs prematurely against the source/editable install, before the release wheel is even built, where the step used to be a near-instant file-existence check.

Worth updating those two workflows (drop their now-redundant explicit pytest step) as part of this change, or keeping validate.sh's pytest invocation opt-in so existing call sites aren't silently doubled.

Loading