Publish on a tag, prove the wheel delivers verdicts, and fix OPA being unusable in CI

[kmadan]

Four commits. Closes #60.

Merge order: this depends on Principled-Evolution/gopal#96. See "Before merging" at the bottom.

Publishing was manual, so it was skipped

GitHub has had v0.8.0 since 28 August while PyPI serves v0.7.3 from 14 May, so pip install aicertify does not install what this repository represents as the product.

A tag now publishes, via PyPI Trusted Publishing, with no API token stored here. Three things gate a release:

• The tag must match pyproject.toml. A tag disagreeing with the version it carries is how published and represented versions come apart.
• The wheel must contain the policy library. The .rego files come from a submodule and are package data, so a packaging or checkout change can drop them silently. The build job counts them in the archive and checks coverage.json is present, since extraction reads it to find each policy's decision rule.
• The wheel must produce verdicts. scripts/smoke_test_wheel.py installs it into a clean virtualenv and runs eight real framework evaluations. It refuses to run from a source checkout: the repository's own .rego files are on disk either way, so a source-tree run passes even when packaging ships nothing.

The smoke test asserts on counts, not exit status. aicertify evaluate prints "OPA Evaluation: Successful" and exits 0 whether it produced 29 verdicts or none, so no exit code separates a working release from an empty one.

Golden tests for every advertised framework

v0.8.0 fixed a condition where EU AI Act reported 4 verdicts out of 29 and six other frameworks reported none, while the process exited successfully. Nothing failed because nothing asserted that a framework produces verdicts. All fourteen frameworks are now covered, in three layers:

1. The library is present — declared >= floor
2. Everything declared arrives — delivered == declared
3. Nothing is unaccounted for — every evaluated package is known to the index

What "declared" is measured against decides whether layer 2 works at all. It comes from gopal's coverage.json filtered to the framework, never from the packages the evaluation returned: a package that fails to evaluate is absent from the results and would be absent from an expectation derived from them, so the count agrees with itself and passes. Read against the index, the same run reads 5 declared and 4 delivered for education.

Layer 1 exists because layer 2 is vacuous alone — with the submodule missing, nothing is declared, nothing delivered, and 0 == 0 is green. It is also the only guard against the library shrinking, since a policy removed from gopal outright leaves both counts equal. Floors are 80% of today's declared count, never below 1.

education is xfail(strict=True), not skipped. Strict means the exemption cannot outlive the defect: once gopal#96 lands and the submodule is bumped, the test passes, pytest reports the unexpected pass as a failure, and the entry must be removed.

OPA was unusable in CI, and one path reported mock verdicts as real

GitHub Actions sets CI=true. OpaEvaluator read that as skip_opa_check and set opa_path to None with the binary installed and on PATH.

None went into argv[0]. Every call through evaluate_policy raised sequence item 0: expected str instance, NoneType found — item 0 being the OPA executable — and evaluate_by_folder_name reported "No valid results from any policy evaluation" for every framework.

The other path is worse. _evaluate_with_local_opa checks for None and returns {"mock_result": true}, so an evaluation under CI produced fabricated verdicts that nothing downstream distinguishes from real ones. That is the same fail-open shape as #78.

This is not confined to this repository's CI. The README offers the tool for "Local or CI", and any consumer pipeline sets CI=true, so policy evaluation was broken wherever it is most advertised to run.

The flag exists so a missing binary does not abort startup, not so a present one goes unused. Skipping the check now still resolves the path. use_external_server keeps opa_path as None, the one case where that is correct.

Nothing caught this because every existing test that exercises OPA resolves the binary itself with shutil.which and invokes it directly, so none went through the evaluator's own path. tests/test_framework_golden.py is the first that does.

Downloads badge restored (#60)

The badge was removed in #59 pending pepy.tech stats. The issue set the condition for putting it back, and https://img.shields.io/pepy/dt/aicertify.json now returns {"message":"2.5k","color":"brightgreen"}. Added to the five READMEs the issue names with the alt text it specifies, plus README-pypi.md, which is the badge row PyPI itself renders.

Verification

• 264 passed, 2 skipped, 1 xfailed under CI=true — the exact condition that was failing
• Golden tests pass 45 with 1 xfail under CI=true, matching local; before the fix every framework failed
• The wheel builds with 198 .rego files and coverage.json, installs into a clean virtualenv, and delivers verdicts for all eight smoke frameworks
• Against the unfixed submodule the smoke test correctly refuses to publish, naming the silent package
• Three CI regression tests; two fail against the previous behaviour

Before merging

1. Merge fix(counts): one manifest for every published figure, and the education package that delivered nothing gopal#96
2. Bump the aicertify/opa_policies submodule past it
3. Delete the education entry from KNOWN_DELIVERY_GAPS — the strict xfail will fail the suite until you do, deliberately

A PyPI Trusted Publisher is already configured for release.yaml / environment pypi. gh workflow run release.yaml -f dry_run=true exercises build and smoke without publishing.