Why it matters
tests/smoke/catalog.json is the only thing that puts an example into CI, but nothing checks that it stays complete. There are also two ways smoke can report green while covering less than it should.
Evidence
- No completeness check. Today all 96 catalog rows match an
examples/* or showcase/* folder. But unlike the plugin manifest (validate.yml:189-246, which is CI-gated), nothing fails when a new example or showcase piece ships without a catalog row. That example would simply never run.
- An empty catalog is green.
tests/smoke/protocol.py:130 makes zero PASSes red. However, the sidecar canary (blender-smoke.yml:245-255) is recorded as a PASS, so if catalog.json became [], run_catalog.py would print "finished 0 entries" and the job would stay green.
- Fail-fast hides regressions.
tests/smoke/run_catalog.py:207-209 returns on the first non-zero exit (if code != 0: … return code). A PR that breaks five examples shows one per CI round-trip, and every example after the failure goes unchecked.
Suggested approach
- Add a
validate.yml step (no Blender needed) that diffs the set of examples/*/ + showcase/*/ directories against the catalog names and fails on either side's orphans, with an allowlist for intentional exclusions.
- Keep canary records out of the catalog PASS count, or assert
catalog_passes >= len(catalog) - skips.
- Change
run_catalog.py to run every entry, collect results, and exit non-zero at the end with a summary of all failures. --fail-fast can stay as an opt-in flag.
Done when
Why it matters
tests/smoke/catalog.jsonis the only thing that puts an example into CI, but nothing checks that it stays complete. There are also two ways smoke can report green while covering less than it should.Evidence
examples/*orshowcase/*folder. But unlike the plugin manifest (validate.yml:189-246, which is CI-gated), nothing fails when a new example or showcase piece ships without a catalog row. That example would simply never run.tests/smoke/protocol.py:130makes zero PASSes red. However, the sidecar canary (blender-smoke.yml:245-255) is recorded as a PASS, so ifcatalog.jsonbecame[],run_catalog.pywould print "finished 0 entries" and the job would stay green.tests/smoke/run_catalog.py:207-209returns on the first non-zero exit (if code != 0: … return code). A PR that breaks five examples shows one per CI round-trip, and every example after the failure goes unchecked.Suggested approach
validate.ymlstep (no Blender needed) that diffs the set ofexamples/*/+showcase/*/directories against the catalognames and fails on either side's orphans, with an allowlist for intentional exclusions.catalog_passes >= len(catalog) - skips.run_catalog.pyto run every entry, collect results, and exit non-zero at the end with a summary of all failures.--fail-fastcan stay as an opt-in flag.Done when