feat: distinguish smoke skips from passes and assert post-exit sidecars

Exit 0 was the only green. A version skip would have been a vacuous pass. The host runner records SKIP only on exit 77 plus a SMOKE_SKIP reason below --min-version; a missing sidecar after Blender exits is FAIL; a leg with zero PASSes is red.

Signed-off-by: fOuttaMyPaint <TMhospitalitystrategies@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: TMHSDigital

.github/workflows/blender-smoke.yml

@@ -270,3 +270,12 @@ jobs:
 
           print(f'Counts verified: {skill_count} skills, {rule_count} rules, {template_count} {template_word}, {snippet_count} snippets, {example_count} examples')
           PYEOF
+
+  validate-harness:
+    name: Validate smoke harness protocol
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Harness unit tests
+        run: python3 tests/smoke/test_harness.py -v

.github/workflows/validate.yml

@@ -157,8 +157,12 @@ way, and a one-paragraph rationale. 30 to 80 lines is the right size.
   the manifest `version` line (see `release.yml` below) — never hand-edit it.
 - `blender-smoke.yml` executes every shipped example (check-only, no render)
   plus snippet/template smoke tests inside REAL headless Blender, on
-  5.2 LTS and 4.5 LTS for every PR (5.1 on the weekly cron). A new example is not
-  shipped until it has a step here.
+  5.2 LTS and 4.5 LTS for every PR (5.1 on the weekly cron). Examples run
+  through `tests/smoke/run_example.py` (catalog: `tests/smoke/catalog.json`).
+  SKIP is exit 77 plus a `SMOKE_SKIP:` reason, and only when `--min-version`
+  is above this Blender; exit 0 with that marker is a vacuous pass and fails.
+  Post-exit sidecars are opt-in (`--expect-sidecar`). A leg with zero PASSes
+  is red. A new example is not shipped until it has a catalog row.
 - `drift-check.yml` consumes `Developer-Tools-Directory/.github/actions/
   drift-check@v1.15` to enforce ecosystem standards-version markers.
 - `release.yml` auto-bumps the version, tags, force-updates floating tags

AGENTS.md

@@ -130,6 +130,20 @@ Stage with **explicit paths only** — never `git add -A` or `git add .`. Cursor
 - **After gallery regeneration** (`python scripts/build_gallery.py`), read the **generated HTML** character by character — the `<img alt>` text and witnesses callouts in `docs/gallery/index.html` and `docs/gallery/<name>/index.html` — not just `examples/gallery.json`. Precedent: the `teaches.split(".")[0]` bug truncated 14/21 card alts at dotted API paths like `bmesh.ops` while the source JSON looked fine (fixed in PR #68).
 - **Playwright gallery captures:** gallery `<img>` tags lazy-load, so force them first (`document.querySelectorAll('img').forEach(i => i.loading = 'eager')`, then wait). **Scroll the target card into view and take a viewport capture** — `scrollIntoView({block:'center', behavior:'instant'})`, short wait, `browser_take_screenshot` with `fullPage` omitted. A `fullPage` capture is NOT a workaround: on a tall gallery page it renders every card image blank even when the images are verified loaded (`complete === true`, `naturalWidth === 1280`, `opacity === 1`) — measured on the 45-card grid at 1425x4516. Verify load state via `browser_evaluate` rather than trusting the pixels.
 
+## Smoke skip and post-exit
+
+The host runner is `tests/smoke/run_example.py`. Shipped examples are listed in
+`tests/smoke/catalog.json` (not a new YAML step per example).
+
+- **SKIP:** print `SMOKE_SKIP: <reason>` and `sys.exit(77)`. Legal only when the
+  catalog/runner `--min-version` is **above** this Blender. Exit 0 with that
+  marker is FAIL (vacuous). Skip on a version that should run is FAIL.
+- **Post-exit sidecar:** opt-in `--expect-sidecar PATH`. The example writes
+  `$BDT_SMOKE_SIDECAR` (set by the runner). The harness asserts after Blender
+  exits. Not a gallery still.
+- **Summary:** `tests/smoke/summarize.py` prints passed/skipped/failed. Zero
+  PASSes makes the job red even if every example skipped cleanly.
+
 ## Example-Run Process
 
 - The canonical example-creation prompt lives at `docs/new-example-prompt.md`; keep it in agreement with this file and `AGENTS.md`.

CLAUDE.md

@@ -46,7 +46,11 @@ you write code:
 Where an API diverges between 4.5, 5.1, and 5.2, asserting each side's actual contract is
 part of the witness—version-gate explicitly and document the divergence rather than
 papering over it. Version-gate on the `bpy.app.version` tuple, not
-`version_string`, which is not bare semver on LTS builds. If you discover an
+`version_string`, which is not bare semver on LTS builds. If an example cannot
+run on a matrix leg, it must print `SMOKE_SKIP: <reason>` and `sys.exit(77)` —
+exit 0 is a pass, not a skip. Set catalog `min_version` so a skip on a version
+that should run is FAIL. Post-exit sidecars (`$BDT_SMOKE_SIDECAR`, harness
+`--expect-sidecar`) are opt-in and are not gallery stills. If you discover an
 undocumented hazard while authoring (a crash, a dangling reference, an ordering
 constraint), that discovery belongs in the code comments and README; it is often
 more valuable than the original subject.
@@ -67,8 +71,8 @@ The example must:
 
 Complete every integration required for a shipped example. Infer the exact current
 shape from neighboring examples and repository configuration, including the example
-directory, README, gallery metadata and assets, plugin manifest, smoke workflow,
-top-level README, and generated gallery pages. After regenerating the gallery with
+directory, README, gallery metadata and assets, plugin manifest, smoke catalog
+(`tests/smoke/catalog.json`), top-level README, and generated gallery pages. After regenerating the gallery with
 `python scripts/build_gallery.py`, read the **generated** output character by
 character—not only `examples/gallery.json` source fields. Open
 `docs/gallery/index.html` and `docs/gallery/<name>/index.html` and inspect the

docs/new-example-prompt.md

@@ -0,0 +1,23 @@
+"""Harness canary: write $BDT_SMOKE_SIDECAR then exit 0.
+
+The host runner asserts the file after this process dies. Pass --omit-sidecar
+to prove a missing sidecar is FAIL. Not a shipped example. No gallery still.
+"""
+import os
+import sys
+
+omit = "--omit-sidecar" in sys.argv
+path = os.environ.get("BDT_SMOKE_SIDECAR")
+if not omit:
+    if not path:
+        print("ERROR: BDT_SMOKE_SIDECAR unset", file=sys.stderr)
+        sys.exit(1)
+    parent = os.path.dirname(path)
+    if parent:
+        os.makedirs(parent, exist_ok=True)
+    with open(path, "w", encoding="utf-8") as fh:
+        fh.write("sidecar-ok\n")
+    print(f"wrote sidecar {path}", flush=True)
+else:
+    print("omitting sidecar (canary red path)", flush=True)
+sys.exit(0)

tests/smoke/canary_sidecar.py

@@ -0,0 +1,8 @@
+"""Harness canary: always skip. Legal only when --min-version is above this Blender.
+
+Prints SMOKE_SKIP and exits 77. Not a shipped example.
+"""
+import sys
+
+print("SMOKE_SKIP: harness canary (always skip)", flush=True)
+sys.exit(77)

tests/smoke/canary_skip.py

@@ -0,0 +1,57 @@
+[
+  {
+    "name": "swatch-grid",
+    "script": "examples/swatch-grid/swatch_grid.py",
+    "args": ["--output", "$OUT/swatch.png", "--engine", "cycles", "--samples", "8", "--width", "640"],
+    "expect_file": "$OUT/swatch.png"
+  },
+  {"name": "turntable", "script": "examples/turntable/turntable.py"},
+  {"name": "gn-sdf-remesh", "script": "examples/gn-sdf-remesh/gn_sdf_remesh.py"},
+  {"name": "depsgraph-export", "script": "examples/depsgraph-export/depsgraph_export.py"},
+  {"name": "wave-displace", "script": "examples/wave-displace/wave_displace.py"},
+  {"name": "driver-wave", "script": "examples/driver-wave/driver_wave.py"},
+  {"name": "bmesh-gear", "script": "examples/bmesh-gear/bmesh_gear.py"},
+  {"name": "shader-node-group", "script": "examples/shader-node-group/shader_node_group.py"},
+  {"name": "temp-override-join", "script": "examples/temp-override-join/temp_override_join.py"},
+  {"name": "gn-instance-grid", "script": "examples/gn-instance-grid/gn_instance_grid.py"},
+  {"name": "gn-modifier-inputs", "script": "examples/gn-modifier-inputs/gn_modifier_inputs.py"},
+  {"name": "shape-key-blend", "script": "examples/shape-key-blend/shape_key_blend.py"},
+  {"name": "curve-bevel-arc", "script": "examples/curve-bevel-arc/curve_bevel_arc.py"},
+  {"name": "compositor-glare", "script": "examples/compositor-glare/compositor_glare.py"},
+  {"name": "damped-track-aim", "script": "examples/damped-track-aim/damped_track_aim.py"},
+  {"name": "color-attribute-wheel", "script": "examples/color-attribute-wheel/color_attribute_wheel.py"},
+  {"name": "parent-inverse-orrery", "script": "examples/parent-inverse-orrery/parent_inverse_orrery.py"},
+  {"name": "grease-pencil-rosette", "script": "examples/grease-pencil-rosette/grease_pencil_rosette.py"},
+  {"name": "gp-lineart-contour", "script": "examples/gp-lineart-contour/gp_lineart_contour.py"},
+  {"name": "armature-bend", "script": "examples/armature-bend/armature_bend.py"},
+  {"name": "text-version-stamp", "script": "examples/text-version-stamp/text_version_stamp.py"},
+  {"name": "image-pixels-testcard", "script": "examples/image-pixels-testcard/image_pixels_testcard.py"},
+  {"name": "uv-layer-grid", "script": "examples/uv-layer-grid/uv_layer_grid.py"},
+  {"name": "png-exr-alpha", "script": "examples/png-exr-alpha/png_exr_alpha.py"},
+  {"name": "vse-cut-list", "script": "examples/vse-cut-list/vse_cut_list.py"},
+  {
+    "name": "vse-cut-list-pixels",
+    "script": "examples/vse-cut-list/vse_cut_list.py",
+    "args": ["--check-pixels", "--engine", "cycles"]
+  },
+  {"name": "gltf-export-roundtrip", "script": "examples/gltf-export-roundtrip/gltf_export_roundtrip.py"},
+  {"name": "lod-decimate-chain", "script": "examples/lod-decimate-chain/lod_decimate_chain.py"},
+  {"name": "vertex-weight-limit", "script": "examples/vertex-weight-limit/vertex_weight_limit.py"},
+  {"name": "triangulate-tangents", "script": "examples/triangulate-tangents/triangulate_tangents.py"},
+  {"name": "gltf-skin-roundtrip", "script": "examples/gltf-skin-roundtrip/gltf_skin_roundtrip.py"},
+  {"name": "vse-gamma-cross", "script": "examples/vse-gamma-cross/vse_gamma_cross.py"},
+  {"name": "light-link-studio", "script": "examples/light-link-studio/light_link_studio.py"},
+  {"name": "collision-hull-proxy", "script": "examples/collision-hull-proxy/collision_hull_proxy.py"},
+  {"name": "custom-normals-shade", "script": "examples/custom-normals-shade/custom_normals_shade.py"},
+  {"name": "sky-texture-sun-elevation", "script": "examples/sky-texture-sun-elevation/sky_texture_sun_elevation.py"},
+  {"name": "mesh-hygiene-audit", "script": "examples/mesh-hygiene-audit/mesh_hygiene_audit.py"},
+  {"name": "prop-origin-transform", "script": "examples/prop-origin-transform/prop_origin_transform.py"},
+  {"name": "soccer-ball-goldberg", "script": "examples/soccer-ball-goldberg/soccer_ball_goldberg.py"},
+  {"name": "car-mirror-symmetry", "script": "examples/car-mirror-symmetry/car_mirror_symmetry.py"},
+  {"name": "attribute-domain-shear", "script": "examples/attribute-domain-shear/attribute_domain_shear.py"},
+  {"name": "degenerate-bevel-weld", "script": "examples/degenerate-bevel-weld/degenerate_bevel_weld.py"},
+  {"name": "modular-kit-snap", "script": "examples/modular-kit-snap/modular_kit_snap.py"},
+  {"name": "lightmap-uv-channel", "script": "examples/lightmap-uv-channel/lightmap_uv_channel.py"},
+  {"name": "socket-attach-points", "script": "examples/socket-attach-points/socket_attach_points.py"},
+  {"name": "vertex-color-ao", "script": "examples/vertex-color-ao/vertex_color_ao.py"}
+]

tests/smoke/catalog.json

@@ -0,0 +1,132 @@
+"""Skip / pass / fail classification for the host smoke harness.
+
+Blender example scripts have two legal ways to finish:
+
+* exit 0 — ran and passed (then optional post-exit sidecar)
+* exit 77 with a ``SMOKE_SKIP: <reason>`` line — unsupported on this Blender
+
+Anything else is FAIL, including the vacuous-pass vector: printing a skip
+marker and exiting 0, or exiting 77 on a version that should run.
+"""
+from __future__ import annotations
+
+import os
+from typing import Optional, Tuple
+
+SKIP_EXIT = 77
+SKIP_PREFIX = "SMOKE_SKIP:"
+
+PASS = "PASS"
+SKIP = "SKIP"
+FAIL = "FAIL"
+
+
+def parse_version(spec: str) -> Tuple[int, ...]:
+    """'5.2' or '5.2.1' or '5.0' -> comparable tuple (pad to 3)."""
+    parts = []
+    for p in spec.strip().split("."):
+        if not p.isdigit():
+            raise ValueError(f"not a version spec: {spec!r}")
+        parts.append(int(p))
+    if not parts:
+        raise ValueError(f"empty version spec: {spec!r}")
+    while len(parts) < 3:
+        parts.append(0)
+    return tuple(parts[:3])
+
+
+def parse_skip_reason(output: str) -> Optional[str]:
+    for line in output.splitlines():
+        s = line.strip()
+        if s.startswith(SKIP_PREFIX):
+            reason = s[len(SKIP_PREFIX):].strip()
+            return reason or None
+    return None
+
+
+def sidecar_ok(path: Optional[str], contains: Optional[str]) -> Tuple[bool, str]:
+    if not path:
+        return False, "sidecar path not set"
+    if not os.path.isfile(path):
+        return False, f"missing post-exit sidecar {path}"
+    if os.path.getsize(path) == 0:
+        return False, f"empty post-exit sidecar {path}"
+    if contains:
+        with open(path, encoding="utf-8") as fh:
+            text = fh.read()
+        if contains not in text:
+            return False, f"sidecar missing expected text {contains!r}"
+    return True, ""
+
+
+def classify(
+    *,
+    proc_exit: int,
+    output: str,
+    min_version: Optional[str] = None,
+    blender_version: Optional[str] = None,
+    forbid_skip: bool = False,
+    expect_sidecar: bool = False,
+    sidecar_path: Optional[str] = None,
+    sidecar_contains: Optional[str] = None,
+) -> Tuple[str, str]:
+    """Return (PASS|SKIP|FAIL, detail).
+
+    Skip is legal only when ``min_version`` is set and ``blender_version``
+    is strictly below it (and ``forbid_skip`` is false). An expected skip
+    does not require a sidecar.
+    """
+    reason = parse_skip_reason(output)
+    skipped = proc_exit == SKIP_EXIT
+
+    if proc_exit == 0 and reason is not None:
+        return FAIL, "SMOKE_SKIP marker with exit 0 (vacuous skip)"
+
+    if skipped:
+        if not reason:
+            return FAIL, f"exit {SKIP_EXIT} without SMOKE_SKIP reason"
+        if forbid_skip:
+            return FAIL, f"skipped where skip is forbidden: {reason}"
+        if min_version is None:
+            return FAIL, f"skipped with no --min-version (unexpected): {reason}"
+        if blender_version is None:
+            return FAIL, f"skipped but blender version unknown: {reason}"
+        if parse_version(blender_version) >= parse_version(min_version):
+            return (
+                FAIL,
+                f"skipped on {blender_version} but min-version {min_version} "
+                f"(should run): {reason}",
+            )
+        return SKIP, reason
+
+    if proc_exit != 0:
+        return FAIL, f"blender exit {proc_exit}"
+
+    if expect_sidecar:
+        ok, detail = sidecar_ok(sidecar_path, sidecar_contains)
+        if not ok:
+            return FAIL, detail
+
+    return PASS, ""
+
+
+def summarize_records(records):
+    """records: iterable of dicts with 'status' key.
+
+    Returns (passed, skipped, failed, harness_exit).
+    harness_exit is 0 on mixed pass+skip, 1 if any FAIL, 2 if zero PASS.
+    """
+    passed = skipped = failed = 0
+    for rec in records:
+        st = rec["status"]
+        if st == PASS:
+            passed += 1
+        elif st == SKIP:
+            skipped += 1
+        else:
+            failed += 1
+    if failed:
+        return passed, skipped, failed, 1
+    if passed == 0:
+        return passed, skipped, failed, 2
+    return passed, skipped, failed, 0

tests/smoke/protocol.py

@@ -0,0 +1,76 @@
+"""Run every shipped example through run_example.classify via run_example.py.
+
+Stops on the first FAIL (same fail-fast as the previous per-step YAML).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+
+def main(argv=None):
+    p = argparse.ArgumentParser()
+    p.add_argument("--blender", required=True)
+    p.add_argument("--catalog", default=os.path.join(HERE, "catalog.json"))
+    p.add_argument("--series", required=True)
+    p.add_argument("--status", default=os.environ.get("BDT_SMOKE_STATUS"))
+    p.add_argument("--out", required=True, help="scratch dir for --output renders")
+    p.add_argument("--xvfb", action="store_true")
+    args = p.parse_args(argv)
+
+    with open(args.catalog, encoding="utf-8") as fh:
+        catalog = json.load(fh)
+
+    os.makedirs(args.out, exist_ok=True)
+    runner = os.path.join(HERE, "run_example.py")
+    n = 0
+    for item in catalog:
+        n += 1
+        name = item["name"]
+        script = item["script"]
+        extra = list(item.get("args") or [])
+        extra = [a.replace("$OUT", args.out) for a in extra]
+        cmd = [
+            sys.executable,
+            runner,
+            "--name",
+            name,
+            "--blender",
+            args.blender,
+            "--script",
+            script,
+            "--series",
+            args.series,
+            "--status",
+            args.status or "",
+        ]
+        if args.xvfb:
+            cmd.append("--xvfb")
+        if item.get("min_version"):
+            cmd.extend(["--min-version", item["min_version"]])
+        if extra:
+            cmd.append("--")
+            cmd.extend(extra)
+        print(f"::group::{name}", flush=True)
+        code = subprocess.call(cmd)
+        print("::endgroup::", flush=True)
+        if code != 0:
+            print(f"catalog abort at {name} (exit {code})", file=sys.stderr)
+            return code
+        expect = item.get("expect_file")
+        if expect:
+            expect = expect.replace("$OUT", args.out)
+            if not os.path.isfile(expect) or os.path.getsize(expect) == 0:
+                print(f"ERROR: expected output missing {expect}", file=sys.stderr)
+                return 1
+    print(f"catalog finished {n} entries", flush=True)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())