[ccf-lisa]

Task: IMPROVEMENTS item 8, stack A PR A2 of 3: fx/cat-a2-rules-test-render on fx/cat-a1-private-catalog.

Summary

An external catalog only gets the factory binary in its CI. Until now factory rules test checked decisions but never ran a catalog's prompts or Templates, and never checked a Result against output.schema. So a typo like {{ .run.reslt }} showed up first as a failed run in prod. Each case now also:

• Renders prompts. Every dispatch decision renders its TaskSpec's prompt through the dispatcher's own path: compileTask with KeepSkills, the task id from the decision key evt/<event>/<rule>/<i>, and the event time as now.
• Renders Templates. Every notify decision renders its Template's GitHub body with templateData and renderBody. On a factory.run.completed event, the run comes with its result.
• Applies sets and links in order (allowed/deferred ones), as the rule controller does. A prompt therefore sees the state an earlier set wrote.
• Keeps goldens at <case>/rendered/<decision>-<prompt|template>-<name>.md. The normal run diffs them, --update writes them and removes stale ones, and a missing golden fails like a missing expected.json.
• Fails on render errors. A text that doesn't render fails the case, with the TaskSpec or Template name and the error. So does a text holding <no value>, <nil>, %! or map[: that is catalog_test.go's checkRendered, and it is how text/template renders a field typo. These failures fail with --update too.
• Checks result schemas. A factory.run.completed run result is validated against its TaskSpec's output.schema with ts.OutputSchema.Validate, through the same step validateResult uses. The Result is rebuilt from the rules' flattened run.result, as the inverse of resultMap.
• Coverage. --coverage also warns, without failing, about TaskSpecs (with a prompt) and Templates that no case renders.
• Output. printRulesTest and --format json report the rendered mismatches and failures per case, with a new failed status and count.

The rendering code lives in internal/controllers as one small exported helper, render.go: RenderEvaluation and CheckRunResult. The spec allows this; the render data lives there, and controllers is the RT workstream's directory. Nothing is copied:

• apply's set/link step is extracted as applyToItem and applies;
• validateResult's schema step is extracted as checkOutputSchema.

catalog_test.go is unchanged. make catalog now runs the new checks over the reference catalog. It found three reference fixtures whose run results their own schemas refuse; see "For triage".

Before/after: a broken placeholder

I changed {{ .task.target.repo }} to {{ .task.target.rpo }} on line 6 of catalog/prompts/plan-epic.md.tmpl locally, then reverted it.

Before (the A1 binary):

$ factory rules test --catalog catalog --coverage
coverage: 21 of 21 selected rules fire in some case
  not selected by any RuleSet: merge-release-docs
  not selected by any RuleSet: merge-whole-stack
catalog/testdata: 66 cases, 66 ok, 0 differ
exit 0

After (this branch):

$ factory rules test --catalog catalog --coverage
FAIL dev-epic-opened-maintainer: does not render or its run result fails its schema
  error: TaskSpec plan-epic: the text contains "<no value>": a missing field or a value printed raw
FAIL epic-1-opened-maintainer: does not render or its run result fails its schema
  error: TaskSpec plan-epic: the text contains "<no value>": a missing field or a value printed raw
FAIL hostile-epic-title-body: does not render or its run result fails its schema
  error: TaskSpec plan-epic: the text contains "<no value>": a missing field or a value printed raw
coverage: 21 of 21 selected rules fire in some case
  not selected by any RuleSet: merge-release-docs
  not selected by any RuleSet: merge-whole-stack
  warning: no case renders the prompt of TaskSpec merge
catalog/testdata: 66 cases, 63 ok, 0 differ, 3 failed
exit 1

Goldens and fixtures

• Reference catalog: go run ./cmd/factory rules test --catalog catalog --update wrote 41 rendered goldens. No expected.json changed.
• The PoC cases that cmd/factory's tests replay (internal/rules/testdata/cases/poc against internal/catalog/testdata/poc) got rendered goldens too. Two changes were needed in that test data, which the dispatcher-core workstream owns:
  • The PoC stub prompts printed every input raw (- {{ $k }}: {{ $v }}), so map inputs rendered as map[...]. They now list the input keys.
  • build-stack-start's plan items lacked the title its schema requires. I added titles, and its expected.json gains them in set plan and in the state.
• internal/rules's TestFixtureNames allows the new rendered/ directory. rules.FixtureRendered names it next to the other fixture file names.
• cmd/factory/testdata/rules/render is a small test catalog with two cases, one prompt and one Template, plus an unused TaskSpec and Template for the coverage warning.

Size: 878 changed lines without goldens (the rendered/ files, expected.json and .golden).

Verification

• make reviewable: passes (generated-code check, lint, go test -race, make catalog with catalog/testdata: 66 cases, 66 ok, 0 differ, openapi-lint).
• gofmt -l over the repo's Go files: prints nothing.
• New tests:
  • cmd/factory/rules_test.go:
    • TestRulesTestRender: a prompt that renders; a prompt with a bad field (fails, names TaskSpec plan); a Template that fails to execute; a changed prompt (golden diff with the first differing line); a result that fails its schema; a missing golden; a stale golden; failures under --update; the JSON report.
    • TestRulesTestRenderUpdate: --update writes the goldens and removes a stale one, and a second run passes.
    • TestRulesTestRenderCoverage: the coverage warnings.
  • internal/controllers/render_test.go: TestCheckRunResult, against resultMap's view and fixture-style data-only views.
• Existing tests: TestRulesTestCoverage and the changed-rule golden gain the new lines. TestRulesTestBaseCatalog compares only the decision lines.
• Before/after: shown above.

Only a real external catalog can check one thing: compliance-framework/factory-catalog's CI, which runs factory rules test --catalog catalog --coverage with this binary. Its fixtures will need --update once to write their rendered/ goldens, and any of its run results that its schemas refuse will fail.

For triage

• Three reference fixtures carried run results that their own schemas refuse. I completed them in this PR, because make catalog must pass (they are under catalog/, which the CAT workstream owns):
  • catalog/testdata/closed-item-suppressed/event.json: an implement pr-opened result without pr. Added compliance-framework/docs#81, the item's linked PR.
  • catalog/testdata/dev-epic-review-changes-requested/event.json and catalog/testdata/loop-cap-release-review-escalate/event.json: docs-review(-verify) done results with result: {}, while schemas/results/review.json requires data.findings and data.prs. Added a request-changes entry for the item's PR.
  • None of their decisions changed.
• catalog-poc.md:26 (CAT's LLD) lists case files as {event.json,workitem.json,expected.json}. Cases now also hold rendered/. Proposed fix: add rendered/ there, as this PR does in rules-engine.md.
• --base-catalog renders nothing. It compares decisions only, as it already skips the goldens. A catalog PR's render failures still show in the normal run, which make catalog does.

Open questions for reviewers

• Should a Template render only its GitHub body, as the spec says, or the body of each destination's provider type? A Slack-only Template would fail today with "has no github body". The reference catalog has GitHub bodies only.

🤖 Generated with Claude Code

[gusfcarvalho]

ccf-review: COMMENT

no findings.

Stack (base-chain): #250 → #251