Pattern quality: Fix status-report archetype missing min-integrity: none in generated workflow - #262
Draft
github-actions[bot] wants to merge 1 commit into
Draft
Conversation
…n tip The status-report archetype's tips list explicitly instructs setting tools.github.min-integrity: none (since status reports only summarize untrusted issue/PR/discussion content, never act on it), but the archetype definition in patterns/workflow-generation.json had no min_integrity field, so generateWorkflowFile never emitted the line in the suggested starter YAML shown to the downstream agent. Downstream agents anchor heavily on the concrete starter YAML in the prompt over prose guidance buried in a bullet list, so the omission meant many generated status-report workflows likely dropped this safety setting despite the tip. Fix: add "min_integrity": "none" to the status-report archetype so the generated frontmatter matches its own documented guidance. Also updated the outdated test that asserted status-report never gets min-integrity, and added a replacement test using documentation-updater (an archetype without github toolsets) to keep coverage for archetypes that legitimately omit min-integrity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was evaluated
Reviewed all 6 samples in
/tmp/gh-aw/data/generated-patterns-and-prompts.json(status-report, issue-triage, code-improvement, documentation-updater, dependency-monitor, pr-review), comparing each generatedpatternobject and prompt/starter-YAML againstpatterns/workflow-generation.json,patterns/archetypes/*.json, and the generator insrc/js/workflow.js.Recurring quality issue found
The
status-reportarchetype's owntipslist instructs:This tip is correctly surfaced in the generated prompt text ("Follow the workflow-specific guidance..."), but the
status-reportarchetype definition inpatterns/workflow-generation.jsonhad nomin_integrityfield. SincegenerateWorkflowFileinsrc/js/workflow.jsonly emitsmin-integritywhen the archetype definition sets it, the "## Suggested workflow file" starter YAML shown to the downstream agent silently omittedmin-integrity: noneundertools.github, even though it listsgithub_toolsets: trueand reads discussions/issues/PRs.Downstream agents strongly anchor on the concrete starter YAML they're told to use "as a starting point," so a contradiction between the prose tip and the omitted example line risks the safety setting being dropped in generated workflows — undermining the exact guidance the pattern library itself recommends.
Notably, the existing test suite even encoded the bug as expected behavior (
does not add min-integrity for archetypes without untrusted external content, asserting status-report never getsmin-integrity), which no longer matches the archetype's stated guidance for reading untrusted content.What changed
patterns/workflow-generation.json: added"min_integrity": "none"to thestatus-reportarchetype definition, so generated status-report workflows includemin-integrity: noneundertools.github, matching the archetype's own documented tip.test/workflow.test.js: replaced the now-incorrect test with one assertingmin-integrity: noneis emitted forstatus-report, and added a new test usingdocumentation-updater(an archetype withoutgithub_toolsets) to preserve coverage for archetypes that legitimately omitmin-integrity.Validation
npm test: 320/320 tests passed (18 test files).npm run build: succeeded,dist/patterns/workflow-generation.jsonregenerated with the new field.Follow-up ideas deliberately left out
pr-review,status-report) that referencemin-integrityin their tips — both are now consistent with their generated output.pattern/manifest.jsonscan-derived success-rate or archetype metadata, since those are regenerated from raw scan data and out of scope per the task's DO NOT rules.