From 2ad69aefa8834592a8a35381a6d59dcca5c35645 Mon Sep 17 00:00:00 2001 From: Nate Sticco Date: Fri, 11 Sep 2026 23:49:01 -0700 Subject: [PATCH 1/5] Unify portable acceptance and completion-target evidence (MoonMind#4265) --- bundle/commands/markdown/moonspec.verify.md | 141 +---------- bundle/docs/AuthoringGuide.md | 5 +- bundle/moonspec.bundle.yaml | 4 + bundle/skills/moonspec-assess/SKILL.md | 60 ++++- .../skills/moonspec-assess/agents/openai.yaml | 4 +- bundle/skills/moonspec-verify/SKILL.md | 75 +++--- .../references/acceptance-policy.md | 103 ++++++++ .../moonspec-verify/scripts/acceptance.py | 236 ++++++++++++++++++ tests/test_portable_acceptance.py | 124 +++++++++ tests/test_scripts_exist.py | 5 +- 10 files changed, 571 insertions(+), 186 deletions(-) create mode 100644 bundle/skills/moonspec-verify/references/acceptance-policy.md create mode 100755 bundle/skills/moonspec-verify/scripts/acceptance.py create mode 100644 tests/test_portable_acceptance.py diff --git a/bundle/commands/markdown/moonspec.verify.md b/bundle/commands/markdown/moonspec.verify.md index d0975078ba..5fe912fdca 100644 --- a/bundle/commands/markdown/moonspec.verify.md +++ b/bundle/commands/markdown/moonspec.verify.md @@ -44,133 +44,13 @@ You **MUST** consider the user input before proceeding (if not empty). ``` - If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently -## Outline +## Verification -This command is the final MoonSpec check. It verifies that the completed implementation satisfies the original instructions or authoritative declarative source. MoonSpec feature artifacts are optional derived context, not required authority. +Use the resolved `moonspec-verify` Skill to verify the original instructions or authoritative declarative source. Load its `SKILL.md` and `references/acceptance-policy.md`; the Skill owns scope resolution, required checks, evidence binding, verdicts, and continuation decisions. This command supplies the user input and runs the extension hooks; it does not define a separate acceptance policy. -1. **Setup**: Resolve the verification baseline in this order: - - explicit user instructions or an explicitly referenced declarative document - - issue-brief verification inputs - - an explicitly provided `spec.md` or feature directory - - an active feature directory discovered from repository context (run `.specify/scripts/bash/check-prerequisites.sh --json --paths-only` to discover it without requiring `plan.md` or `tasks.md`) - Use absolute paths for file-backed sources. Do not require `spec.md`, `plan.md`, or `tasks.md` when another usable baseline exists. +Resolve the Skill from `$MOONMIND_ACTIVE_SKILLS_DIR` when exported, otherwise the host's available Skill resolver or `.agents/skills/moonspec-verify`. Pass `$ARGUMENTS` unchanged, including the original issue/source, constraints, candidate, completion target, and evidence references. Do not require `spec.md`, `plan.md`, or `tasks.md` when another usable original baseline exists. -2. **Load verification sources**: - - **Required**: the selected original-instruction, declarative-document, issue-brief, or spec baseline - - **If present**: `AGENTS.md` for project principles, repo constraints, and test discipline - - **If present**: `spec.md`, `plan.md`, `tasks.md`, `research.md`, `data-model.md`, `contracts/`, `quickstart.md`, `checklists/` - - Extract requirements, acceptance-critical behavior, constraints, non-goals, stable claims, edge cases, and test expectations directly from the selected baseline - - When a spec exists, also extract its user story, `SCN-*`, `FR-*`, `SC-*`, independent test, and source design mappings such as `DESIGN-REQ-*` or `DOC-REQ-*` - - Extract relevant AGENTS.md `MUST` constraints and any explicit quality gates when present - - Treat `spec.md`, `plan.md`, and `tasks.md` as optional derived context. They can help identify scope, expected files, commands, and sequencing, but they are never required authority or proof that behavior is implemented. - -3. **Build a verification inventory**: - - Create an internal checklist keyed by source requirements and traceable items: - - One row per explicit source-direct requirement or stable declarative claim - - One row per `FR-*`, acceptance scenario or `SCN-*`, and observable `SC-*` when a spec provides them - - One row per relevant AGENTS.md principle, repo constraint, or testing-discipline item that affects implementation or verification - - One row per in-scope `DESIGN-REQ-*` or `DOC-REQ-*` - - For each row, track expected behavior, likely production code touchpoints, expected unit or integration tests, current status, evidence, and remaining gap - -4. **Inspect implementation evidence**: - - Review changed and relevant source files discovered from the selected baseline, repository search, optional tasks or plan, contracts, and quickstart - - When `tasks.md` exists, treat checked tasks as a process check, not as implementation evidence - - Confirm production code exists for each functional requirement - - Inspect production code before tests so behavior is verified directly, not inferred from test names alone - - Inspect startup wiring, registration, configuration binding, migrations, routing, background jobs, public contracts, or other integration points when the requirement depends on them - - Confirm unit tests exist for domain behavior and edge cases - - Confirm integration tests exist for acceptance scenarios, external interfaces, persistence, workflows, or other system interactions - - Confirm in-scope source design requirements have code-or-test traceability; copied requirement text in `spec.md` is not implementation evidence - - Treat comments, TODOs, dead code, unreferenced helpers, and documentation-only changes as non-evidence unless the requirement is explicitly documentation-only - - Confirm implementation does not add hidden scope that contradicts the original request, source design, spec, or relevant repo guidance - -5. **Run verification commands when available**: - - Run unit test commands from `AGENTS.md`, README, build or CI configuration, `plan.md`, `tasks.md`, or project conventions - - Run integration test commands from `AGENTS.md`, README, build or CI configuration, `plan.md`, `tasks.md`, quickstart, or project conventions - - Run quickstart validation when `quickstart.md` exists and can be executed safely - - Do not edit tracked files during verification. Normal disposable test artifacts are acceptable only when already ignored by the project. - - If a command is unsafe, unavailable, or requires missing credentials/services, record it as "Not run" with the exact reason - - Treat unit, compile, typecheck, lint, and repo-local hermetic checks as controlling evidence. - - Treat integration, e2e, smoke, quickstart, map-entry, UI/browser, deployment, or external-service tests as advisory when they require unavailable credentials, services, deployed environments, proprietary/binary assets, large fixtures, game/editor map assets such as `.umap` files, simulators, or unsupported local tools. - - Advisory tests must be reported, but their unavailability or environment-caused failure must not fail verification by itself. If an advisory failure reveals a concrete in-scope implementation defect, gate on that underlying defect instead of the suite label. - -6. **Classify each verification item**: - - Use these statuses: - - `VERIFIED`: implementation and validation evidence satisfy the item - - `PARTIAL`: some implementation exists, but behavior, wiring, or test coverage is incomplete - - `MISSING`: no meaningful implementation evidence exists - - `CONFLICT`: implementation contradicts the spec, original request, source design, or relevant repo guidance - - `NO_DETERMINATION`: the selected baseline or repository evidence is too ambiguous to make a defensible call - - Do not mark the feature complete unless every in-scope source requirement, relevant AGENTS.md principle, and source design requirement is `VERIFIED` - - Missing scenario-driven unit coverage or repo-local hermetic validation for required behavior is at least a high-severity gap - - Missing integration/e2e/map/deployment coverage is non-blocking when the required assets, services, credentials, or runtime fixtures are unavailable in the current checkout/runtime - - Separate missing implementation from missing validation when both matter - -7. **Compare implementation to the original request**: - - For each requirement and acceptance scenario, classify evidence using the verification statuses above - - Check whether success criteria are directly validated, indirectly supported, or unverified - - Check whether assumptions made during specification still hold - - Check whether integration tests cover the end-to-end behavior implied by the original request - - Treat missing required unit tests or repo-local hermetic checks as a verification failure unless the selected verification baseline explicitly makes that class irrelevant - - Do not choose a failing verdict solely because advisory integration/e2e/map/deployment validation cannot run or fails due to unavailable non-repo assets, services, credentials, or tooling. Use `FULLY_IMPLEMENTED` when all controlling evidence verifies; record advisory limitations as residual risk or non-blocking gaps. - - Treat violated AGENTS.md `MUST` rules as blocking failures - -8. **Produce the final verification report**: - - ```markdown - # MoonSpec Verification Report - - **Feature**: [name] - **Spec**: [path or N/A for source-direct/issue-brief mode] - **Original Request Source**: [original instructions, declarative document path, issue brief, or spec.md `Input`] - **Verdict**: FULLY_IMPLEMENTED | ADDITIONAL_WORK_NEEDED | NO_DETERMINATION - **Confidence**: HIGH | MEDIUM | LOW - - ## Test Results - - | Suite | Command | Result | Notes | - |-------|---------|--------|-------| - | Unit | [command] | PASS/FAIL/NOT RUN | [notes] | - | Integration | [command] | PASS/FAIL/NOT RUN | [notes] | - - ## Requirement Coverage - - | Requirement | Evidence | Status | Notes | - |-------------|----------|--------|-------| - | FR-001 | [file/test/reference] | VERIFIED/PARTIAL/MISSING/CONFLICT/NO_DETERMINATION | [notes] | - - ## Acceptance Scenario Coverage - - | Scenario | Evidence | Status | Notes | - |----------|----------|--------|-------| - - ## Story Scope, Principles, And Source Claim Coverage - - | Item | Evidence | Status | Notes | - |------|----------|--------|-------| - | DOC-REQ-001 / AGENTS.md principle | [file/test/reference] | VERIFIED/PARTIAL/MISSING/CONFLICT/NO_DETERMINATION | [notes] | - - ## Original Request Alignment - - - [Pass/fail summary against the verbatim original request] - - ## Gaps - - - [Blocking gaps first] - - ## Remaining Work - - - [Ordered, concrete code or test changes required before completion] - - ## Decision - - - FULLY_IMPLEMENTED only if implementation, unit tests, repo-local hermetic checks, in-scope source claims, relevant AGENTS.md principles, and original request alignment all verify. Advisory integration/e2e/map/deployment evidence should be included when available but does not block completion when unavailable for environment or asset reasons. - ``` - -9. **Report completion**: - - If `FULLY_IMPLEMENTED`: state that final verification passed, list the test commands run, and call out any residual risk - - If `ADDITIONAL_WORK_NEEDED`: list blocking gaps and the smallest credible next implementation or test slice - - If `NO_DETERMINATION`: state exactly what evidence, inspection, or command execution is needed to reach a defensible decision +Run the Skill read-only and preserve its complete report and objective evidence. Perform post-verification hooks only after that result is available. ## Post-Execution Checks @@ -186,16 +66,3 @@ This command is the final MoonSpec check. It verifies that the completed impleme - Optional hooks: report the hook command and prompt. - Mandatory hooks: output `EXECUTE_COMMAND: {command}` and execute or delegate it. - If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently - -## Key Rules - -- Verification is read-only except for normal test artifacts. -- Original instructions, a declarative source document, an issue brief, or an optional `spec.md` may define the bounded verification scope. -- `spec.md`, `plan.md`, and `tasks.md` are disposable derived artifacts. Their absence never blocks verification when another usable baseline exists. -- When `spec.md` names a canonical source document, interpret the story against that document's in-scope stable claims; the canonical document remains the durable desired-state authority unless verified drift is handed off to doc reconciliation. -- Do not require unrelated claims from a larger canonical design to verify for this story, but do not let the temporary spec silently override an in-scope canonical conflict. -- Relevant AGENTS.md guidance defines repo principles, constraints, and test discipline for the story. -- `plan.md` and `tasks.md` are optional context but never proof of implementation. -- Unit tests and repo-local hermetic checks are controlling expected evidence. Integration/e2e/map/deployment checks are expected evidence when available, but they are advisory and non-blocking when they depend on unavailable non-repo assets, services, credentials, or tooling. -- Prefer direct, citeable repository evidence from production code, wiring, configuration, and tests. -- Do not mark the feature complete when required behavior is only inferred and not verified. diff --git a/bundle/docs/AuthoringGuide.md b/bundle/docs/AuthoringGuide.md index 3edbafb1ce..3b250051cf 100644 --- a/bundle/docs/AuthoringGuide.md +++ b/bundle/docs/AuthoringGuide.md @@ -37,9 +37,10 @@ Keep templates runtime-neutral where possible. If a template references a MoonSp ## Add Or Update Scripts -Scripts live under `bundle/scripts/bash/`. +Shared Bash scripts live under `bundle/scripts/bash/`. A Skill may ship a portable +Python helper beside `SKILL.md` under its `scripts/` directory. -Scripts must be executable in ordinary Bash environments and should resolve paths relative to the consumer repository root. Register exported scripts in the bundle manifest. +Scripts must run with their declared interpreter (`bash` or `python3`) and should resolve paths relative to the consumer repository root or accept an explicit root. Register exported scripts in the bundle manifest. Skill helpers must not import consumer-specific runtime packages. ## Update Projection Recipes diff --git a/bundle/moonspec.bundle.yaml b/bundle/moonspec.bundle.yaml index ac7b0a5216..9e306cb134 100644 --- a/bundle/moonspec.bundle.yaml +++ b/bundle/moonspec.bundle.yaml @@ -120,6 +120,8 @@ exports: skill: moonspec-orchestrate scripts: + - id: acceptance + path: skills/moonspec-verify/scripts/acceptance.py - id: common path: scripts/bash/common.sh - id: create-new-feature @@ -134,6 +136,8 @@ exports: path: scripts/bash/validate-implementation-scope.sh docs: + - id: acceptance-policy + path: skills/moonspec-verify/references/acceptance-policy.md - id: document-model path: docs/MoonSpecDocumentModel.md - id: bundle-integration diff --git a/bundle/skills/moonspec-assess/SKILL.md b/bundle/skills/moonspec-assess/SKILL.md index 3cfdba1bed..62ebc70b57 100644 --- a/bundle/skills/moonspec-assess/SKILL.md +++ b/bundle/skills/moonspec-assess/SKILL.md @@ -1,16 +1,66 @@ --- name: moonspec-assess -description: Assess source acceptance coverage before MoonSpec planning. +description: Assess existing implementation against an original issue brief or source acceptance matrix before implementation or MoonSpec planning. metadata: + required-skills: "moonspec-verify" required-capabilities: - git --- +Read the portable acceptance policy from the resolved `moonspec-verify` bundle +before assessing, verifying, or completing work. Resolve it at +`$MOONMIND_ACTIVE_SKILLS_DIR/moonspec-verify/references/acceptance-policy.md`; +outside MoonMind use `.agents/skills/moonspec-verify/references/acceptance-policy.md`. +It owns scope, mandatory versus optional evidence, reuse, and completion rules. +Preserve the original scope and previously met requirements as regression constraints; +prior reports are context, not current proof. Candidate success alone cannot close +or transition an issue as already landed. Completion requires objective evidence +on the intended completion target under that policy. + + # MoonSpec Assess -Use this skill after `moonspec-specify` and before `moonspec-plan` when a source-backed spec may include `artifacts/moonspec/source-acceptance.json`. +## Issue-brief assessment + +When supplied `issue_provider`, `issue_ref`, `brief_artifact_path`, and +`assessment_artifact_path`, use the original trusted issue brief and constraints. +No MoonSpec packet is required. This is a bounded initial inspection; collect +available objective evidence but do not start a long suite or mutate code/issues. + +Preserve the trusted loaded brief at `brief_artifact_path` with keys +`issue_provider`, `issue_ref`, `issue_url`, `title`, `description`, +`acceptance_criteria`, `labels`, `preset_brief`, `constraints`, `source_resolution`, +`trusted_source`, `truncated`, and `truncated_fields`. Copy source text exactly; +map provider summary/title to `title` and body/description to `description`. +Recover truncated fields through the authorized owning reader (MoonMind's trusted +Jira reader in managed runs, authenticated issue tooling for GitHub). Record any +unrecovered fields without inventing requirements; incomplete scope cannot certify +whole-issue completion. Missing optional enrichment is only a limitation. + +Write the initial assessment to `assessment_artifact_path` and the step result: +`issue_provider`, `issue_ref`, `issue_url`, `verdict`, `branch`, `base_ref`, `mode`, +`summary`, and `requirements` (stable `id`, original `description`, `status` of +`met`, `partially_met`, `not_met`, or `unverifiable`, and evidence/reason). +Record candidate `subject`, original `scope`, and `completionTarget` identities +from the shared policy. Keep comparison base and publication branch separate. +For an unavailable mandatory check, name its owner, missing evidence, and resume +check; complete independent safe inspection first. + +Use `FULLY_IMPLEMENTED` for apparently present implementation, `PARTIALLY_IMPLEMENTED` +for some implementation with gaps, `NOT_IMPLEMENTED` for missing implementation, +or `BLOCKED` when unavailable mandatory source prevents trustworthy assessment +or only unavailable prerequisites remain. Preserve mixed executable work as a +bounded backlog with its explicit prerequisite handoff. These are +initial inspection verdicts, not objective acceptance or landing evidence. The +verifier must check/reuse objective evidence before publication or completion. +Do not overwrite this artifact with a later verifier verdict. + +## Source-matrix assessment -Before assessing, resolve the active feature directory and feature identifier from `.specify/feature.json` or the selected `specs//spec.md`. Consume `artifacts/moonspec/source-acceptance.json` only when its `featureId` matches the active feature. If the artifact is absent or belongs to a different feature, report `not_applicable` and do not write or require an assessment artifact for this story. +After `moonspec-specify`, resolve the active feature directory and identifier from +`.specify/feature.json` or the selected `specs//spec.md`. Consume +`artifacts/moonspec/source-acceptance.json` only for that feature. If absent or +unrelated, report `not_applicable`; do not require an assessment packet. ## Output @@ -37,7 +87,7 @@ Allowed row statuses: ## Bounded Backlog -Produce `boundedBacklog` for every missing, partial, conflict, or required-unverified row. The backlog is the authoritative implementation backlog for downstream planning, tasks, implementation, and verification. +Produce `boundedBacklog` for every missing, partial, conflict, or required-unverified row. The backlog prioritizes remaining implementation for downstream planning, tasks, and implementation. Verification retains the complete original scope, including previously met requirements as regression constraints. Separate executable repository changes from required evidence or decisions owned outside the current runtime. Name the owner, missing evidence, and resume check @@ -47,4 +97,4 @@ repository work, but if only unavailable mandatory prerequisites remain, report Optional deployment diagnostics are limitations, not blockers; explicitly required deployment acceptance cannot be silently excluded because access is unavailable. -Do not choose `FULLY_IMPLEMENTED` unless every repo-verifiable source row is verified and every manual/provider-only row is explicitly scoped. +Do not choose `FULLY_IMPLEMENTED` for objective acceptance unless every mandatory source row is verified under the shared acceptance policy. A bounded initial assessment may identify apparently present implementation; it cannot replace objective verification or prove landing. diff --git a/bundle/skills/moonspec-assess/agents/openai.yaml b/bundle/skills/moonspec-assess/agents/openai.yaml index d5e3dedc39..dfcb310655 100644 --- a/bundle/skills/moonspec-assess/agents/openai.yaml +++ b/bundle/skills/moonspec-assess/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "MoonSpec Assess" - short_description: "Assess source acceptance coverage before planning." - default_prompt: "Use $moonspec-assess to produce artifacts/moonspec/acceptance-assessment.json from matching source acceptance evidence." + short_description: "Assess issue or source coverage before implementation." + default_prompt: "Use $moonspec-assess to inspect the original issue brief or matching source acceptance matrix and preserve the initial assessment separately from objective verification." diff --git a/bundle/skills/moonspec-verify/SKILL.md b/bundle/skills/moonspec-verify/SKILL.md index 972cfc8979..6a6d817a9e 100644 --- a/bundle/skills/moonspec-verify/SKILL.md +++ b/bundle/skills/moonspec-verify/SKILL.md @@ -10,6 +10,10 @@ metadata: Use this skill to perform the final MoonSpec verification workflow. +Read [the acceptance policy](references/acceptance-policy.md) before selecting +scope, evidence, verdict, reuse, or completion. This bundle owns that policy; +caller instructions supply scope and evidence, not alternative acceptance rules. + ## Scope Verify only. Do not modify source code, tests, specs, plans, tasks, docs, migrations, or configuration. Normal disposable test artifacts are acceptable only when already ignored by the project. @@ -104,10 +108,10 @@ If the user provides issue-brief verification inputs, use issue-brief verificati In issue-brief verification mode: -1. Read the issue brief artifact and assessment artifact. -2. Use the issue summary, description, acceptance criteria, loaded preset brief, and the assessment's unmet and partially-met requirements as the verification baseline. -3. Treat a `PARTIALLY_IMPLEMENTED` assessment as a bounded backlog: verify only the previously unmet or partially met requirements unless the issue brief explicitly requires broader validation. -4. Treat `FULLY_IMPLEMENTED` as already verified only when no implementation step made code changes after that assessment. +1. Read the original issue brief artifact and any supplied assessment artifact. An absent optional assessment does not replace or invalidate a usable original baseline. +2. Use the original issue summary, description, acceptance criteria, and constraints as the verification baseline. The assessment is process context. +3. Use a `PARTIALLY_IMPLEMENTED` assessment to prioritize gaps; verify the complete selected scope, retaining previously met requirements as regression constraints. +4. An initial `FULLY_IMPLEMENTED` assessment only says implementation appears present. Reuse objective evidence only under the acceptance policy; otherwise execute the missing verification. 5. Inspect production code and tests directly; do not treat the assessment itself as proof that new work is complete. 6. Do not require `spec.md`, `plan.md`, `tasks.md`, or a standalone constitution file. @@ -117,8 +121,8 @@ its `remainingWork`, and the original assessment are hypotheses and process context only; they are not current implementation evidence. For every prior `PARTIAL`, `MISSING`, or remaining-work item, inspect the current production code and tests and replace the old status, evidence, line numbers, and notes -with current findings. Never copy or incrementally edit a previous verifier -JSON report as the new report. If the current head differs from the head named +with current findings. Do not carry stale classifications into a new report. Reuse matching objective +evidence under the acceptance policy and explicitly record that reuse. If the current head differs from the head named by the assessment or previous report, no gap may remain solely because the old artifact said it existed or because the candidate is not on the base branch. @@ -135,28 +139,13 @@ the source requires before a repository change, such as proving legacy tasks have drained before removing their handlers. Verify that prerequisite or return the bounded evidence handoff; do not perform an unauthorized deployment. -## Controlling vs Advisory Verification - -Separate verification evidence into controlling and advisory classes before choosing the verdict. - -Controlling evidence can block `FULLY_IMPLEMENTED`: - -- direct production-code inspection for every in-scope requirement. -- unit, compile, typecheck, lint, and other repo-local hermetic checks that exercise in-scope behavior and can run with the checked-out repository plus documented local dependencies. -- hermetic integration tests when their required fixtures, services, and assets are present in the current runtime and the failure identifies a concrete in-scope implementation defect. -- explicit original-instruction, declarative-document, AGENTS.md, issue-brief, or spec `MUST` requirements that are repo-verifiable in the current runtime. - -Unless the selected source explicitly makes them an acceptance or safety -prerequisite, the following are advisory evidence and must not fail verification -by themselves: +## Evidence policy -- integration, e2e, smoke, quickstart, map-entry, UI/browser, deployment, or external-service tests that require credentials, host services, deployed environments, proprietary or binary assets, large fixtures, game/editor map assets, simulators, or other runtime inputs not available in the checkout. -- failures whose root cause is the unavailable advisory environment, such as missing `.umap` files, absent `/Game/Maps/...` assets, unavailable services, or unsupported local tools. -- manual-only checks, production deployment checks, or provider-specific validation unavailable to the verifier. - -When an advisory command is unavailable or fails for an advisory-environment reason, classify the command as `NOT RUN` when detected before execution, or as `FAIL` with a clearly marked non-blocking advisory note when discovered by running it. Record the missing asset, service, fixture, or environment condition in Test Results, Gaps, Diagnostics, or residual risk, but do not put it in Remaining Work and do not choose `ADDITIONAL_WORK_NEEDED`, `NO_DETERMINATION`, or `BLOCKED` solely for that reason. - -If an integration or e2e command failure reveals a concrete in-scope implementation defect that can be fixed in the repository, classify and gate on that underlying defect, not on the suite label. If the implementation, controlling tests, source claims, AGENTS.md principles, and original request alignment all verify, `FULLY_IMPLEMENTED` is allowed even when advisory integration/e2e/map smoke evidence is missing or non-blocking. +Apply [the acceptance policy](references/acceptance-policy.md) to every source +requirement and check. Mandatory acceptance remains controlling regardless of +tool availability; optional enrichment and diagnostics remain advisory. Record +actual command results and exact missing evidence. Isolated implementation tests +must proceed independently of any separately authorized production operation. If the user provides a specific `spec.md` or feature directory, use it and discover sibling artifacts from that directory when present. @@ -369,8 +358,8 @@ or the concrete pre-submission blocker in Test Results and Diagnostics. Complete this discovery before using `NOT RUN`, `recoverableInCurrentRuntime: false`, `needs_human`, or `blocked` on the grounds that build or test tooling is unavailable. -Use `NOT RUN` with an exact reason when a command requires unavailable credentials, missing services, unsafe side effects, unsupported local tools, or excessive environment setup. -Use the controlling/advisory split above when interpreting unavailable assets, +Use `NOT RUN` with exact owning-boundary failure evidence after attempting supported, authorized setup and execution paths. +Use the acceptance policy when interpreting unavailable assets, services, credentials, or tooling. Preserve explicitly mandatory prerequisites; report optional diagnostics as advisory limitations. @@ -388,7 +377,7 @@ Rules: - Do not mark the feature `FULLY_IMPLEMENTED` unless every in-scope source requirement, relevant AGENTS.md principle, source design requirement, and acceptance-critical behavior is `VERIFIED`. - Missing required unit tests or repo-local hermetic checks is a verification failure unless the selected verification baseline clearly makes that test class irrelevant. -- Missing repo-local hermetic integration coverage for acceptance-critical behavior is a verification gap. Unavailable external evidence follows the controlling/advisory policy above: mandatory prerequisites require a handoff, while optional diagnostics remain non-blocking. +- Missing repo-local hermetic integration coverage for acceptance-critical behavior is a verification gap. Unavailable external evidence follows the acceptance policy: mandatory prerequisites require a handoff, while optional diagnostics remain non-blocking. - Separate missing implementation from missing validation when both matter. - Treat violated AGENTS.md `MUST` rules as blocking failures. - Treat original request misalignment as blocking even if later tasks are complete. @@ -406,10 +395,11 @@ Choose exactly one verdict: Prefer `ADDITIONAL_WORK_NEEDED` over `NO_DETERMINATION` when a concrete missing code or test gap is visible. Use `BLOCKED` only for environment failures; it says nothing about implementation completeness. Do not choose a blocking verdict solely because advisory validation cannot run. -Use `FULLY_IMPLEMENTED` when all controlling evidence verifies. When only -mandatory external evidence or an authority decision remains unavailable, use -`NO_DETERMINATION` with `needs_human`. When missing or broken runtime capabilities -prevent required hermetic checks, use `BLOCKED` with `blocked`. These stopping +Use `FULLY_IMPLEMENTED` when the acceptance policy is satisfied. Missing required +external evidence uses `NO_DETERMINATION` with an evidence retry when obtainable, +or `blocked` when unavailable after supported attempts. Use `needs_human` only +for an actual human-owned authority or information decision. Missing runtime +capabilities preventing required checks use `BLOCKED` with `blocked`. These stopping verdicts preserve uncertainty about completeness; they are not assertion failures. When the verdict is `ADDITIONAL_WORK_NEEDED`, include a structured Remaining Work section that remediation steps can consume. Each item must identify: @@ -445,8 +435,7 @@ JSON is requested, use `recommendedNextAction` and The verifier remains read-only; it must not rerun itself to implement the fix. - For `NO_DETERMINATION`, use `reattempt_current_step` only when another verification attempt can obtain different controlling evidence. -- Use `needs_human` when progress requires a scope or authority decision, or an - external dependency that an authorized remediation step cannot supply. Use +- Use `needs_human` when progress requires a scope or authority decision, that only a human can supply. Use `blocked` when no authorized execution path can currently proceed. Explain what must change and preserve the evidence and concrete remaining work. @@ -455,7 +444,7 @@ check that an authorized remediation step can actually execute. Distinguish repository work from prerequisites such as a missing container-test capability, live deployment access, or a required operator decision. Complete feasible repository work while preserving those prerequisites; once only unavailable -prerequisites remain, use `needs_human` or `blocked` and name the required owner, +prerequisites remain, use the appropriate stop action from the acceptance policy and name the required owner, the evidence they must supply, and the check that resumes verification. A newer commit, more unexecuted tests, or a rewritten report does not resolve an unchanged prerequisite. Do not spend another attempt repeating that same blocked check @@ -480,6 +469,16 @@ node, or pull request destination in `recommendedNextAction`. a separate authorized remediation step can complete. If the action is omitted, the caller retains its verdict-based default; omission is not an explicit stop. +## Evidence binding + +For objective success, include `validatedRefs.acceptance` using the complete +`acceptance/v1` contract in [the acceptance policy](references/acceptance-policy.md). +Capture subject and scope before checks and confirm content is unchanged afterward. +Preserve source/base identity separately. Bind each mandatory requirement to actual +command/artifact evidence. A dirty workspace needs its content digest or durable +checkpoint; never use HEAD alone. Report candidate success separately from target +completion. Do not overwrite the initial assessment with this later result. + ## Report Return a Markdown report in the response. Do not write a file unless the user explicitly asks for one. @@ -612,6 +611,6 @@ If no hooks are registered or `.specify/extensions.yml` does not exist, skip sil - Do not require unrelated claims from a larger canonical design to verify for this story, but do not let the temporary spec silently override an in-scope canonical conflict. - Relevant AGENTS.md guidance defines repo principles, constraints, and test discipline for the story. - `plan.md` and `tasks.md` are useful context but never proof of implementation. -- Apply the Controlling vs Advisory Verification policy consistently. Missing optional diagnostics cannot block acceptance; missing explicitly mandatory prerequisites cannot be waived because the environment lacks access. +- Apply the shared acceptance policy consistently. Missing optional diagnostics cannot block acceptance; missing explicitly mandatory prerequisites cannot be waived because the environment lacks access. - Prefer direct, citeable repository evidence from production code, wiring, configuration, and tests. - Do not mark the feature complete when required behavior is only inferred and not verified. diff --git a/bundle/skills/moonspec-verify/references/acceptance-policy.md b/bundle/skills/moonspec-verify/references/acceptance-policy.md new file mode 100644 index 0000000000..ff95db077a --- /dev/null +++ b/bundle/skills/moonspec-verify/references/acceptance-policy.md @@ -0,0 +1,103 @@ +# Acceptance policy + +This is the portable semantic policy for assessment, implementation verification, +evidence reuse, and completion. Presets supply the original scope, constraints, +candidate, completion target, and evidence; they do not redefine this policy. + +## Scope and obligations + +Resolve the baseline from original implementation instructions, the selected +canonical source, or the issue brief. Derived plans, assessment backlogs, previous +verifier reports, and process instructions are context, not new requirements. +Preserve source content and stable requirement IDs. Recheck prior gaps against +current code; previously met requirements remain regression constraints. + +Classify obligations by their source, before checking tool availability: + +- Optional enrichment improves context. Missing retrieval is a disclosed limitation. +- Optional diagnostics provide additional confidence. Their absence does not gate + acceptance, but an observed in-scope defect still requires remediation. +- Mandatory implementation acceptance requires objective evidence, including the + checks required by the source and repository guidance. Missing tools do not + turn these obligations into exclusions. Prepare authorized isolated test + dependencies and complete independent safe work before reporting a blocker. +- Separately authorized production execution is a different operation. Verify + implementation through isolated migration, browser, or workflow tests without + deploying or mutating production. Preserve a production prerequisite when the + selected operation explicitly requires it (for example draining live handlers + before their removal). Do not infer deployment authority from implementation. + +Recover truncated mandatory source content through its authorized owning reader. +When recovery cannot supply it, assess the visible scope and identify the missing +source fields. Never fabricate a requirement from a truncation marker. Set scope +`complete: false`; do not certify the unseen whole issue or complete it. An absent +optional MoonSpec packet is not a gap when a usable original baseline exists. + +## Assessment and verification + +Initial assessment is a bounded inspection. It may report `FULLY_IMPLEMENTED` to +mean implementation appears present, but this does not prove tests passed or work +landed. Preserve the initial assessment artifact and verdict unchanged. Later +verification has its own artifact and cannot rewrite that historical meaning. + +Objective verification may return `FULLY_IMPLEMENTED` only when the complete +selected scope and every mandatory requirement have valid evidence. Use the +existing verdicts and continuation actions. Record implementation defects, +missing verification, unavailable environment, budget exhaustion, and a human +decision separately. Bounded authorized fixes go to the existing remediation +owner; new obtainable evidence goes to evidence retry. Exhausted attempts retain +the last truthful verdict. `needs_human` is reserved for information or authority +only a human can supply, not elaborate tests or an unchanged unavailable service. + +Bind objective evidence in the existing `validatedRefs.acceptance` field using +`acceptance/v1` (see the supplying contract in +`moonmind/workflows/skills/acceptance_contract.py` when hosted by MoonMind): + +- `subject`: repository identity, actual revision, and content digest. Capture a + dirty candidate's content/checkpoint identity; HEAD alone is insufficient. +- `scope`: source reference, digest of original source and constraints, complete + scope flag, and stable mandatory requirement IDs. For issue-driven work, + `sourceRef` is the original canonical issue reference (for example + `owner/repo#123` or `ENG-123`); the source digest includes the complete selected + source content and constraints, including any controlling canonical document. +- `completionTarget`: intended ref, its observed revision and content digest. + Resolve an explicitly configured target first, otherwise the remote default + branch. A feature branch's upstream is not the default/completion target. +- `evidence`: one record per mandatory requirement with its ID and objective + evidence references. All records certify the enclosing subject and scope. +- `freshness`: the required policy identity and optional expiry for time-sensitive + evidence. Content-bound evidence need not expire arbitrarily. + +Use `scripts/acceptance.py` beside this Skill to capture Git identities and check +reuse. It uses an isolated Git index for dirty content and never switches the +user's checkout. A checkpoint ref/digest supplied by an existing workspace owner +may be used instead. Keep the original base/source revision, candidate revision, +and publication branch distinct in the report. + +Reuse objective evidence only for the same subject content, repository, original +scope, requirement IDs, completion policy, and required freshness. A changed +candidate, source, or expired proof requires the affected verification again. +Identical valid evidence needs no arbitrary full-suite repeat. An unchanged +checkout and a previous verdict alone are not objective evidence. + +## Completion target and publication + +Candidate verification approves implementation for the existing review/publication +path. It does not prove landing. Clean or already-pushed candidate commits still +require that path. Skip publication and close/transition an already-completed issue +only with objective evidence on the intended completion target, freshly resolved +through the owning repository reader. Never infer landing from an empty diff, +branch name, process exit, prose, assessment verdict, or a PR link alone. + +Explicit main/trunk verification inspects the resolved target ref, even when a +feature branch or detached HEAD is checked out. Use `git show`/`git grep` at the +pinned ref for reads and an isolated detached worktree at that ref for tests. +Preserve unrelated local edits. A non-main default, empty diff, or squash merge is +normal. Current target behavior suffices; historical merge links are supplemental. +An explicitly selected alternative completion target is equally valid. + +Completion evidence may reuse candidate checks when the target's complete content +is identical (including squash merges); otherwise verify the actual target. +Re-resolve the target before a completion side effect. If it moved, verify/reuse +against that new identity before closing. Unavailable target/source evidence +withholds completion and returns exact evidence work to the existing owner. diff --git a/bundle/skills/moonspec-verify/scripts/acceptance.py b/bundle/skills/moonspec-verify/scripts/acceptance.py new file mode 100755 index 0000000000..84c1a84192 --- /dev/null +++ b/bundle/skills/moonspec-verify/scripts/acceptance.py @@ -0,0 +1,236 @@ +#!/usr/bin/env python3 +"""Portable subject capture and evidence reuse for moonspec-verify. + +No provider API, MoonMind import, or credential discovery. Callers perform the +authorized fetch first and supply the actual source and completion target. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import json +import os +from pathlib import Path +import subprocess +import tempfile + + +def git(repo: Path, *args: str, env: dict | None = None) -> str: + return ( + subprocess.check_output( + ["git", "-C", str(repo), *args], env=env, stderr=subprocess.PIPE + ) + .decode() + .strip() + ) + + +def capture( + repo: Path, repository: str, target: str, *, target_mode: bool = False +) -> dict: + """Capture actual content, including unstaged/untracked work, without checkout.""" + repo = Path(git(repo, "rev-parse", "--show-toplevel")) + target_revision = git(repo, "rev-parse", "--verify", f"{target}^{{commit}}") + target_tree = git(repo, "rev-parse", f"{target_revision}^{{tree}}") + target_ref = git(repo, "rev-parse", "--symbolic-full-name", target) + if target_ref.startswith("refs/remotes/"): + target_ref = ( + "refs/heads/" + target_ref.removeprefix("refs/remotes/").split("/", 1)[1] + ) + if not target_ref.startswith("refs/heads/"): + raise ValueError( + "completion policy must name a branch, not a detached revision" + ) + revision = target_revision if target_mode else git(repo, "rev-parse", "HEAD") + if target_mode: + tree = target_tree + else: + # A private index leaves the user's staging intact. Git's ignore rules + # exclude disposable test artifacts; real untracked source is included. + with tempfile.TemporaryDirectory(prefix="acceptance-index-") as temp: + original_objects = Path(git(repo, "rev-parse", "--git-path", "objects")) + if not original_objects.is_absolute(): + original_objects = repo / original_objects + temporary_objects = Path(temp) / "objects" + temporary_objects.mkdir() + env = dict( + os.environ, + GIT_INDEX_FILE=str(Path(temp) / "index"), + GIT_OBJECT_DIRECTORY=str(temporary_objects), + GIT_ALTERNATE_OBJECT_DIRECTORIES=str(original_objects), + GIT_OPTIONAL_LOCKS="0", + ) + git(repo, "read-tree", "HEAD", env=env) + git(repo, "add", "--all", "--", ".", env=env) + tree = git(repo, "write-tree", env=env) + # A gitlink alone cannot identify dirty nested source. Require its + # owning checkpoint mechanism instead of silently certifying HEAD. + dirty_submodules = git( + repo, + "submodule", + "foreach", + "--quiet", + "--recursive", + 'test -z "$(git status --porcelain --untracked-files=all)" || echo "$displaypath"', + ) + if dirty_submodules: + raise ValueError( + "dirty submodules require a content-bound workspace checkpoint" + ) + return { + "subject": { + "repository": repository, + "revision": revision, + "contentDigest": f"git-tree:{tree}", + }, + "completionTarget": { + "ref": target_ref, + "revision": target_revision, + "contentDigest": f"git-tree:{target_tree}", + }, + } + + +def scope( + source: bytes, source_ref: str, requirement_ids: list[str], *, complete: bool +) -> dict: + if not requirement_ids or len(set(requirement_ids)) != len(requirement_ids): + raise ValueError( + "supply the unique mandatory requirement IDs from the original scope" + ) + return { + "sourceRef": source_ref, + "sourceDigest": "sha256:" + hashlib.sha256(source).hexdigest(), + "requirementIds": sorted(requirement_ids), + "complete": complete, + } + + +def reuse(report: dict, current: dict, *, now: datetime | None = None) -> dict: + """Check an objective report against freshly captured subject/scope/policy.""" + try: + return _reuse(report, current, now=now) + except (AttributeError, KeyError, TypeError, ValueError): + return { + "reusable": False, + "completionEligible": False, + "reasons": [ + "malformed acceptance binding; obtain complete objective evidence" + ], + } + + +def _reuse(report: dict, current: dict, *, now: datetime | None = None) -> dict: + evidence = report.get("validatedRefs", {}).get("acceptance", {}) + reasons = [] + if ( + report.get("verdict") != "FULLY_IMPLEMENTED" + or report.get("invalid") + or report.get("degraded") + ): + reasons.append("no accepted objective verdict") + if evidence.get("schemaVersion") != "acceptance/v1": + reasons.append("objective acceptance binding missing") + prior_subject = evidence.get("subject", {}) + subject = current.get("subject", {}) + # A commit containing identical content (including a squash merge) can + # reuse content evidence. The report retains the original tested revision. + for key in ("repository", "contentDigest"): + if not subject.get(key) or prior_subject.get(key) != subject[key]: + reasons.append(f"subject {key} changed or missing") + if not prior_subject.get("revision") or not subject.get("revision"): + reasons.append("revision identity missing") + current_scope = current.get("scope", {}) + if ( + current_scope.get("complete") is not True + or evidence.get("scope") != current_scope + ): + reasons.append("source scope changed or incomplete") + required = current_scope.get("requirementIds", []) + rows = evidence.get("evidence", []) + if ( + not required + or len(set(required)) != len(required) + or sorted(row.get("requirementId", "") for row in rows) != sorted(required) + or any( + not row.get("evidenceRefs") or not all(row["evidenceRefs"]) for row in rows + ) + ): + reasons.append("mandatory requirement evidence missing") + target = current.get("completionTarget", {}) + if ( + not target.get("ref") + or evidence.get("completionTarget", {}).get("ref") != target["ref"] + ): + reasons.append("completion policy changed or missing") + freshness = evidence.get("freshness", {}) + if ( + not current.get("freshnessPolicy") + or freshness.get("policy") != current["freshnessPolicy"] + ): + reasons.append("required evidence freshness changed or missing") + if freshness.get("validUntil"): + try: + expiry = datetime.fromisoformat( + freshness["validUntil"].replace("Z", "+00:00") + ) + if expiry.tzinfo is None or (now or datetime.now(timezone.utc)) >= expiry: + reasons.append("evidence expired") + except (TypeError, ValueError): + reasons.append("invalid evidence expiry") + reusable = not reasons + return { + "reusable": reusable, + "completionEligible": bool( + reusable + and target.get("revision") + and target.get("contentDigest") == subject.get("contentDigest") + ), + "reasons": reasons, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + capture_parser = commands.add_parser("capture") + capture_parser.add_argument("--repo", type=Path, default=Path.cwd()) + capture_parser.add_argument("--repository", required=True) + capture_parser.add_argument("--target", required=True) + capture_parser.add_argument("--target-mode", action="store_true") + capture_parser.add_argument( + "--source", + type=Path, + required=True, + help="Original source and constraints, preserved verbatim", + ) + capture_parser.add_argument("--source-ref", required=True) + capture_parser.add_argument("--requirement", action="append", required=True) + capture_parser.add_argument("--incomplete", action="store_true") + capture_parser.add_argument("--freshness-policy", default="content") + reuse_parser = commands.add_parser("reuse") + reuse_parser.add_argument("--report", type=Path, required=True) + reuse_parser.add_argument("--current", type=Path, required=True) + args = parser.parse_args() + if args.command == "capture": + result = capture( + args.repo, args.repository, args.target, target_mode=args.target_mode + ) + result["scope"] = scope( + args.source.read_bytes(), + args.source_ref, + args.requirement, + complete=not args.incomplete, + ) + result["freshnessPolicy"] = args.freshness_policy + else: + result = reuse( + json.loads(args.report.read_text()), json.loads(args.current.read_text()) + ) + print(json.dumps(result, indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_portable_acceptance.py b/tests/test_portable_acceptance.py new file mode 100644 index 0000000000..03b5aa4034 --- /dev/null +++ b/tests/test_portable_acceptance.py @@ -0,0 +1,124 @@ +"""Exercise the shipped acceptance helper outside any orchestration host.""" + +import copy +import json +from pathlib import Path +import subprocess +import sys +from types import ModuleType + +import pytest + +SCRIPT = ( + Path(__file__).resolve().parents[1] + / "bundle/skills/moonspec-verify/scripts/acceptance.py" +) +acceptance = ModuleType("acceptance") +exec(compile(SCRIPT.read_text(), str(SCRIPT), "exec"), acceptance.__dict__) + + +def git(repo, *args): + return ( + subprocess.check_output(["git", "-C", str(repo), *args], stderr=subprocess.PIPE) + .decode() + .strip() + ) + + +@pytest.fixture +def verified_candidate(tmp_path): + repo = tmp_path / "repo" + repo.mkdir() + git(repo, "init", "-b", "release") + git(repo, "config", "user.email", "fixture@example.test") + git(repo, "config", "user.name", "Acceptance fixture") + app = repo / "app.py" + app.write_text("print(0)\n") + git(repo, "add", ".") + git(repo, "commit", "-m", "baseline") + git(repo, "switch", "-c", "feature") + app.write_text("print(42)\n") + before = git(repo, "diff") + objects_before = git(repo, "count-objects", "-v") + current = acceptance.capture(repo, "example/repo", "release") + assert git(repo, "count-objects", "-v") == objects_before + current["scope"] = acceptance.scope( + b"Print 42", "example/repo#1", ["AC-1"], complete=True + ) + current["freshnessPolicy"] = "content" + observed = subprocess.check_output([sys.executable, str(app)], text=True) + assert observed == "42\n" + evidence = tmp_path / "check.json" + evidence.write_text(json.dumps({"output": observed, "subject": current["subject"]})) + binding = {key: current[key] for key in ("subject", "scope", "completionTarget")} + binding.update( + schemaVersion="acceptance/v1", + evidence=[{"requirementId": "AC-1", "evidenceRefs": [str(evidence)]}], + freshness={"policy": "content"}, + ) + assert git(repo, "diff") == before + return ( + repo, + current, + {"verdict": "FULLY_IMPLEMENTED", "validatedRefs": {"acceptance": binding}}, + ) + + +def test_dirty_candidate_needs_publication_and_squash_target_can_reuse( + verified_candidate, +): + repo, current, report = verified_candidate + assert acceptance.reuse(report, current) == { + "reusable": True, + "completionEligible": False, + "reasons": [], + } + git(repo, "add", ".") + tree = git(repo, "write-tree") + squash = git(repo, "commit-tree", tree, "-p", "release", "-m", "squash") + git(repo, "update-ref", "refs/heads/release", squash) + current.update( + acceptance.capture(repo, "example/repo", "release", target_mode=True) + ) + assert acceptance.reuse(report, current)["completionEligible"] + assert git(repo, "branch", "--show-current") == "feature" + + +@pytest.mark.parametrize( + "change", ["candidate", "scope", "truncated", "expiry", "missing", "malformed"] +) +def test_changed_or_missing_mandatory_proof_invalidates_reuse( + verified_candidate, change +): + repo, original, report = verified_candidate + current = copy.deepcopy(original) + if change == "candidate": + (repo / "app.py").write_text("print(-1)\n") + current.update(acceptance.capture(repo, "example/repo", "release")) + elif change == "scope": + current["scope"] = acceptance.scope( + b"Print 43", "example/repo#1", ["AC-1"], complete=True + ) + elif change == "truncated": + current["scope"]["complete"] = False + elif change == "expiry": + report["validatedRefs"]["acceptance"]["freshness"]["validUntil"] = ( + "2000-01-01T00:00:00Z" + ) + elif change == "missing": + report["validatedRefs"]["acceptance"]["evidence"] = [] + else: + report["validatedRefs"]["acceptance"] = None + decision = acceptance.reuse(report, current) + assert not decision["reusable"] and not decision["completionEligible"] + assert decision["reasons"] + + +def test_explicit_target_preserves_detached_dirty_checkout(verified_candidate): + repo, current, _ = verified_candidate + git(repo, "switch", "--detach") + before = git(repo, "diff") + target = acceptance.capture(repo, "example/repo", "release", target_mode=True) + assert target["subject"]["contentDigest"] != current["subject"]["contentDigest"] + assert git(repo, "diff") == before + assert git(repo, "branch", "--show-current") == "" diff --git a/tests/test_scripts_exist.py b/tests/test_scripts_exist.py index 11149e7f14..3d195edd4f 100644 --- a/tests/test_scripts_exist.py +++ b/tests/test_scripts_exist.py @@ -6,10 +6,11 @@ BUNDLE = ROOT / "bundle" -def test_exported_scripts_exist_and_are_bash() -> None: +def test_exported_scripts_exist_and_have_portable_interpreters() -> None: manifest = yaml.safe_load((BUNDLE / "moonspec.bundle.yaml").read_text()) for script in manifest["exports"]["scripts"]: path = BUNDLE / script["path"] assert path.exists() - assert path.read_text().startswith("#!/usr/bin/env bash") + interpreter = {".sh": "bash", ".py": "python3"}[path.suffix] + assert path.read_text().startswith(f"#!/usr/bin/env {interpreter}") From 06fb68cb271313a181ac2cf804ee634b388f1f48 Mon Sep 17 00:00:00 2001 From: Nate Sticco Date: Fri, 11 Sep 2026 23:57:46 -0700 Subject: [PATCH 2/5] Delegate verification hooks to the portable skill once --- bundle/commands/markdown/moonspec.verify.md | 53 +-------------------- 1 file changed, 2 insertions(+), 51 deletions(-) diff --git a/bundle/commands/markdown/moonspec.verify.md b/bundle/commands/markdown/moonspec.verify.md index 5fe912fdca..e01d777a0a 100644 --- a/bundle/commands/markdown/moonspec.verify.md +++ b/bundle/commands/markdown/moonspec.verify.md @@ -10,59 +10,10 @@ $ARGUMENTS You **MUST** consider the user input before proceeding (if not empty). -## Pre-Execution Checks - -**Check for extension hooks (before final verification)**: -- Check if `.specify/extensions.yml` exists in the project root. -- If it exists, read it and look for entries under the `hooks.before_verify` key -- If the YAML cannot be parsed or is invalid, skip hook checking silently and continue normally -- Filter out hooks where `enabled` is explicitly `false`. Treat hooks without an `enabled` field as enabled by default. -- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions: - - If the hook has no `condition` field, or it is null/empty, treat the hook as executable - - If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation -- For each executable hook, output the following based on its `optional` flag: - - **Optional hook** (`optional: true`): - ``` - ## Extension Hooks - - **Optional Pre-Hook**: {extension} - Command: `/{command}` - Description: {description} - - Prompt: {prompt} - To execute: `/{command}` - ``` - - **Mandatory hook** (`optional: false`): - ``` - ## Extension Hooks - - **Automatic Pre-Hook**: {extension} - Executing: `/{command}` - EXECUTE_COMMAND: {command} - - Wait for the result of the hook command before proceeding to the Outline. - ``` -- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently - ## Verification -Use the resolved `moonspec-verify` Skill to verify the original instructions or authoritative declarative source. Load its `SKILL.md` and `references/acceptance-policy.md`; the Skill owns scope resolution, required checks, evidence binding, verdicts, and continuation decisions. This command supplies the user input and runs the extension hooks; it does not define a separate acceptance policy. +Use the resolved `moonspec-verify` Skill to verify the original instructions or authoritative declarative source. Load its `SKILL.md` and `references/acceptance-policy.md`; the Skill owns scope resolution, required checks, evidence binding, verdicts, and continuation decisions. This command supplies the user input; the Skill owns acceptance and pre/post-verification hooks. Resolve the Skill from `$MOONMIND_ACTIVE_SKILLS_DIR` when exported, otherwise the host's available Skill resolver or `.agents/skills/moonspec-verify`. Pass `$ARGUMENTS` unchanged, including the original issue/source, constraints, candidate, completion target, and evidence references. Do not require `spec.md`, `plan.md`, or `tasks.md` when another usable original baseline exists. -Run the Skill read-only and preserve its complete report and objective evidence. Perform post-verification hooks only after that result is available. - -## Post-Execution Checks - -**Check for extension hooks (after final verification)**: -- Check if `.specify/extensions.yml` exists in the project root. -- If it exists, read it and look for entries under the `hooks.after_verify` key -- If the YAML cannot be parsed or is invalid, skip hook checking silently and continue normally -- Filter out hooks where `enabled` is explicitly `false`. Treat hooks without an `enabled` field as enabled by default. -- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions: - - If the hook has no `condition` field, or it is null/empty, treat the hook as executable - - If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation -- For executable hooks: - - Optional hooks: report the hook command and prompt. - - Mandatory hooks: output `EXECUTE_COMMAND: {command}` and execute or delegate it. -- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently +Run the Skill read-only and preserve its complete report and objective evidence. Let the Skill execute its hooks once; do not run a second command-level hook lifecycle. From 9270df225054e42439e384c797d598434355a87c Mon Sep 17 00:00:00 2001 From: Nate Sticco Date: Sat, 12 Sep 2026 00:24:57 -0700 Subject: [PATCH 3/5] Preserve verification evidence and prerequisite retry budgets --- bundle/skills/moonspec-verify/SKILL.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/bundle/skills/moonspec-verify/SKILL.md b/bundle/skills/moonspec-verify/SKILL.md index 972cfc8979..6cb73c1ee6 100644 --- a/bundle/skills/moonspec-verify/SKILL.md +++ b/bundle/skills/moonspec-verify/SKILL.md @@ -279,6 +279,23 @@ For each row, track: ## Inspect Evidence +Persist the inventory in the verification artifact instead of leaving it only in +agent context. Each row carries a stable requirement ID, the production boundary, +executable check commands, required capability, expected terminal evidence and +the exact candidate commit or checkpoint. In issue-brief mode, reuse the +assessment requirements verification map; fill missing historical entries from +visible source requirements before executing checks. Never change acceptance +scope to fit the available environment. + +Prerequisite preparation has at most two authorized setup attempts per owning +service and candidate. Persist readiness, setup actions, failures and resume +commands with the inventory; continuation preserves this budget. Use available +repository and managed container capabilities before declaring a dependency +unavailable. A verification infrastructure failure preserves the same saved +candidate and requests evidence retry. A demonstrated code defect requests +remediation. Missing authority names the actual granting owner. These outcomes +must not be conflated or consume each other's budgets. + Inspect production code before tests so behavior is verified directly. Use repository search to find: From abfbaf6e726b9230c5193a97984b40f0d2648e87 Mon Sep 17 00:00:00 2001 From: Nate Sticco Date: Sat, 12 Sep 2026 00:32:26 -0700 Subject: [PATCH 4/5] Reject malformed acceptance evidence and dirty embedded repositories --- .../moonspec-verify/scripts/acceptance.py | 57 +++++--- tests/test_portable_acceptance.py | 129 ++++++++++++++++++ 2 files changed, 167 insertions(+), 19 deletions(-) diff --git a/bundle/skills/moonspec-verify/scripts/acceptance.py b/bundle/skills/moonspec-verify/scripts/acceptance.py index 84c1a84192..584dcd2dc3 100755 --- a/bundle/skills/moonspec-verify/scripts/acceptance.py +++ b/bundle/skills/moonspec-verify/scripts/acceptance.py @@ -27,6 +27,14 @@ def git(repo: Path, *args: str, env: dict | None = None) -> str: ) +def nonblank_strings(value: object) -> bool: + return ( + isinstance(value, list) + and bool(value) + and all(isinstance(item, str) and item.strip() for item in value) + ) + + def capture( repo: Path, repository: str, target: str, *, target_mode: bool = False ) -> dict: @@ -64,21 +72,32 @@ def capture( ) git(repo, "read-tree", "HEAD", env=env) git(repo, "add", "--all", "--", ".", env=env) + # Inspect every staged gitlink, including embedded repositories + # absent from .gitmodules. A gitlink cannot bind dirty nested + # source to the captured tree. + for entry in git(repo, "ls-files", "--stage", "-z", env=env).split("\0"): + if not entry.startswith("160000 "): + continue + nested = repo / entry.split("\t", 1)[1] + if (nested / ".git").exists(): + dirty = git( + nested, + "--no-optional-locks", + "status", + "--porcelain", + "--untracked-files=all", + "--ignore-submodules=none", + ) + else: + # An empty uninitialized submodule has no local content. + # Populated paths without their owning Git metadata cannot + # be certified by the recorded revision alone. + dirty = nested.is_dir() and any(nested.iterdir()) + if dirty: + raise ValueError( + "dirty gitlinks require a content-bound workspace checkpoint" + ) tree = git(repo, "write-tree", env=env) - # A gitlink alone cannot identify dirty nested source. Require its - # owning checkpoint mechanism instead of silently certifying HEAD. - dirty_submodules = git( - repo, - "submodule", - "foreach", - "--quiet", - "--recursive", - 'test -z "$(git status --porcelain --untracked-files=all)" || echo "$displaypath"', - ) - if dirty_submodules: - raise ValueError( - "dirty submodules require a content-bound workspace checkpoint" - ) return { "subject": { "repository": repository, @@ -96,7 +115,9 @@ def capture( def scope( source: bytes, source_ref: str, requirement_ids: list[str], *, complete: bool ) -> dict: - if not requirement_ids or len(set(requirement_ids)) != len(requirement_ids): + if not nonblank_strings(requirement_ids) or len(set(requirement_ids)) != len( + requirement_ids + ): raise ValueError( "supply the unique mandatory requirement IDs from the original scope" ) @@ -151,12 +172,10 @@ def _reuse(report: dict, current: dict, *, now: datetime | None = None) -> dict: required = current_scope.get("requirementIds", []) rows = evidence.get("evidence", []) if ( - not required + not nonblank_strings(required) or len(set(required)) != len(required) or sorted(row.get("requirementId", "") for row in rows) != sorted(required) - or any( - not row.get("evidenceRefs") or not all(row["evidenceRefs"]) for row in rows - ) + or any(not nonblank_strings(row.get("evidenceRefs")) for row in rows) ): reasons.append("mandatory requirement evidence missing") target = current.get("completionTarget", {}) diff --git a/tests/test_portable_acceptance.py b/tests/test_portable_acceptance.py index 03b5aa4034..de48e7d9cf 100644 --- a/tests/test_portable_acceptance.py +++ b/tests/test_portable_acceptance.py @@ -122,3 +122,132 @@ def test_explicit_target_preserves_detached_dirty_checkout(verified_candidate): assert target["subject"]["contentDigest"] != current["subject"]["contentDigest"] assert git(repo, "diff") == before assert git(repo, "branch", "--show-current") == "" + + +@pytest.mark.parametrize( + "refs", ["artifact:check", {}, None, [], [""], [" \t"], [17], [["check"]]] +) +def test_reuse_cli_rejects_malformed_evidence_refs(verified_candidate, tmp_path, refs): + repo, current, report = verified_candidate + git(repo, "add", ".") + git(repo, "commit", "-m", "verified candidate") + git(repo, "update-ref", "refs/heads/release", "HEAD") + current.update(acceptance.capture(repo, "example/repo", "release")) + assert acceptance.reuse(report, current)["completionEligible"] + report["validatedRefs"]["acceptance"]["evidence"][0]["evidenceRefs"] = refs + report_path = tmp_path / "report.json" + current_path = tmp_path / "current.json" + report_path.write_text(json.dumps(report)) + current_path.write_text(json.dumps(current)) + decision = json.loads( + subprocess.check_output( + [ + sys.executable, + str(SCRIPT), + "reuse", + "--report", + str(report_path), + "--current", + str(current_path), + ], + text=True, + ) + ) + assert not decision["reusable"] and not decision["completionEligible"] + assert decision["reasons"] + + +@pytest.mark.parametrize( + "ids", [[], [""], [" \t"], ["AC-1", ""], ["AC-1", "AC-1"], [17], "AC-1"] +) +def test_scope_and_reuse_reject_invalid_mandatory_ids(verified_candidate, ids): + _, current, report = verified_candidate + with pytest.raises(ValueError, match="mandatory requirement IDs"): + acceptance.scope(b"Print 42", "example/repo#1", ids, complete=True) + current["scope"]["requirementIds"] = ids + report["validatedRefs"]["acceptance"]["scope"] = copy.deepcopy(current["scope"]) + report["validatedRefs"]["acceptance"]["evidence"] = [ + {"requirementId": requirement_id, "evidenceRefs": ["artifact:check"]} + for requirement_id in ids + ] + decision = acceptance.reuse(report, current) + assert not decision["reusable"] and not decision["completionEligible"] + assert decision["reasons"] + + +@pytest.mark.parametrize("requirement", ["", " \t"]) +def test_capture_cli_rejects_blank_mandatory_id( + verified_candidate, tmp_path, requirement +): + repo, _, _ = verified_candidate + source = tmp_path / "issue.txt" + source.write_text("Print 42") + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "capture", + "--repo", + str(repo), + "--repository", + "example/repo", + "--target", + "release", + "--source", + str(source), + "--source-ref", + "example/repo#1", + "--requirement", + requirement, + ], + capture_output=True, + text=True, + ) + assert result.returncode != 0 + assert "mandatory requirement IDs" in result.stderr + assert not result.stdout + + +@pytest.mark.parametrize("registered", [False, True]) +@pytest.mark.parametrize("change", ["unstaged", "staged", "untracked"]) +def test_capture_rejects_dirty_gitlinks_without_changing_staging( + verified_candidate, registered, change +): + repo, _, _ = verified_candidate + nested = repo / "embedded repo" + nested.mkdir() + git(nested, "init", "-b", "main") + git(nested, "config", "user.email", "fixture@example.test") + git(nested, "config", "user.name", "Acceptance fixture") + source = nested / "nested.py" + source.write_text("print(1)\n") + git(nested, "add", ".") + git(nested, "commit", "-m", "nested baseline") + if registered: + git(repo, "submodule", "add", str(nested), nested.name) + git(repo, "submodule", "absorbgitdirs") + git(repo, "config", "submodule.embedded repo.ignore", "all") + clean = acceptance.capture(repo, "example/repo", "release") + git(repo, "add", "app.py") + if change == "untracked": + (nested / "new.py").write_text("print(2)\n") + else: + source.write_text("print(2)\n") + if change == "staged": + git(nested, "add", ".") + parent_index = git(repo, "ls-files", "--stage") + nested_index = git(nested, "ls-files", "--stage") + nested_status = git(nested, "status", "--porcelain", "--untracked-files=all") + objects_before = git(repo, "count-objects", "-v") + with pytest.raises(ValueError, match="content-bound workspace checkpoint"): + acceptance.capture(repo, "example/repo", "release") + assert git(repo, "ls-files", "--stage") == parent_index + assert git(nested, "ls-files", "--stage") == nested_index + assert ( + git(nested, "status", "--porcelain", "--untracked-files=all") == nested_status + ) + assert git(repo, "count-objects", "-v") == objects_before + git(nested, "add", ".") + git(nested, "commit", "-m", "nested verified change") + changed = acceptance.capture(repo, "example/repo", "release") + assert changed["subject"]["contentDigest"] != clean["subject"]["contentDigest"] From 9513cc41a28b84f7db2a2bb1d356028feceabbed Mon Sep 17 00:00:00 2001 From: Nate Sticco Date: Sat, 12 Sep 2026 01:40:41 -0700 Subject: [PATCH 5/5] Bound verification retries after prerequisite budget exhaustion --- bundle/skills/moonspec-verify/SKILL.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/bundle/skills/moonspec-verify/SKILL.md b/bundle/skills/moonspec-verify/SKILL.md index fd3e1d8d18..0f5ce16b1f 100644 --- a/bundle/skills/moonspec-verify/SKILL.md +++ b/bundle/skills/moonspec-verify/SKILL.md @@ -281,9 +281,15 @@ service and candidate. Persist readiness, setup actions, failures and resume commands with the inventory; continuation preserves this budget. Use available repository and managed container capabilities before declaring a dependency unavailable. A verification infrastructure failure preserves the same saved -candidate and requests evidence retry. A demonstrated code defect requests -remediation. Missing authority names the actual granting owner. These outcomes -must not be conflated or consume each other's budgets. +candidate. Request evidence retry only when an authorized setup attempt remains +or a concrete environment or authority change allows a controlling check to +obtain different evidence without another setup attempt. An exhausted setup +budget remains exhausted across continuation. If the prerequisite is unchanged +and no authorized recovery action remains, follow the Continuation Decision +rules to return `blocked`, or `needs_human` only for an actual human-owned +authority decision. A demonstrated code defect requests remediation. Missing +authority names the actual granting owner. These outcomes must not be conflated +or consume each other's budgets. Inspect production code before tests so behavior is verified directly.