Repository navigation
M4-Docs: Document the mutant catalogue and enforce 10-family coverage - #183
Merged
Merged
Conversation
…ants (#89) Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # docs/experiment-authoring.md # tests/unit/test_mutation_registry.py
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # docs/experiment-authoring.md # tests/unit/test_mutation_registry.py
…tencil-ftcs-limit (#90) Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
… (issue #93) Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
…nto devin/issue-93-mutant-catalogue-docs
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
5 tasks
akurkar07
added a commit
that referenced
this pull request
Oct 6, 2026
Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
7 tasks done
akurkar07
added a commit
that referenced
this pull request
Oct 6, 2026
* M7-Release: Prepare the v1.0.0 release metadata (#72) Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> * CHANGELOG: cite PR #183 for the mutant catalogue entry Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the M4 mutant catalogue as it actually exists on
mainand adds an acceptance test so family coverage cannot silently regress.docs/methodology.md: new## Controlled defect injectionsection (the README section moved here under Docs: Restructure the README into status, quickstart and links to the V1 design spec #96's 300-line README rule). It defines "operator" as a defect family realised by curated mutant directories, gives the 10-family table with a Committed mutants column (task/id per family; heat1d planned via M4-Heat: Add boundary, discretisation and stability mutants for heat1d #90), and replaces the old "three checks" + property-test sentence with the four checksvalidate_reference/validate_mutantactually enforce.docs/experiment-authoring.md"Add a mutation":MutationDefinitionmanifest field table (M4-Registry: Define the mutant manifest and discover every task's mutants #85), operator definition, numbered four checks (M4-Validation: Prove each mutant passes public tests and fails its declared scientific gate #86),scripts/validate_mutants.pyusage and fail-closed behaviour (M4-CI: Make validate_mutants.py validate every mutant and fail closed #88), and a note on what to update when adding a mutant.README.md: the status row now says 11 package mutants cover all 10 families and links the catalogue; the "does not provide yet" line says heat1d only (kepler gained mutants in M4-Kepler: Add unit-error and non-conservative mutants for the two-body task #172).tests/acceptance/test_mutation_coverage.py: usesdiscover_mutants(Path("tasks"))to assert that everyMutationFamilyhas a package mutant.INFEASIBLE_FAMILIESis an explicit, commented, currently empty exception dict whose entries must cite an issue. Two more checks: a test fails if an exempted family gains a mutant, and every package mutant must appear in the experiment-authoring catalogue table.CHANGELOG.md: Unreleased entry.Closes #93
Proposed milestone description (maintainer edit, M4)
Acceptance criteria
docs/experiment-authoring.md"Add a mutation" documents the M4-Registry: Define the mutant manifest and discover every task's mutants #85 manifest fields, the operator definition, the four M4-Validation: Prove each mutant passes public tests and fails its declared scientific gate #86 checks andscripts/validate_mutants.py(M4-CI: Make validate_mutants.py validate every mutant and fail closed #88). Checked againstschema.MutationDefinition,mutations/validation.py, the Task Validation workflow and the Makefile.docs/methodology.mdbecause Docs: Restructure the README into status, quickstart and links to the V1 design spec #96 moved it out of the README, and the README status line was kept consistent.tests/acceptance/test_mutation_coverage.pypasses. Negative check: temporarily removingwave1d/unstable-time-recurrencemade thestability_errorcase fail with the uncovered-family message.git grep -n "9 defect\|9 operators"returns nothing.uv run pytest -qpasses (see below).Note: heat1d mutants (#90) are still open. All 10 families are already covered by oscillator, kepler and wave1d mutants, so this does not block the issue.
Test output
uv run pytest tests/acceptance/test_mutation_coverage.py tests/unit/test_docs_index.py -q: 15 passeduv run pytest -q: 619 passed, 3 skipped (Docker-backed tests ran)uv run ruff check .: clean.uv run ruff format --check src tests tasks scripts: 128 files already formatted.uv run mypy src: no issues in 32 source files.Written by Devin