M4-Docs: Document the mutant catalogue and enforce 10-family coverage

CHANGELOG.md

@@ -66,6 +66,8 @@ First release candidate (V1). Nothing has been tagged yet. Numbers are pull requ
   Makefile through uv plus pre-commit config (#168); schema-valid `CITATION.cff` (#122);
   status-first README (#151), docs index (#161), research-track roadmap (#162), Study 2
   corrections (#124, #128) and the documented evaluation protocol (#132).
+- Mutant catalogue and validation documentation, plus an acceptance test requiring package
+  mutant coverage for all ten mutation families or a documented infeasibility issue (issue #93).
 - This changelog and the [release procedure](docs/releasing.md).
 
 ### Changed

README.md

@@ -24,7 +24,7 @@ What is implemented on `main` today. The [V1 design target](docs/methodology.md)
 | Task packages (4: oscillator, kepler, heat1d, wave1d) | Implemented | `tasks/` | [M2](https://github.com/akurkar07/InvariantLab/milestone/1) |
 | Reference solvers and oracles | Implemented | `src/invariantlab/verification/` | [M2](https://github.com/akurkar07/InvariantLab/milestone/1) |
 | Verification: Layers 0-6 (seven layers) and `invariantlab verify` | Implemented ([layer status](docs/verification.md#layer-status)) | `src/invariantlab/verification/` | [#110](https://github.com/akurkar07/InvariantLab/issues/110)-[#117](https://github.com/akurkar07/InvariantLab/issues/117) |
-| Defect injection / mutants | Partial (registry; three validated package mutants, two legacy oscillator study mutants) | `src/invariantlab/mutations/`, `tasks/*/mutations/` | [M4](https://github.com/akurkar07/InvariantLab/milestone/3) |
+| Defect injection / mutants | Partial (registry and validation; 13 package mutants covering all 10 defect families on all four tasks; two legacy oscillator study mutants; [catalogue](docs/methodology.md#controlled-defect-injection)) | `src/invariantlab/mutations/`, `tasks/*/mutations/` | [M4](https://github.com/akurkar07/InvariantLab/milestone/3) |
 | Model adapters | Implemented | `src/invariantlab/models/adapter.py` | [M5](https://github.com/akurkar07/InvariantLab/milestone/4) |
 | Repair runner + `audit-run` | Implemented | `src/invariantlab/experiments/repair.py` | [M5](https://github.com/akurkar07/InvariantLab/milestone/4) |
 | Reporting / dashboard / HF export | Partial (`report`, static `report.html`, local HF export via `export-hf`) | `src/invariantlab/reporting/` | [#106](https://github.com/akurkar07/InvariantLab/issues/106), [#107](https://github.com/akurkar07/InvariantLab/issues/107) |
@@ -100,7 +100,6 @@ It currently provides:
 
 It does **not** provide yet:
 
-- controlled mutants for kepler and heat1d (only the oscillator and wave1d have package mutants)
 - plots, interactive filtering or a served dashboard (the static `report.html` is the V1 dashboard)
 
 ## Empirical results

docs/experiment-authoring.md

@@ -82,6 +82,10 @@ tasks/<task>/mutations/<mutation>/
 └── solver.py
 ```
 
+An **operator**, in the M4 milestone's terminology, is a defect family
+(`invariantlab.schema.MutationFamily`) realised as one or more curated mutant directories;
+V1 has no AST/programmatic operators.
+
 Example:
 
 ```yaml
@@ -93,25 +97,41 @@ source: solver.py
 expected_effect: stale acceleration in the second velocity half-step
 ```
 
+`MutationDefinition` (`invariantlab.schema.MutationDefinition`) accepts these manifest
+fields; extra fields are rejected:
+
+| Field | Meaning |
+|---|---|
+| `id` | Mutant id; must match its directory name. |
+| `task_id` | Task contract id. |
+| `family` | A `MutationFamily` defect family. |
+| `interface` | `package` (default) or `legacy_study`. |
+| `source` | Solver source path inside the mutant directory (default `solver.py`). |
+| `expected_effect` | Required, non-empty description of the intended defect. |
+| `expected_failures` | List of `{test: scientific pytest node id, message: regex}` entries; required and non-empty for `package` mutants. |
+| `max_changed_lines` | Maximum added plus removed diff lines (default 10; must be at least 1). |
+
 The mutation identifier is written to run records. The directory path is only configuration,
 so moving from the legacy `mutation: update-order` field to a path does not change the
 Study 2 record identifier.
 
-Package mutants use `interface: package`, declare `expected_failures` entries with a
-scientific pytest node id in `test` and an output regex in `message`, and may set
-`max_changed_lines` (default 10). `invariantlab.mutations.discover_mutants` validates
-the manifests and returns registered mutants; it raises `MutationRegistryError` with
-all discovered problems when any declaration is invalid.
-
-`invariantlab.mutations.validate_reference` requires at least one testcase in each public
-and scientific suite and no failures, errors, or skips. `validate_mutant` verifies that
-(1) public tests pass, (2) every declared scientific failure occurs with its declared
-message and there are no unexpected failures or errors, and (3) the parseable mutant source
-diff stays within its line budget and imports no `invariantlab` modules. These implement
-the four reference/mutant validation checks; each suite timeout defaults to
-`budgets.wall_seconds`. Both run tests on the repository virtual environment against a
-temporary task copy without `mutations/`; their stable result names are consumed by #64
-and #88.
+`invariantlab.mutations.discover_mutants` validates the manifests and returns registered
+mutants; it raises `MutationRegistryError` with all discovered problems when any declaration
+is invalid. `validate_reference` and `validate_mutant` enforce four checks: (1) reference
+public and scientific suites each collect at least one test and have no failures, errors, or
+skips; (2) a mutant's public suite has no failures or errors and at least one passing test;
+(3) failing scientific node ids equal the declared `expected_failures`, each is a JUnit
+failure (not an error or skip) whose message matches its declared regex, and any error fails;
+(4) mutant source differs from `src/solver.py`, its unified diff stays within
+`max_changed_lines` added plus removed lines, it parses, and it imports no `invariantlab`
+modules. Each suite timeout defaults to `budgets.wall_seconds`. Both run tests on the
+repository virtual environment against a temporary task copy without `mutations/`; their
+stable result names are consumed by #64 and #88.
+
+Run mutant validation with `uv run python scripts/validate_mutants.py --task-dir tasks/
+[--task <name>]`. It runs in the Task Validation workflow's `validate` CI job and through
+`make validate-tasks`; it fails closed (exit 1) on zero package mutants, registry errors, or
+any failed reference or mutant validation.
 
 Curated package mutants (each passes `validate_mutant`; checked by
 `tests/unit/test_mutation_registry.py::test_real_task_package_mutants_validate`):
@@ -144,6 +164,11 @@ The wave1d public and scientific suites reject unstable CFL with the same input
 so a partly loosened CFL guard fails no scientific test; the wave1d `stability_error` mutant
 is a scheme-level instability instead.
 
+When adding a mutant, also add it to
+`test_real_task_package_mutants_validate`'s pinned list and this catalogue.
+`tests/acceptance/test_mutation_coverage.py` enforces package-mutant coverage of every
+`MutationFamily` (or a documented infeasibility issue).
+
 ## Add a model
 
 Create a YAML file under `configs/models/`; its `adapter` field selects the backend.

docs/limitations.md

@@ -31,8 +31,8 @@ that tracks it. The [CHANGELOG](../CHANGELOG.md) links this page from its
   `update-order` / `sign-error` mutants are still `interface: legacy_study` and are graded
   by the legacy oscillator study harness (`tasks/oscillator/candidate_runner.py` +
   `verifier.py`). `task_suite` is still ignored.
-- **Mutant coverage.** Package mutants exist for `oscillator`, `kepler` and `wave1d`; `heat1d`
-  has none yet. Covered families: `sign_error`, `update_order_error`,
+- **Mutant coverage.** Package mutants exist for all four tasks; `heat1d` has no
+  `boundary_error` mutant (any wall defect fails its public example). Covered families: `sign_error`, `update_order_error`,
   `non_conservative_update`, `unit_error`, `hard_coded_shortcut`, `precision_defect`,
   `termination_defect`, `discretisation_error`, `boundary_error` and `stability_error`.
 - **Reporting.** The static `report.html` is the V1 dashboard: no plots, interactive

docs/methodology.md

@@ -43,6 +43,49 @@ public-test success
 scientific correctness
 ```
 
+## Controlled defect injection
+
+The benchmark uses deterministic, reviewable curated mutants to inject controlled defects.
+
+An **operator**, in the M4 milestone's terminology, is a defect family
+(`invariantlab.schema.MutationFamily`, with 10 values), realised as one or more curated
+mutant directories at `tasks/<task>/mutations/<id>/` containing `mutation.yaml` and
+`solver.py`; `solver.py` is a full replacement for `src/solver.py`. V1 has no
+AST/programmatic operators.
+
+| Mutation family | Example | Typical scientific symptom | Committed mutants |
+|---|---|---|---|
+| `sign_error` | Reverse a force or derivative sign | Trajectory evolves in the wrong direction | `wave1d/sign-error-startup` |
+| `update_order_error` | Overwrite the previous state before the next step | Recurrence uses a stale or incorrect state | `wave1d/update-order-overwrite` |
+| `boundary_error` | Apply a boundary condition at the wrong node | Boundary value or spatial profile is incorrect | `wave1d/dirichlet-wrong-node` |
+| `discretisation_error` | Omit the square on the Courant number | Wave amplitude or propagation speed is wrong | `wave1d/courant-not-squared`, `heat1d/discretisation-grid-spacing` |
+| `stability_error` | Change the sign in a leapfrog recurrence | Numerical error grows exponentially with time | `wave1d/unstable-time-recurrence`, `heat1d/sixth-order-stencil-ftcs-limit` |
+| `unit_error` | Round the astronomical-unit conversion | Orbital trajectory has a systematic scale error | `kepler/unit-error-au-rounding` |
+| `non_conservative_update` | Dampen a velocity update | Energy drifts over an otherwise conservative trajectory | `kepler/non-conservative-velocity-damping`, `oscillator/non-conservative-damping` |
+| `hard_coded_shortcut` | Special-case public fixture sizes | Correct-looking public fixtures, incorrect general inputs | `oscillator/hard-coded-fixture-shortcut` |
+| `precision_defect` | Round position updates through `float32` | Small per-step errors accumulate | `oscillator/float32-position` |
+| `termination_defect` | Stop integration before the requested final step | Output ends early or repeats a stale state | `oscillator/early-termination` |
+
+Heat1d has no `boundary_error` mutant: any wall defect fails its public example, so that
+family is covered on wave1d ([issue #90](https://github.com/akurkar07/InvariantLab/issues/90)).
+
+`validate_reference` and `validate_mutant` enforce four checks:
+
+1. **Reference:** public and scientific suites each collect at least one test and have no
+   failures, errors, or skips.
+2. **Public profile:** a mutant's public suite has no failures or errors and at least one
+   passing test.
+3. **Expected scientific reason:** the failing scientific node ids equal the declared
+   `expected_failures`; each is a JUnit failure (not an error or skip) whose message matches
+   its declared regex. Any error fails validation.
+4. **Controlled defect:** mutant source differs from `src/solver.py`, its unified diff stays
+   within `max_changed_lines` added plus removed lines, it parses, and it imports no
+   `invariantlab` modules.
+
+`scripts/validate_mutants.py` runs these checks in CI. See
+[Add a mutation](experiment-authoring.md#add-a-mutation) for the manifest schema and
+validation command.
+
 ## Evaluation protocol
 
 V1 measures **one-shot repair of a mutated solver**. The implemented runner is

tests/acceptance/test_mutation_coverage.py

@@ -0,0 +1,49 @@
+"""Every MutationFamily is realised by at least one registered package mutant."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from invariantlab.mutations import discover_mutants
+from invariantlab.schema import MutationFamily
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+# Families with no feasible package mutant must cite the issue where infeasibility was found.
+INFEASIBLE_FAMILIES: dict[MutationFamily, str] = {}
+
+
+@pytest.fixture
+def covered_families(monkeypatch: pytest.MonkeyPatch) -> set[MutationFamily]:
+    monkeypatch.chdir(REPO_ROOT)
+    return {mutant.definition.family for mutant in discover_mutants(Path("tasks"))}
+
+
+@pytest.mark.parametrize("family", list(MutationFamily), ids=lambda family: family.value)
+def test_every_mutation_family_has_a_mutant(
+    family: MutationFamily, covered_families: set[MutationFamily]
+) -> None:
+    assert family in covered_families or family in INFEASIBLE_FAMILIES, (
+        f"{family.value!r} has no package mutant; add one or cite its infeasibility issue "
+        "in INFEASIBLE_FAMILIES."
+    )
+
+
+def test_infeasible_families_have_no_mutants(
+    covered_families: set[MutationFamily],
+) -> None:
+    overlap = covered_families.intersection(INFEASIBLE_FAMILIES)
+    assert not overlap, (
+        f"infeasible families already have mutants: {sorted(f.value for f in overlap)}"
+    )
+
+
+def test_catalogue_lists_every_package_mutant(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.chdir(REPO_ROOT)
+    catalogue = (REPO_ROOT / "docs" / "experiment-authoring.md").read_text(encoding="utf-8")
+    for mutant in discover_mutants(Path("tasks")):
+        row = f"| {mutant.task_dir.name} | `{mutant.definition.id}` |"
+        assert row in catalogue, (
+            f"catalogue is missing {mutant.task_dir.name}/{mutant.definition.id}"
+        )