diff --git a/CHANGELOG.md b/CHANGELOG.md index aba9bdf..2bf3b16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,8 @@ First release candidate (V1). Nothing has been tagged yet. Numbers are pull requ Makefile through uv plus pre-commit config (#168); schema-valid `CITATION.cff` (#122); status-first README (#151), docs index (#161), research-track roadmap (#162), Study 2 corrections (#124, #128) and the documented evaluation protocol (#132). +- Mutant catalogue and validation documentation, plus an acceptance test requiring package + mutant coverage for all ten mutation families or a documented infeasibility issue (issue #93). - This changelog and the [release procedure](docs/releasing.md). ### Changed diff --git a/README.md b/README.md index 2a4bf6c..3a669d1 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ What is implemented on `main` today. The [V1 design target](docs/methodology.md) | Task packages (4: oscillator, kepler, heat1d, wave1d) | Implemented | `tasks/` | [M2](https://github.com/akurkar07/InvariantLab/milestone/1) | | Reference solvers and oracles | Implemented | `src/invariantlab/verification/` | [M2](https://github.com/akurkar07/InvariantLab/milestone/1) | | Verification: Layers 0-6 (seven layers) and `invariantlab verify` | Implemented ([layer status](docs/verification.md#layer-status)) | `src/invariantlab/verification/` | [#110](https://github.com/akurkar07/InvariantLab/issues/110)-[#117](https://github.com/akurkar07/InvariantLab/issues/117) | -| Defect injection / mutants | Partial (registry; three validated package mutants, two legacy oscillator study mutants) | `src/invariantlab/mutations/`, `tasks/*/mutations/` | [M4](https://github.com/akurkar07/InvariantLab/milestone/3) | +| Defect injection / mutants | Partial (registry and validation; 13 package mutants covering all 10 defect families on all four tasks; two legacy oscillator study mutants; [catalogue](docs/methodology.md#controlled-defect-injection)) | `src/invariantlab/mutations/`, `tasks/*/mutations/` | [M4](https://github.com/akurkar07/InvariantLab/milestone/3) | | Model adapters | Implemented | `src/invariantlab/models/adapter.py` | [M5](https://github.com/akurkar07/InvariantLab/milestone/4) | | Repair runner + `audit-run` | Implemented | `src/invariantlab/experiments/repair.py` | [M5](https://github.com/akurkar07/InvariantLab/milestone/4) | | Reporting / dashboard / HF export | Partial (`report`, static `report.html`, local HF export via `export-hf`) | `src/invariantlab/reporting/` | [#106](https://github.com/akurkar07/InvariantLab/issues/106), [#107](https://github.com/akurkar07/InvariantLab/issues/107) | @@ -100,7 +100,6 @@ It currently provides: It does **not** provide yet: -- controlled mutants for kepler and heat1d (only the oscillator and wave1d have package mutants) - plots, interactive filtering or a served dashboard (the static `report.html` is the V1 dashboard) ## Empirical results diff --git a/docs/experiment-authoring.md b/docs/experiment-authoring.md index 14c029a..f22adef 100644 --- a/docs/experiment-authoring.md +++ b/docs/experiment-authoring.md @@ -82,6 +82,10 @@ tasks//mutations// └── solver.py ``` +An **operator**, in the M4 milestone's terminology, is a defect family +(`invariantlab.schema.MutationFamily`) realised as one or more curated mutant directories; +V1 has no AST/programmatic operators. + Example: ```yaml @@ -93,25 +97,41 @@ source: solver.py expected_effect: stale acceleration in the second velocity half-step ``` +`MutationDefinition` (`invariantlab.schema.MutationDefinition`) accepts these manifest +fields; extra fields are rejected: + +| Field | Meaning | +|---|---| +| `id` | Mutant id; must match its directory name. | +| `task_id` | Task contract id. | +| `family` | A `MutationFamily` defect family. | +| `interface` | `package` (default) or `legacy_study`. | +| `source` | Solver source path inside the mutant directory (default `solver.py`). | +| `expected_effect` | Required, non-empty description of the intended defect. | +| `expected_failures` | List of `{test: scientific pytest node id, message: regex}` entries; required and non-empty for `package` mutants. | +| `max_changed_lines` | Maximum added plus removed diff lines (default 10; must be at least 1). | + The mutation identifier is written to run records. The directory path is only configuration, so moving from the legacy `mutation: update-order` field to a path does not change the Study 2 record identifier. -Package mutants use `interface: package`, declare `expected_failures` entries with a -scientific pytest node id in `test` and an output regex in `message`, and may set -`max_changed_lines` (default 10). `invariantlab.mutations.discover_mutants` validates -the manifests and returns registered mutants; it raises `MutationRegistryError` with -all discovered problems when any declaration is invalid. - -`invariantlab.mutations.validate_reference` requires at least one testcase in each public -and scientific suite and no failures, errors, or skips. `validate_mutant` verifies that -(1) public tests pass, (2) every declared scientific failure occurs with its declared -message and there are no unexpected failures or errors, and (3) the parseable mutant source -diff stays within its line budget and imports no `invariantlab` modules. These implement -the four reference/mutant validation checks; each suite timeout defaults to -`budgets.wall_seconds`. Both run tests on the repository virtual environment against a -temporary task copy without `mutations/`; their stable result names are consumed by #64 -and #88. +`invariantlab.mutations.discover_mutants` validates the manifests and returns registered +mutants; it raises `MutationRegistryError` with all discovered problems when any declaration +is invalid. `validate_reference` and `validate_mutant` enforce four checks: (1) reference +public and scientific suites each collect at least one test and have no failures, errors, or +skips; (2) a mutant's public suite has no failures or errors and at least one passing test; +(3) failing scientific node ids equal the declared `expected_failures`, each is a JUnit +failure (not an error or skip) whose message matches its declared regex, and any error fails; +(4) mutant source differs from `src/solver.py`, its unified diff stays within +`max_changed_lines` added plus removed lines, it parses, and it imports no `invariantlab` +modules. Each suite timeout defaults to `budgets.wall_seconds`. Both run tests on the +repository virtual environment against a temporary task copy without `mutations/`; their +stable result names are consumed by #64 and #88. + +Run mutant validation with `uv run python scripts/validate_mutants.py --task-dir tasks/ +[--task ]`. It runs in the Task Validation workflow's `validate` CI job and through +`make validate-tasks`; it fails closed (exit 1) on zero package mutants, registry errors, or +any failed reference or mutant validation. Curated package mutants (each passes `validate_mutant`; checked by `tests/unit/test_mutation_registry.py::test_real_task_package_mutants_validate`): @@ -144,6 +164,11 @@ The wave1d public and scientific suites reject unstable CFL with the same input so a partly loosened CFL guard fails no scientific test; the wave1d `stability_error` mutant is a scheme-level instability instead. +When adding a mutant, also add it to +`test_real_task_package_mutants_validate`'s pinned list and this catalogue. +`tests/acceptance/test_mutation_coverage.py` enforces package-mutant coverage of every +`MutationFamily` (or a documented infeasibility issue). + ## Add a model Create a YAML file under `configs/models/`; its `adapter` field selects the backend. diff --git a/docs/limitations.md b/docs/limitations.md index fa866de..394c0e3 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -31,8 +31,8 @@ that tracks it. The [CHANGELOG](../CHANGELOG.md) links this page from its `update-order` / `sign-error` mutants are still `interface: legacy_study` and are graded by the legacy oscillator study harness (`tasks/oscillator/candidate_runner.py` + `verifier.py`). `task_suite` is still ignored. -- **Mutant coverage.** Package mutants exist for `oscillator`, `kepler` and `wave1d`; `heat1d` - has none yet. Covered families: `sign_error`, `update_order_error`, +- **Mutant coverage.** Package mutants exist for all four tasks; `heat1d` has no + `boundary_error` mutant (any wall defect fails its public example). Covered families: `sign_error`, `update_order_error`, `non_conservative_update`, `unit_error`, `hard_coded_shortcut`, `precision_defect`, `termination_defect`, `discretisation_error`, `boundary_error` and `stability_error`. - **Reporting.** The static `report.html` is the V1 dashboard: no plots, interactive diff --git a/docs/methodology.md b/docs/methodology.md index 8815acf..e992c30 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -43,6 +43,49 @@ public-test success scientific correctness ``` +## Controlled defect injection + +The benchmark uses deterministic, reviewable curated mutants to inject controlled defects. + +An **operator**, in the M4 milestone's terminology, is a defect family +(`invariantlab.schema.MutationFamily`, with 10 values), realised as one or more curated +mutant directories at `tasks//mutations//` containing `mutation.yaml` and +`solver.py`; `solver.py` is a full replacement for `src/solver.py`. V1 has no +AST/programmatic operators. + +| Mutation family | Example | Typical scientific symptom | Committed mutants | +|---|---|---|---| +| `sign_error` | Reverse a force or derivative sign | Trajectory evolves in the wrong direction | `wave1d/sign-error-startup` | +| `update_order_error` | Overwrite the previous state before the next step | Recurrence uses a stale or incorrect state | `wave1d/update-order-overwrite` | +| `boundary_error` | Apply a boundary condition at the wrong node | Boundary value or spatial profile is incorrect | `wave1d/dirichlet-wrong-node` | +| `discretisation_error` | Omit the square on the Courant number | Wave amplitude or propagation speed is wrong | `wave1d/courant-not-squared`, `heat1d/discretisation-grid-spacing` | +| `stability_error` | Change the sign in a leapfrog recurrence | Numerical error grows exponentially with time | `wave1d/unstable-time-recurrence`, `heat1d/sixth-order-stencil-ftcs-limit` | +| `unit_error` | Round the astronomical-unit conversion | Orbital trajectory has a systematic scale error | `kepler/unit-error-au-rounding` | +| `non_conservative_update` | Dampen a velocity update | Energy drifts over an otherwise conservative trajectory | `kepler/non-conservative-velocity-damping`, `oscillator/non-conservative-damping` | +| `hard_coded_shortcut` | Special-case public fixture sizes | Correct-looking public fixtures, incorrect general inputs | `oscillator/hard-coded-fixture-shortcut` | +| `precision_defect` | Round position updates through `float32` | Small per-step errors accumulate | `oscillator/float32-position` | +| `termination_defect` | Stop integration before the requested final step | Output ends early or repeats a stale state | `oscillator/early-termination` | + +Heat1d has no `boundary_error` mutant: any wall defect fails its public example, so that +family is covered on wave1d ([issue #90](https://github.com/akurkar07/InvariantLab/issues/90)). + +`validate_reference` and `validate_mutant` enforce four checks: + +1. **Reference:** public and scientific suites each collect at least one test and have no + failures, errors, or skips. +2. **Public profile:** a mutant's public suite has no failures or errors and at least one + passing test. +3. **Expected scientific reason:** the failing scientific node ids equal the declared + `expected_failures`; each is a JUnit failure (not an error or skip) whose message matches + its declared regex. Any error fails validation. +4. **Controlled defect:** mutant source differs from `src/solver.py`, its unified diff stays + within `max_changed_lines` added plus removed lines, it parses, and it imports no + `invariantlab` modules. + +`scripts/validate_mutants.py` runs these checks in CI. See +[Add a mutation](experiment-authoring.md#add-a-mutation) for the manifest schema and +validation command. + ## Evaluation protocol V1 measures **one-shot repair of a mutated solver**. The implemented runner is diff --git a/tests/acceptance/test_mutation_coverage.py b/tests/acceptance/test_mutation_coverage.py new file mode 100644 index 0000000..b6d66d7 --- /dev/null +++ b/tests/acceptance/test_mutation_coverage.py @@ -0,0 +1,49 @@ +"""Every MutationFamily is realised by at least one registered package mutant.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from invariantlab.mutations import discover_mutants +from invariantlab.schema import MutationFamily + +REPO_ROOT = Path(__file__).resolve().parents[2] +# Families with no feasible package mutant must cite the issue where infeasibility was found. +INFEASIBLE_FAMILIES: dict[MutationFamily, str] = {} + + +@pytest.fixture +def covered_families(monkeypatch: pytest.MonkeyPatch) -> set[MutationFamily]: + monkeypatch.chdir(REPO_ROOT) + return {mutant.definition.family for mutant in discover_mutants(Path("tasks"))} + + +@pytest.mark.parametrize("family", list(MutationFamily), ids=lambda family: family.value) +def test_every_mutation_family_has_a_mutant( + family: MutationFamily, covered_families: set[MutationFamily] +) -> None: + assert family in covered_families or family in INFEASIBLE_FAMILIES, ( + f"{family.value!r} has no package mutant; add one or cite its infeasibility issue " + "in INFEASIBLE_FAMILIES." + ) + + +def test_infeasible_families_have_no_mutants( + covered_families: set[MutationFamily], +) -> None: + overlap = covered_families.intersection(INFEASIBLE_FAMILIES) + assert not overlap, ( + f"infeasible families already have mutants: {sorted(f.value for f in overlap)}" + ) + + +def test_catalogue_lists_every_package_mutant(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(REPO_ROOT) + catalogue = (REPO_ROOT / "docs" / "experiment-authoring.md").read_text(encoding="utf-8") + for mutant in discover_mutants(Path("tasks")): + row = f"| {mutant.task_dir.name} | `{mutant.definition.id}` |" + assert row in catalogue, ( + f"catalogue is missing {mutant.task_dir.name}/{mutant.definition.id}" + )