diff --git a/CHANGELOG.md b/CHANGELOG.md index b9712467..3f76b220 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ All notable user-facing changes are tracked here. Public statements in release n - Strict typed Python surface: `py.typed`, `MatchStatus`, `Literal` policies, `TypedDict` summaries, a typed `TargetRow`, and mypy strict and pyright gates (`make typecheck`). - Example and snapshot testing flow: executable doctests, Rust `expect-test`-style snapshots, static API contract tests, and a typed quickstart (`make examples-test`). - The whole `dotmatch`/`quickdna` package now passes the strict typed gate, with Hypothesis invariants and a Python 3.9 runtime compatibility check (`make compat39`). -- Scientific reliability gate: scalar oracle equivalence across scan/indexed/exact/Hamming policies, stream count conservation, lossless integer count round-trips above float precision, gzip/plain FASTQ equivalence, seeded panel determinism, FASTQ/count/AssaySpec fuzzing, byte-deterministic assignment/panel/feature-matrix artifacts, indexed-vs-scan agreement, panel safety-vs-distance checks, unique-only feature-matrix counts, and mutation-sensitivity cases (`make scientific-tests`). See `docs/scientific-reliability.md`. +- Scientific reliability gate: scalar oracle equivalence across scan/indexed/exact/Hamming policies, stream count conservation, lossless integer count round-trips above float precision, gzip/plain FASTQ equivalence, seeded panel determinism, FASTQ/count/AssaySpec fuzzing, byte-deterministic assignment/panel/feature-matrix artifacts, indexed-vs-scan agreement, panel safety-vs-distance checks, unique-only feature-matrix counts, sensitivity artifact determinism, and mutation-sensitivity cases (`make scientific-tests`). See `docs/scientific-reliability.md`. ### Changed - Assignment result records (`MatchResult`, `AssignmentStats`, `PosteriorAssignment`, `FastqRecord`, `StreamAssignment`) are immutable `NamedTuple`s. Field access and keyword construction are unchanged. They are now tuple-compatible, so equality with a plain tuple is true; `dataclasses.asdict`, `fields`, and `replace` no longer apply. diff --git a/docs/scientific-reliability.md b/docs/scientific-reliability.md index ed700bfa..c1c03fbb 100644 --- a/docs/scientific-reliability.md +++ b/docs/scientific-reliability.md @@ -26,7 +26,8 @@ protects that contract; it does not by itself prove biological correctness. agreement, repeated-identifier agreement, gzip/plain equivalence, and target ID uniqueness are checked before assignment. - **Determinism.** Seeded panel design and feature-matrix artifacts are - reproducible for the same inputs; assignment TSVs are byte-identical. + reproducible for the same inputs; assignment TSVs are byte-identical, and + sensitivity artifacts are byte-identical apart from the recorded wall clock. - **Boundary fuzzing.** Generated valid and corrupted FASTQ, count TSV, and AssaySpec TOML inputs must either parse into typed records or raise a documented input error, never an unexpected exception. diff --git a/python/tests/test_scientific_invariants.py b/python/tests/test_scientific_invariants.py index fbfeec0d..001c622d 100644 --- a/python/tests/test_scientific_invariants.py +++ b/python/tests/test_scientific_invariants.py @@ -20,6 +20,7 @@ design_panel, hamming_distance, ) +from dotmatch.sensitivity import run_sensitivity from hypothesis import given, settings from hypothesis import strategies as st @@ -322,3 +323,39 @@ def test_feature_matrix_counts_only_unique_assignments( expected[key] = expected.get(key, 0) + 1 observed = {(row["cell_barcode"], row["target_id"]): int(row["count"]) for row in rows} assert observed == expected + + +def test_sensitivity_artifacts_are_deterministic_except_wall_clock(tmp_path: Path) -> None: + targets = tmp_path / "targets.tsv" + targets.write_text("target_id\ttarget_seq\nA\tACGTACGT\nB\tTTTTTTTT\n", encoding="utf-8") + reads = tmp_path / "reads.fastq" + reads.write_text( + "@r1\nACGTACGT\n+\nIIIIIIII\n" + "@r2\nACGTACGA\n+\nIIIIIIII\n" + "@r3\nTTTTTTTT\n+\nIIIIIIII\n", + encoding="ascii", + ) + first = run_sensitivity( + targets=targets, + reads=reads, + target_start=0, + target_length=8, + out_dir=tmp_path / "first", + batch_size=2, + ) + second = run_sensitivity( + targets=targets, + reads=reads, + target_start=0, + target_length=8, + out_dir=tmp_path / "second", + batch_size=2, + ) + + first.pop("execution", {}).pop("seconds", None) + second.pop("execution", {}).pop("seconds", None) + assert first == second + for name in sorted(path.name for path in (tmp_path / "first").iterdir()): + if name == "summary.json": + continue + assert (tmp_path / "first" / name).read_bytes() == (tmp_path / "second" / name).read_bytes(), name