Run the pipeline for real in CI, on a cohort whose truth is known

[Paururo]

What changed and why

Every test so far stopped short of the science. -stub-run replaces each process's script block, and the unit tests run bin/ one script at a time, so no script block had any coverage. A new CI job, End-to-end run (Docker), runs BAMpiro for real, in its pinned images, on a cohort that tests/e2e/cohort.py simulates at test time, and holds every deliverable against what was planted.

The cohort. One seed, about a second to write, 2 MB, not committed: a 24 kb genome with five genes and three samples read at 100x.

Sample	Layout	Planted	The run must report
S1	single-end, two runs	Fixed SNPs; two SNPs of one codon of a minus-strand gene on the same reads (Phe>Met); a fixed in-frame deletion; S2's frameshift in 20% of the reads, all on the forward strand	Phe100Met read whole; p.Arg135del; the frameshift kept as LowSupport
S2	paired-end	Fixed SNPs; His>Glu on the same reads; a fixed frameshift insertion; a SNP in one copy of a repeat	His145Glu; the insertion PASS; the repeat SNP never called
S3	paired-end, a mixture	Fixed SNPs; a lost 1.5 kb stretch holding a gene. At 25%: a SNP, Glu>Leu (first base alone: a stop), a frameshift deletion. At 30% and 25%: two SNPs of one codon on different molecules	A private deletion naming the gene; MNV-masked; the unphased pair not merged

What the run is held to (17 checks).

• Each sample's main VCF holds exactly the SNPs planted in it, each annotated base by base.
• The SNP matrix gives each planted SNP its fraction and protein change.
• The codon-level table and its marks in the SNP matrix.
• The indel matrix gives each call its fraction and filter.
• The consensus: fixed SNPs written, minorities never written as a base, the repeat masked, the lost stretch left uncalled.
• The SNP distances: 11, 9 and 10.
• The deletions table.
• The report's own copy of the matrices.
• The recorded tool versions.

What it found on its first runs.

• GET_MNV failed on every sample in a real run. The get_MNV image has no tabix, so the .tbi the process declared never appeared. This was fixed in Deliver indels, and read two SNPs of one codon as one change #13 before it merged (d9a667c).
• software_versions.txt recorded a message as two tools' versions. For bwa-mem2 it kept the SIMD banner the tool prints first, and under the docker profile it kept genmap's error about an unwritable home. Fixed here: the version is the first line that carries a version number, and the step gets a writable home when it has none.
• The SNP matrix and the report list a site no sample calls. Position 13149 has two error reads (5%) at the edge of the masked repeat in S2. It is listed because both read FreeBayes' raw VCF, which keeps every site FreeBayes weighed. This is pinned as today's behaviour and not changed here, because outputs.md says a row is a called site and changing it deserves its own change.

Outputs and defaults

• What a run produces: nothing changes except pipeline_info/software_versions.txt, which now records real versions for bwa-mem2 and, under docker, genmap.
• New profile test_e2e (conf/test_e2e.config): production thresholds, resources capped to a GitHub runner, used as -profile docker,test_e2e.
• New CI job on every pull request:
  • about 2.5 minutes, pulling about 1.3 GB of images;
  • when it fails, it keeps the cohort, results, trace and log as the e2e-run artifact.
• Locally: tests/run_tests.sh e2e, or BAMPIRO_E2E=1 python -m pytest tests/e2e. It needs Docker and is not part of the default tests/run_tests.sh.
• CI comments no longer call the repository private or its minutes metered.

How it was verified

# CI on this branch (workflow_dispatch, run 36045165294): all 8 jobs pass
#   End-to-end run (Docker): 17 passed in 137.92s
ruff check .              # clean
mkdocs build --strict     # passes

• Before the first CI run, the cohort's truth was checked against the tools themselves:
  • bwa and FreeBayes, with the pipeline's settings, make exactly the planted calls (25 SNP, 4 indel) and no other;
  • get_MNV reads the three codon changes and gives the unphased pair no read;
  • SnpEff writes the 15 protein changes the cohort computes.
• The table checks were run against tables built locally with the pipeline's own scripts. Each check catches a deliberately broken copy: a dropped codon change, a LowSupport turned PASS, a SNP nobody carries.
• The branch's CI runs, in order:
  1. The first found the .tbi bug.
  2. The second failed on my own escape in modules/qc.nf.
  3. The third ran the whole pipeline and failed 3 checks. All three were the test's own expectations, now corrected; the corrected checks pass against that run's artifact.
  4. The fourth is green.
• Not covered:
  • the TB-specific branches (pathotypr, canonical annotation, liftover, blind spots, Kraken, gene conversion), since the genome is synthetic and only the default features run;
  • Singularity;
  • Nextflow's minimum version, since the job uses latest-stable.

Checklist

• tests/run_tests.sh passes: every leg in CI; locally lint, the default stub run and the parameter tests (no Docker daemon here, so the e2e leg ran in CI)
• ruff check . is clean, and I did not restyle code the change does not touch
• New or changed behaviour has a test, and tests pin what the code does today (the versions fix is held by the e2e check; the matrix's extra site is pinned, with a note)
• A new process in modules/ has a stub: block, and -profile test_full reaches it (not applicable: no new process)
• tests/data/ was not edited by hand (the e2e cohort is written at test time and not committed)
• User-visible changes have a CHANGELOG.md entry and, where relevant, a docs update