Skip to content

Draw each reference's own genes under its own samples - #10

Merged
Paururo merged 3 commits into
mainfrom
fix/per-reference-genes
Sep 23, 2026
Merged

Paururo merged 3 commits into
mainfrom
fix/per-reference-genes

Conversation

@Paururo

@Paururo Paururo commented Sep 23, 2026

Copy link
Copy Markdown
Member

What changed and why

The report's genome page took the GFF, the masked regions and the length of the first reference of the run for every sample. On the 185-sample cohort mapped against two assemblies, E1ASM0057's 72 samples were drawn over E1ASM0035's genes. The variable-genes table, go-to-gene, the masked regions and the callability insight all came from E1ASM0035.

  • Every reference's GFF, masked regions and FASTA index now reach the report (--ref-gff, --ref-mask, --ref-fai, as REF=path). The report builds one genome per reference: its length, genes and masked regions, placed contig after contig in the index's order, as its samples' profiles are binned.
  • The genome page shows one reference at a time, with a button per reference and its sample count, opening on the one most samples were mapped to.
  • Clicking a deletion opens its reference, and go-to-gene finds a gene in any reference.
  • A sample's own genome profile uses its own reference, and so does its SNP density per Mb.

Reading those GFFs also turned up two faults of the report's GFF reader:

  • Placeholder names. A GFF built from a table writes pandas' NaN as Name=nan. The reader took it for a name and merged genes by name, so on E1ASM0035 3,001 of 3,989 genes became one gene called nan: the landscape knew 988 genes, and the per-gene depth tables named the rest nan. A placeholder (nan, NA, .) is now read as no name, as the gene-conversion annotation already does.
  • RefSeq CDS. A RefSeq GFF names every CDS after its protein, so each H37Rv gene came back twice (dnaA and NP_214515.1). A CDS whose Parent is a gene of the file is now that gene. Genes that share a name at different places stay two.
Genes read Before Now
E1ASM0035 (Prokka-style, Name=nan) 988 3,989
E1ASM0057 970 3,967
H37Rv (RefSeq) 7,884 3,978

Outputs and defaults

  • QC_REPORT takes every reference's GFF, repeat/exclude list and FASTA index instead of the first reference's GFF and list.
  • qc_report.py has new options --ref-gff, --ref-mask and --ref-fai. With them, --gff and --mask-bed are not used. Both still work for a report made by hand.
  • The report payload gains genomes (one entry per reference). The top-level genes, mask_* and genome_len hold the reference with the most samples, so a single-reference report reads as before.
  • Gene names in the per-sample gene_depth.tsv tables fall back to the locus tag or the ID where the GFF writes Name=nan.

How it was verified

tests/run_tests.sh unit   # 2,187 passed, 39 skipped
tests/run_tests.sh js     # 120 passed
tests/run_tests.sh lint   # clean
mkdocs build --strict     # passes

The cohort's report was re-rendered with both references' real GFFs and indexes and opened in headless Chrome: E1ASM0035 shows its 113 samples over 4.41 Mb, E1ASM0057 its 72 over 4.35 Mb, each with its own variable genes. Switching between them works, and there are no console errors.

Not run here: the Nextflow step that pairs each reference with its files. CI's stub run checks the wiring but not the script, which runs on the cluster.

Checklist

  • tests/run_tests.sh passes locally (unit, js and lint; the pipeline leg in CI)
  • ruff check . is clean, and I did not restyle code the change does not touch
  • New or changed behaviour has a test, and tests pin what the code does today
  • A new process in modules/ has a stub: block (not applicable: no new process)
  • tests/data/ was not edited by hand (fixtures come from make_test_data.py)
  • User-visible changes have a CHANGELOG.md entry and, where relevant, a docs update

The report's GFF reader took a placeholder written as data for a name
and merged genes by name. A GFF built from a table writes pandas' NaN as
Name=nan: on one of the cohort's references 3,001 of 3,989 genes became
a single gene called nan, so the genome landscape, the variable-genes
table and go-to-gene knew 988 genes, and the per-gene depth tables named
the rest nan. A placeholder (nan, NA, '.') is now read as no name, as
the gene-conversion annotation already does, and the name falls through
to the locus tag or the ID.

A RefSeq GFF names every CDS after its protein, so each gene of H37Rv
came back twice, as dnaA and as NP_214515.1. A CDS whose Parent is a
gene of the file is now that gene, a gene and a same-named CDS that
overlap stay one, and genes that share a name at different places stay
two.
The report took the GFF, the masked regions and the length of the first
reference of the run for the whole genome page. On a cohort mapped
against two assemblies, the second one's samples were drawn over the
first one's genes, and its variable-genes table, go-to-gene, masked
regions and genome insights were the first one's too.

Every reference's GFF, masked regions and FASTA index now reach the
report (--ref-gff, --ref-mask, --ref-fai), and it builds one genome per
reference: its length, genes and masked regions, placed contig after
contig in the index's order, as its samples' profiles are binned. The
genome page shows one reference at a time, opening on the one most
samples were mapped to, with a button for each; clicking a deletion
opens its reference, go-to-gene finds a gene in any of them, a sample's
own profile uses its own reference, and the SNP density per Mb divides
by it. A single-reference report looks as it did.
Nextflow 26's strict parser reads per_ref(gffs) as a call to a function
named per_ref, which a closure held in a variable is not, so every run
stopped before it started; Nextflow 24 took it. The pairing is now
refPairs in utils.nf, beside the other helpers.
@Paururo
Paururo merged commit d29ef14 into main Sep 23, 2026
7 checks passed
@Paururo
Paururo deleted the fix/per-reference-genes branch September 23, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant