feat(io): optional pure-Rust parallel gzip input decoding (rapidgzip-core)

CHANGELOG.md

@@ -13,6 +13,15 @@ Sections commonly used: Features, Bug fixes, Other changes.
 
 ### Other changes
 
+- **Optional `rapidgzip` feature** for parallel gzip/BGZF decoding of
+  `--readFilesIn`, backed by the pure-Rust `rapidgzip-core` (inflate backend is
+  zlib-rs, the same one `flate2` uses here, so it adds no C toolchain). Off by
+  default and inert until `RUSTAR_GZ_DECODE_THREADS` is set: the decoder reaches
+  3356 MB/s on 8 threads against 1552 MB/s for one `flate2` thread, but the
+  aligner only consumes ~140 MB/s of decompressed FASTQ, so decode was never the
+  bottleneck and enabling it measured slightly slower with ~360 MB more resident.
+  Output is byte-identical to the `flate2` path.
+
 - `cluster_seeds` reuses its window-bin map across reads on a thread instead
   of rebuilding it per read. Merging two windows re-keys every bin in the
   merged span, so the per-read pre-sizing was only a floor and the map
@@ -42,6 +51,13 @@ Sections commonly used: Features, Bug fixes, Other changes.
 
 ### Bug fixes
 
+- **Multi-member gzip input is no longer truncated.** Compressed input was
+  decoded with `flate2::read::GzDecoder`, which stops at the end of the first
+  gzip member; a `.gz` made of several concatenated members (bcl2fastq output,
+  `cat a.fq.gz b.fq.gz`, any BGZF file) was read partially with no error and no
+  warning. All four read paths now use `MultiGzDecoder`: FASTQ input, the solo
+  barcode whitelist, solo counting, and the `emptydrops` binary.
+
 - Read names are cut at `--readNameSeparator` (default `/`), as STAR does. A
   read named `foo/1` was previously emitted as `foo/1` where STAR emits `foo`.
 

Cargo.toml

@@ -64,8 +64,19 @@ caps-sa = "0.6"
 mimalloc = { version = "0.1", default-features = false }
 libmimalloc-sys = { version = "0.1.49", features = ["extended"] } # mi_option_set (purge_delay); see main.rs
 libdeflater = "1.25.2"
+# Parallel gzip/BGZF/zlib decoder for `--readFilesIn *.gz`. Pure Rust: its only
+# dependencies are `bon`, `crossbeam-deque` and `libz-rs-sys` (zlib-rs), which is
+# the same inflate backend `flate2` already uses here, so it adds no C toolchain
+# requirement. Behind a feature only until the dependency discussion in #224
+# settles. See `set_gz_decode_threads` for the measurements.
+rapidgzip-core = { version = "0.3", optional = true }
 noodles-bgzf = { version = "0.49", features = ["libdeflate"] }
 
+[features]
+default = []
+# Parallel gzip input decoding (pure Rust).
+rapidgzip = ["dep:rapidgzip-core"]
+
 [dev-dependencies]
 assert_cmd = "2"
 predicates = "3"

README.md

@@ -257,6 +257,26 @@ cargo clippy --all-targets  # Lint
 cargo fmt                   # Format
 ```
 
+### Optional feature: `rapidgzip`
+
+```bash
+cargo build --release --features rapidgzip
+```
+
+Decodes gzipped `--readFilesIn` with [rapidgzip-core](https://crates.io/crates/rapidgzip-core), a
+pure-Rust parallel gzip decoder, instead of a single `flate2` thread. It pulls in no C toolchain:
+its inflate backend is zlib-rs, the same one `flate2` already uses here.
+
+**Off by default, and inert even when compiled in** until `RUSTAR_GZ_DECODE_THREADS` is set to a
+worker count (`0` keeps `flate2`). The decoder is fast (3356 MB/s on 8 threads against 1552 MB/s for
+one `flate2` thread on the same file), but this aligner consumes decompressed FASTQ at only
+~140 MB/s, so decode already runs at under a tenth of its capacity. Enabling it measured slightly
+slower and used ~360 MB more memory. It is worth turning on when the consumer is fast enough to
+drain a single inflate thread, roughly 7 M reads/s for a 100 bp library.
+
+The portable alternative needs no feature and no rebuild: pipe an external decompressor with
+`--readFilesCommand` (`gzcat`, `gunzip -c`, `igzip -dc`, `rapidgzip-rust -dc`).
+
 ## Development
 
 The majority of rustar-aligner's code was written by [Claude Code](https://claude.ai/code) (Anthropic's AI coding assistant), with technical direction, architecture decisions, and validation by the project maintainer.

src/bin/emptydrops.rs

@@ -19,7 +19,7 @@ use std::fs::File;
 use std::io::{BufRead, BufReader, BufWriter, Read, Write};
 use std::path::{Path, PathBuf};
 
-use flate2::read::GzDecoder;
+use flate2::read::MultiGzDecoder;
 use rustar_aligner::rng::{SplitMix64, cumulative_weights, sample_cumulative};
 
 struct Args {
@@ -89,7 +89,7 @@ fn find(d: &Path, base: &str) -> PathBuf {
 fn reader(p: &Path) -> Box<dyn BufRead> {
     let f = File::open(p).unwrap();
     if p.extension().is_some_and(|e| e == "gz") {
-        Box::new(BufReader::new(GzDecoder::new(f)))
+        Box::new(BufReader::new(MultiGzDecoder::new(f)))
     } else {
         Box::new(BufReader::new(f))
     }