Skip to content

feat(io): bound coordinate-sort memory with an external merge #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alejandrogzi wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(io): bound coordinate-sort memory with an external merge #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
000cd34
feat(io): bound coordinate-sort memory with an external merge
alejandrogzi Aug 6, 2026
fc16d8a
test(io): cover bounded-memory coordinate-sort spilling
alejandrogzi Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,20 @@ Sections commonly used: Features, Bug fixes, Other changes.
- **STARsolo `features.tsv` column 2 now emits the GTF `gene_name`**
(symbol), with the STAR gene_id fallback, instead of duplicating the
gene_id.
- **`--outSAMtype BAM SortedByCoordinate` no longer buffers the whole
output in RAM.** The coordinate sort is now external: it fills a
`--limitBAMsortRAM` buffer, spills sorted runs beside the output, and
k-way merges them on finish. Peak memory is flat in output size
(measured: 610 MB at a 64 MiB budget for 300 k through 2.4 M records,
versus 648 MB → 2,132 MB before, a growth of 723 B/record that
extrapolated to ~116 GB for a 160 M-record human sample). Output is
byte-identical to the previous in-memory sort — coordinate ties still
resolve to input order, since runs merge with the run index as
tiebreak (verified at 2.4 M records through a 219-run multi-pass
merge). `--limitBAMsortRAM N` now spills above `N` rather than
aborting the run, and `--limitBAMsortRAM 0` means 512 MiB instead of
"unlimited". Runs beyond 64 are merged in balanced passes so a small
budget on a large run cannot exhaust file descriptors.

### Bumps

Expand Down
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ A Rust reimplementation of [STAR](https://github.com/alexdobin/STAR) (Spliced Tr

rustar-aligner aims to be a faithful port of STAR, matching the original behavior as closely as possible. It uses the same genome index format, accepts the same `--camelCase` command-line parameters, and produces compatible SAM/BAM output.

**Current status**: End-to-end single-end and paired-end RNA-seq alignment with splice junction detection, two-pass mode, chimeric alignment detection (including multi-junction Tier 3), gene-level quantification, **single-cell quantification (STARsolo: Gene / GeneFull / SJ / Velocyto features, barcode correction, UMI dedup, EmptyDrops_CR cell calling)**, WASP allele-specific filtering, paired-end mate-overlap merging, coverage-track output, adapter/CellRanger clipping, and multi-threaded parallel processing. Default SAM output is byte-identical to STAR's (`NH HI AS nM` attributes); solo count matrices are byte-identical to STARsolo's. Pure-Rust core with an in-tree deterministic RNG (no `rand` dependency). 578 tests passing (553 unit + 25 integration), 0 clippy warnings. See [Performance & Benchmarks](#performance--benchmarks) for a native three-way comparison against STARsolo and CellRanger.
**Current status**: End-to-end single-end and paired-end RNA-seq alignment with splice junction detection, two-pass mode, chimeric alignment detection (including multi-junction Tier 3), gene-level quantification, **single-cell quantification (STARsolo: Gene / GeneFull / SJ / Velocyto features, barcode correction, UMI dedup, EmptyDrops_CR cell calling)**, WASP allele-specific filtering, paired-end mate-overlap merging, coverage-track output, adapter/CellRanger clipping, and multi-threaded parallel processing. Default SAM output is byte-identical to STAR's (`NH HI AS nM` attributes); solo count matrices are byte-identical to STARsolo's. Pure-Rust core with an in-tree deterministic RNG for alignment tie-breaking. 622 tests passing (591 unit + 31 integration), 0 clippy warnings. See [Performance & Benchmarks](#performance--benchmarks) for a native three-way comparison against STARsolo and CellRanger.

## Quick Start

Expand Down Expand Up @@ -208,7 +208,7 @@ resident; the 16 GB sparse index is stable at ~54 s.</sub>

- Single-end and paired-end alignment with mate rescue
- Read-end alignment mode (`--alignEndsType Local` (default) / `EndToEnd` / `Extend5pOfRead1` / `Extend5pOfReads12` / `Extend3pOfRead1`)
- SAM, unsorted BAM, and coordinate-sorted BAM output (`--outSAMtype SAM`, `BAM Unsorted`, or `BAM SortedByCoordinate`)
- SAM, unsorted BAM, and coordinate-sorted BAM output (`--outSAMtype SAM`, `BAM Unsorted`, or `BAM SortedByCoordinate`). The coordinate sort is external: it buffers up to `--limitBAMsortRAM`, spills sorted runs beside the output, and merges them, so peak memory is independent of output size
- Multi-threaded parallel alignment (`--runThreadN`)
- GTF-based junction annotation with scoring bonus (`--sjdbGTFfile`)
- Two-pass mode for novel junction discovery (`--twopassMode Basic`)
Expand All @@ -227,8 +227,8 @@ resident; the 16 GB sparse index is stable at ~54 s.</sub>
- Unmapped read output to FASTQ (`--outReadsUnmapped Fastx` → `Unmapped.out.mate1` / `mate2`)
- Gzip-compressed FASTQ input (`--readFilesCommand zcat`)
- Read group tags (`--outSAMattrRGline`)
- Deterministic input-order output regardless of thread count (`--outSAMorder Paired` / `PairedKeepInputOrder`, both accepted; rustar always preserves FASTQ order)
- Deterministic in-tree RNG for reproducible tie-breaking (`--runRNGseed`; no `rand` dependency)
- Deterministic input-order output regardless of thread count (`--outSAMorder Paired` / `PairedKeepInputOrder`, both accepted)
- Deterministic in-tree RNG for reproducible alignment tie-breaking (`--runRNGseed`)
- SAM optional tags: NH, HI, AS, nM, NM, XS, jM, jI, MD (default `NH HI AS nM` matches STAR; `NM` opt-in)
- `--outSAMattributes` control (Standard/All/None/explicit list)
- SECONDARY flag (0x100) on multi-mapper alignments
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/reference/cli-parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Run `rustar-aligner --help` for the full machine-generated listing.
| `--outFileNamePrefix` | `./` | Prefix (path + filename stem) for all output files. |
| `--outSAMtype` | `SAM` | `SAM`, `BAM Unsorted`, `BAM SortedByCoordinate`, or `None`. |
| `--outBAMcompression` | `1` | BGZF level. `-1`/`0` = uncompressed; `1`–`8` = flate2 levels; `≥9` = max. |
| `--limitBAMsortRAM` | `0` | Max RAM (bytes) for sorted BAM. `0` = unlimited. |
| `--limitBAMsortRAM` | `0` | Max RAM for the coordinate sort; accepts a suffix (`8G`, `512M`). Records beyond it spill to sorted runs beside the output and are merged, so output is unaffected by this value. `0` means 512 MiB. Very small values still work but cost wall time (many spill runs plus an extra merge pass). |
| `--outStd` | `None` | Route primary output to stdout: `None`, `SAM`, `BAM_Unsorted`, `BAM_SortedByCoordinate`. |

## Output: SAM/BAM records
Expand Down
Loading