Submit tasks as job arrays and fix RNAs in distill summaries#472
Submit tasks as job arrays and fix RNAs in distill summaries#472tpall wants to merge 25 commits intoWrightonLabCSU:devfrom
Conversation
…nctionality - Update groupby_column default value to "input_fasta" in distill.py - Adjust input paths in distill.nf for consistency - Enhance argument handling in SUMMARIZE process
…rid of FASTA_COLUMN environment variable
|
Hey @tpall Thanks for this. The job array addition it nice. There is a larger planned update to batch a lot of the inputs into singular jobs to reduce the burden on the queue, since running DRAM with lots of inputs can overwhelm a SLURM scheduler, but adding in job arrays, which weren't supported on the version of Nextflow we initially developed DRAM2 on, but we recently moved to >=24 (we should lock in >=24.04.0 since there are early 24.* prereleases out there if we add in job arrays). I will have to do some testing on utilizing job arrays and their implication. Because from my initial testing it seems like it stops the next stop from proceeding until their are enough inputs to fill an array. Which might be ok. But if we are going to be doing batching anyway, it might not be that important and not worth it. Also thanks for some of the other QoL updates like updating some of the syntax to DSL2 (Channel -> channel, etc.). I will have to more fully review the code, which I can get to in a couple weeks. I have deadline for next week, and probably won't be able to review much before then. But I will leave just a couple of quick thoughts. Thanks again |
| } | ||
|
|
||
| withName: 'DRAM:ANNOTATE:CALL:.*|DRAM:ANNOTATE:DB_SEARCH:.*' { | ||
| array = params.array_size |
There was a problem hiding this comment.
I would like to support people running DRAM2 with local executor (such as on their own computer if they want), which doesn't support array. So the array should only be used with executors that support it.
| maxRetries = 2 | ||
| } | ||
|
|
||
| withName: 'DRAM:ANNOTATE:CALL:.*|DRAM:ANNOTATE:DB_SEARCH:.*' { |
There was a problem hiding this comment.
jobs under DRAM:ANNOTATE:QC:COLLECT.* could also have a job array
| withName: 'DRAM:ANNOTATE:CALL:.*|DRAM:ANNOTATE:DB_SEARCH:.*' { | ||
| array = params.array_size | ||
| } | ||
|
|
There was a problem hiding this comment.
this code here in base.config for the job array should probably be in modules.config
Added COLLECT_RNA steps to array
Adds a DECOMPRESS_FASTA module (bbtools reformat.sh in the existing bbmap container) and routes only .gz inputs through it via a branch on the fasta channel. Basename stripping is unified so sample.fa and sample.fa.gz produce the same downstream name, keeping outputs identical regardless of input compression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
It was already defined in nextflow.config (default 10) and consumed by conf/base.config, but absent from nextflow_schema.json, so runs emitted a schema-validation warning. Added alongside queue_size under Process Options. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the never-included trees module and all scripts only it referenced (parse_annotations.py, update_annots_trees.py, color_labels.R), plus update_tree.py which had no references at all. Also removes assets/trees/ refpkgs (only consumed by the dropped module) and the DRAM-v1 standalone DB setup scripts under assets/internal/ which were never wired into the DSL2 pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both are unreferenced DRAM-v1 helpers under bin/assets/forms/. They shell out to a DRAM-setup.py CLI that isn't part of the DSL2 pipeline, and upstream has already removed them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eight references used DB_CHANNEL_SETUP (all caps) but the workflow is defined as DB_channel_SETUP. Groovy is case-sensitive so the references failed at runtime with "No such variable: DB_CHANNEL_SETUP". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Variable was declared as formattedOutputchannels (lowercase 'c') at line 124, but seven references used formattedOutputChannels (uppercase 'C'), tripping MissingPropertyException at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The mmseqs2 index files (*.mmsdb, .mmsdb.idx, .mmsdb_h, ...) are intermediate, memory-mapped artifacts only consumed by MMSEQS_SEARCH via the Nextflow channel; they are never read from results/. Copying them per-bin can be hundreds of GB and was filling GPFS quota, causing publishDir copy failures that aborted the pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 participants
Changes
array_size = 10parameter to nextflow.config andarrayto conf/base.config for more efficient cluster execution.Channeltochannel) for consistency across workflows and improve readability + usage of implicit variable within closures (e.g.it.nametoit -> it.name)Computing environment and command