Summary
Three problems in covid/scripts/process-covid-run, one of which is a standing
storage cost on a filer with 2.4 TB free.
1. Copies fastqs instead of symlinking (line 39)
find $1 -name "*.fastq.gz" -exec cp {} ${covid_run_dir}/samples/ \;Every COVID analysis holds a full byte-for-byte duplicate of its batch's
fastqs — a systematic 2x on the covid corpus, on a filesystem at 96% (48 TB
used of 50 TB) with two rsyncs writing into it every three hours.
The run directory already symlinks reads -> samples (line 34), so the pattern
is established. ln -s would reclaim the duplicate immediately.
Caveat worth checking before changing this: the copies are what make covid
analysis dirs linkable back to their source batch by content — (basename, size)
set matching resolves 205 of 264 links, including cases name matching gets
actively wrong. Symlinks preserve that (the target still stats identically), but
verify before switching.
2. Not idempotent (line 27)
mkdir /local/incoming/covid/runs/$(basename $1)
No -p, so any re-run fails immediately. This is likely part of why re-analysis
was done by copying directories to _org / .re20240317 suffixes instead.
3. Discards both of its arguments
$1 (the source batch dir) is used and never recorded. There is no stored
pointer from a covid analysis dir back to the batch it came from — only
160 of 373 dirs name-match one, and reconstructing the rest requires content
matching.$2 (the primer set: qiagen/swift) is passed to assembly.sh and written
nowhere. It is unrecoverable for any batch lacking an SRA run template. Worse,
assembly.sh silently defaults to qiagen on an unrecognised value.
Suggested fix
Write a small run-manifest.json at the top of the run dir recording source
batch path, primer set, reference, container digests and start time. That single
file would make items 2 and 3 non-issues and remove the need to infer either
fact later.
Provenance
reports/work-260730.seqtrack.md.
Summary
Three problems in
covid/scripts/process-covid-run, one of which is a standingstorage cost on a filer with 2.4 TB free.
1. Copies fastqs instead of symlinking (line 39)
Every COVID analysis holds a full byte-for-byte duplicate of its batch's
fastqs — a systematic 2x on the covid corpus, on a filesystem at 96% (48 TB
used of 50 TB) with two rsyncs writing into it every three hours.
The run directory already symlinks
reads -> samples(line 34), so the patternis established.
ln -swould reclaim the duplicate immediately.Caveat worth checking before changing this: the copies are what make covid
analysis dirs linkable back to their source batch by content —
(basename, size)set matching resolves 205 of 264 links, including cases name matching gets
actively wrong. Symlinks preserve that (the target still stats identically), but
verify before switching.
2. Not idempotent (line 27)
mkdir /local/incoming/covid/runs/$(basename $1)No
-p, so any re-run fails immediately. This is likely part of why re-analysiswas done by copying directories to
_org/.re20240317suffixes instead.3. Discards both of its arguments
$1(the source batch dir) is used and never recorded. There is no storedpointer from a covid analysis dir back to the batch it came from — only
160 of 373 dirs name-match one, and reconstructing the rest requires content
matching.
$2(the primer set:qiagen/swift) is passed toassembly.shand writtennowhere. It is unrecoverable for any batch lacking an SRA run template. Worse,
assembly.shsilently defaults to qiagen on an unrecognised value.Suggested fix
Write a small
run-manifest.jsonat the top of the run dir recording sourcebatch path, primer set, reference, container digests and start time. That single
file would make items 2 and 3 non-issues and remove the need to infer either
fact later.
Provenance
reports/work-260730.seqtrack.md.