Summary
check_shock_uploads.py decides whether a fastq is archived by matching on the
sample-ID prefix only, so it cannot see that R1 uploaded and R2 did not. It
reports such a batch as complete.
Location
scripts/check_shock_uploads.py:322
shock_norm.startswith(sample_id + '_')
with sample_id = file_norm.split('_')[0] (line ~318). Both
100007_S23_L001_R1_001 and ..._R2_001 therefore "match" if either is
present in SHOCK.
Why it matters
This is the only reconciliation tool between local fastqs and SHOCK, and
/nfs/seq-data is at 96% (2.4 TB free of 50 TB). Any future
archive-then-delete workflow would use this to decide a file is safely stored.
It is not strong enough to gate a delete — a half-uploaded pair reads as
fully uploaded.
Two further problems in the same file
- The verdict is discarded. Complete/Partial/Missing is computed at
lines ~372-377 and never persisted, so the answer must be recomputed from
scratch every time and no history exists.
- TLS verification is disabled —
verify=False at line 113 with warnings
suppressed at line 9.
Suggested fix
Match per file, not per sample prefix, and prefer the checksum. Note that
SHOCK_functions.sh:53 computes the md5 of the decompressed stream
(gunzip -c FILE | md5sum), so joining on the gzip md5 matches nothing — the
two must not be conflated.
Anything gating a deletion should additionally require an independent re-read of
the object's checksum, separated in time from the write, rather than trusting a
read that may be served from cache.
Provenance
reports/work-260730.seqtrack.md.
Summary
check_shock_uploads.pydecides whether a fastq is archived by matching on thesample-ID prefix only, so it cannot see that R1 uploaded and R2 did not. It
reports such a batch as complete.
Location
scripts/check_shock_uploads.py:322with
sample_id = file_norm.split('_')[0](line ~318). Both100007_S23_L001_R1_001and..._R2_001therefore "match" if either ispresent in SHOCK.
Why it matters
This is the only reconciliation tool between local fastqs and SHOCK, and
/nfs/seq-datais at 96% (2.4 TB free of 50 TB). Any futurearchive-then-delete workflow would use this to decide a file is safely stored.
It is not strong enough to gate a delete — a half-uploaded pair reads as
fully uploaded.
Two further problems in the same file
lines ~372-377 and never persisted, so the answer must be recomputed from
scratch every time and no history exists.
verify=Falseat line 113 with warningssuppressed at line 9.
Suggested fix
Match per file, not per sample prefix, and prefer the checksum. Note that
SHOCK_functions.sh:53computes the md5 of the decompressed stream(
gunzip -c FILE | md5sum), so joining on the gzip md5 matches nothing — thetwo must not be conflated.
Anything gating a deletion should additionally require an independent re-read of
the object's checksum, separated in time from the write, rather than trusting a
read that may be served from cache.
Provenance
reports/work-260730.seqtrack.md.