out2spreadsheet.py debug prints corrupt summary.tsv in 345 of 359 COVID analyses

[wilke]

Summary

out2spreadsheet.py writes leftover debug output to stdout, and stdout is
summary.tsv. 345 of 359 COVID analysis directories (96%) have a corrupted
summary.tsv, and every affected run exited 0.

Root cause

covid/scripts/process-run.sh:40 (and get_bams.sh:18, create_summary.sh:12,
recompute-run.sh:51, pull-from-jim.sh:32):

for i in output/* ; do python3 ${base}/scripts/out2spreadsheet.py $i ; done > summary.tsv

So anything the script prints lands in the TSV. out2spreadsheet.py prints twice:

115    d = re.match(r"^(\d{2})(\d{2})(\d{2})", filename)
116    i = re.match(r"^[^-_\.]+\.([^-_\.]+).+\.out", filename)
118    print(d,i)                                              # <-- debug leak
...
122    print(i[1], mapping.id2site(i[1]), l, filename)          # <-- debug leak

Resulting summary.tsv content:

<_sre.SRE_Match object; span=(0, 6), match='230612'>	<_sre.SRE_Match object; ...>
ID 117875 not in sides

A third signature, ID <n> not in sides, also reaches stdout from the
site-mapping lookup around line 119; I did not pin its exact origin.

Downstream consequence

process-run.sh joins summary.tsv to build summary.all.tsv. Fed this
garbage, the join produces nothing — 172 of the same runs carry a 0-byte
summary.all.tsv.

Measured impact

summary.tsv verdict	count
contaminated	345
valid	9
truncated	2
absent	3
total analyses	359

Note two distinct repr formats occur — <re.Match object (Python 3.7+) and
<_sre.SRE_Match object (earlier). Detection keyed only on the modern form
misses 31 files.

Reproduce

cd /nfs/seq-data/covid/runs
for d in */; do
  f="${d}summary.tsv"; [ -f "$f" ] || continue
  head -c 512 "$f" | grep -qE '<(_sre\.SRE_Match|re\.Match) object|not in sides' \
    && echo "CONTAMINATED $d"
done | wc -l

Suggested fix

Delete both print() calls at lines 118 and 122, or route them to sys.stderr.
Then re-run the 345 affected analyses — the COVID summary results are not
trustworthy until this is done.

Provenance

Found by seqtrack artifact-verdict derivation. Full analysis:
reports/work-260730.seqtrack.md.

[wilke]

Pinned the third signature. It is a fourth debug print, in the shared mapping
library rather than in out2spreadsheet.py:

covid/scripts/lib/mapping.py:87

print("ID " + str(Id) + " not in sides")

Same defect, same consequence: it goes to stdout, which is summary.tsv.
It fires whenever a sample ID has no site mapping, so it also silently marks
exactly the samples whose site metadata is missing — related to the undefined
SiteID.tsv entries tracked separately.

The same file exists in three copies, all with the print at line 87:

• covid/scripts/lib/mapping.py (the one actually used)
• covid/sequencing-center-covid/scripts/lib/mapping.py
• covid/tmp/scripts/lib/mapping.py

So the full fix is four prints across two files, not two: out2spreadsheet.py
lines 118 and 122, and lib/mapping.py:87. Anything diagnostic should go to
sys.stderr, which the wrappers leave untouched.