Skip to content

ML4H Submission Final: Code + Experiment Run Results - #52

Open
will-pang wants to merge 67 commits into
mainfrom
ml4h-merge-tranche-1-wp-20260830
Open

ML4H Submission Final: Code + Experiment Run Results#52
will-pang wants to merge 67 commits into
mainfrom
ml4h-merge-tranche-1-wp-20260830

Conversation

@will-pang

@will-pang will-pang commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

Overview of Changes

Results

Seed 1

Labs (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5056 0.8987 0.9586 0.3968 0.4523 1x RTX 2080 Ti 0.1 GB 1h00m run
RNN 0.7823 0.9474 0.9767 0.7326 0.7455 1x RTX 2080 Ti 0.2 GB 32m24s run
Transformer 0.6362 0.9175 0.9642 0.5850 0.5904 1x RTX 2080 Ti 4.6 GB 48m05s run
Bottleneck Transformer 0.6545 0.9262 0.9659 0.5347 0.6062 1x RTX 2080 Ti 4.6 GB 32m29s run
EHRMamba 0.7437 0.9405 0.9738 0.7045 0.7094 1x RTX 2080 Ti 0.7 GB 27m26s run
JambaEHR 0.7660 0.9451 0.9752 0.7216 0.7320 1x RTX 2080 Ti 6.1 GB 48m06s run

Labs + Notes (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5781 0.9474 0.9627 0.4502 0.6113 1x RTX A6000 5.9 GB 2h52m run
RNN 0.8505 0.9830 0.9795 0.7502 0.8463 1x RTX A6000 5.9 GB 2h18m run
Transformer 0.6653 0.9412 0.9686 0.6140 0.6787 1x RTX A6000 5.9 GB 2h49m run
Bottleneck Transformer 0.7061 0.9661 0.9693 0.6336 0.7197 1x RTX 6000 Ada 5.9 GB 1h29m run
EHRMamba 0.8225 0.9802 0.9770 0.7224 0.8247 1x RTX 6000 Ada 5.9 GB 56m14s run
JambaEHR 0.8370 0.9815 0.9784 0.7453 0.8309 1x RTX 6000 Ada 5.9 GB 1h08m run

Labs + Notes + CXR (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5703 0.9344 0.9610 0.4782 0.5732 1x RTX 6000 Ada 8.0 GB 3h04m run
RNN 0.8359 0.9792 0.9790 0.7756 0.8204 1x RTX 6000 Ada 8.0 GB 2h00m run
Transformer 0.6797 0.9457 0.9663 0.6474 0.6673 1x RTX 6000 Ada 8.0 GB 2h49m run
Bottleneck Transformer 0.6820 0.9581 0.9670 0.6090 0.6835 1x RTX 6000 Ada 8.0 GB 2h21m run
EHRMamba 0.8049 0.9725 0.9770 0.7452 0.8099 1x RTX 6000 Ada 8.0 GB 1h36m run
JambaEHR 0.8244 0.9750 0.9779 0.7279 0.8096 1x RTX 6000 Ada 8.1 GB 1h41m run

Seed 2

Labs (Will)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Compute VRAM Runtime WandB Link
MLP 0.48 0.89 0.96 0.37 NVIDIA A100-SXM4-40GB 0.19 GB 1 hr 28 min Link
RNN 0.75 0.94 0.98 0.70 NVIDIA A100-SXM4-40GB 0.20 GB 0 hr 40 min Link
Transformer 0.57 0.90 0.96 0.55 NVIDIA A100-SXM4-40GB 1.51 GB 0 hr 45 min Link
Bottleneck Transformer 0.59 0.91 0.96 0.53 NVIDIA A100-SXM4-40GB 0.72 GB 0 hr 46 min Link
EHRMamba 0.65 0.89 0.97 0.65 NVIDIA A100-SXM4-40GB 0.86 GB 0 hr 21 min Link
JambaEHR 0.64 0.87 0.97 0.67 NVIDIA A100-SXM4-40GB 2.97 GB 0 hr 36 min Link

Labs + Notes (Will)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Compute VRAM Runtime WandB Link
MLP 0.58 0.94 0.96 0.49 NVIDIA A100-SXM4-40GB 0.71 GB 1 hr 47 min Link
RNN 0.84 0.98 0.98 0.77 NVIDIA A100-SXM4-40GB 0.72 GB 1 hr 31 min Link
Transformer 0.68 0.94 0.97 0.63 NVIDIA A100-SXM4-40GB 2.53 GB 1 hr 44 min Link
Bottleneck Transformer 0.70 0.96 0.97 0.64 NVIDIA A100-SXM4-40GB 1.36 GB 1 hr 34 min Link
EHRMamba 0.71 0.92 0.97 0.70 NVIDIA A100-SXM4-40GB 1.46 GB 1 hr 6 min Link
JambaEHR 0.76 0.97 0.97 0.72 NVIDIA A100-SXM4-40GB 4.17 GB 1 hr 9 min Link

Labs + Notes + CXR (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5913 0.9455 0.9629 0.4989 0.5801 1x RTX 6000 Ada 7.4 GB 3h29m run
RNN 0.8442 0.9787 0.9794 0.7572 0.8599 1x RTX 6000 Ada 7.4 GB 2h30m run
Transformer 0.6855 0.9393 0.9691 0.6409 0.7108 1x RTX 6000 Ada 7.4 GB 2h26m run
Bottleneck Transformer 0.7133 0.9625 0.9697 0.6418 0.7271 1x RTX 6000 Ada 7.4 GB 2h06m run
EHRMamba 0.8001 0.9674 0.9745 0.7401 0.8195 1x RTX 6000 Ada 7.4 GB 1h46m run
JambaEHR 0.8103 0.9734 0.9763 0.7517 0.8285 1x RTX 6000 Ada 7.4 GB 2h02m run

Seed 3

Labs (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5042 0.9011 0.9584 0.4069 0.4661 1x RTX 2080 Ti 0.2 GB 58m39s run
RNN 0.7605 0.9432 0.9749 0.7160 0.7479 1x RTX 2080 Ti 0.2 GB 45m10s run
Transformer 0.6230 0.9043 0.9634 0.5778 0.5755 1x RTX 2080 Ti 7.0 GB 50m30s run
Bottleneck Transformer 0.6468 0.9210 0.9647 0.5915 0.6045 1x RTX 2080 Ti 7.0 GB 49m26s run
EHRMamba 0.7326 0.9355 0.9722 0.6806 0.7102 1x RTX 2080 Ti 0.8 GB 24m03s run
JambaEHR 0.7444 0.9388 0.9713 0.7002 0.7406 1x RTX 2080 Ti 8.9 GB 47m41s run

Labs + Notes (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5933 0.9429 0.9617 0.5036 0.5642 1x RTX A6000 6.9 GB 1h46m run
RNN 0.8439 0.9785 0.9797 0.7798 0.8381 1x RTX A6000 6.9 GB 1h31m run
Transformer 0.6907 0.9421 0.9662 0.6356 0.6828 1x RTX A6000 6.9 GB 2h25m run
Bottleneck Transformer 0.7174 0.9587 0.9654 0.6447 0.7055 1x RTX 6000 Ada 6.9 GB 1h30m run
EHRMamba 0.8157 0.9722 0.9762 0.7099 0.8142 1x RTX 6000 Ada 6.9 GB 1h00m run
JambaEHR 0.8324 0.9769 0.9779 0.7516 0.8167 1x RTX 6000 Ada 6.9 GB 1h06m run

Labs + Notes + CXR (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5940 0.9368 0.9620 0.4690 0.5934 1x RTX 6000 Ada 7.7 GB 3h22m run
RNN 0.8439 0.9807 0.9787 0.7502 0.8510 1x RTX 6000 Ada 7.7 GB 2h27m run
Transformer 0.6800 0.9454 0.9650 0.6339 0.6846 1x RTX 6000 Ada 7.7 GB 2h13m run
Bottleneck Transformer 0.7225 0.9633 0.9695 0.6515 0.7208 1x RTX 6000 Ada 7.7 GB 2h14m run
EHRMamba 0.8083 0.9758 0.9765 0.7254 0.8219 1x RTX 6000 Ada 7.7 GB 1h41m run
JambaEHR 0.8172 0.9743 0.9770 0.7513 0.8183 1x RTX 6000 Ada 7.7 GB 1h33m run

Seed 4

Labs (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.4912 0.8920 0.9580 0.4184 0.4946 1x RTX 2080 Ti 0.2 GB 1h15m run
RNN 0.7509 0.9432 0.9736 0.6826 0.7596 1x RTX 2080 Ti 0.2 GB 33m03s run
Transformer 0.6070 0.9085 0.9624 0.5594 0.6210 1x RTX 2080 Ti 7.0 GB 45m17s run
Bottleneck Transformer 0.6111 0.9229 0.9646 0.5342 0.6361 1x RTX 2080 Ti 7.0 GB 41m18s run
EHRMamba 0.7174 0.9345 0.9712 0.6574 0.7222 1x RTX 2080 Ti 0.8 GB 32m19s run
JambaEHR 0.7265 0.9405 0.9713 0.6662 0.7438 1x RTX 2080 Ti 8.9 GB 55m34s run

Labs + Notes (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5913 0.9470 0.9633 0.4816 0.5679 1x RTX A6000 6.6 GB 1h44m run
RNN 0.8551 0.9833 0.9801 0.7679 0.8419 1x RTX A6000 6.6 GB 2h39m run
Transformer 0.6834 0.9485 0.9658 0.6308 0.6788 1x RTX A6000 6.6 GB 2h16m run
Bottleneck Transformer 0.7264 0.9660 0.9692 0.6572 0.6968 1x RTX A6000 6.6 GB 1h32m run
EHRMamba 0.8218 0.9781 0.9782 0.7583 0.8106 1x RTX A6000 6.6 GB 1h14m run
JambaEHR 0.8369 0.9792 0.9783 0.7608 0.8326 1x RTX A6000 6.6 GB 1h26m run

Labs + Notes + CXR (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5759 0.9385 0.9643 0.4743 0.6013 1x RTX 6000 Ada 8.5 GB 3h30m run
RNN 0.8363 0.9757 0.9787 0.7552 0.8414 1x RTX 6000 Ada 8.5 GB 2h16m run
Transformer 0.6634 0.9396 0.9674 0.6056 0.6822 1x RTX 6000 Ada 8.5 GB 2h20m run
Bottleneck Transformer 0.6987 0.9578 0.9697 0.6235 0.7172 1x RTX 6000 Ada 8.5 GB 2h38m run
EHRMamba 0.7894 0.9669 0.9765 0.7181 0.8035 1x RTX 6000 Ada 8.5 GB 1h29m run
JambaEHR 0.8105 0.9709 0.9773 0.7374 0.8207 1x RTX 6000 Ada 8.5 GB 1h56m run

Seed 5

Labs (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.4569 0.8960 0.9579 0.3345 0.4488 1x A10 0.1 GB 36m04s run
RNN 0.7486 0.9414 0.9739 0.6772 0.7422 1x A10 0.1 GB 28m21s run
Transformer 0.6065 0.9055 0.9624 0.5775 0.5848 1x A10 0.8 GB 53m52s run
Bottleneck Transformer 0.6234 0.9240 0.9665 0.5557 0.5949 1x A10 0.5 GB 33m12s run
EHRMamba 0.7180 0.9303 0.9736 0.6694 0.7113 1x A10 0.6 GB 26m22s run
JambaEHR 0.7294 0.9374 0.9732 0.6760 0.7262 1x A10 1.9 GB 35m51s run

Labs + Notes (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5772 0.9405 0.9610 0.4869 0.5461 1x A10 6.1 GB 2h26m run
RNN 0.8434 0.9782 0.9788 0.7474 0.8252 1x A10 6.1 GB 1h57m run
Transformer 0.6776 0.9427 0.9679 0.6264 0.6586 1x A10 6.1 GB 2h09m run
Bottleneck Transformer 0.7111 0.9611 0.9697 0.6052 0.6772 1x A10 6.1 GB 2h06m run
EHRMamba 0.8134 0.9747 0.9770 0.7256 0.7951 1x A10 6.1 GB 1h41m run
JambaEHR 0.8283 0.9785 0.9780 0.7429 0.8031 1x A10 6.1 GB 2h03m run

Labs + Notes + CXR (Rian)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Test PR-AUC Compute Peak VRAM Runtime WandB Link
MLP 0.5642 0.9378 0.9602 0.4092 0.5668 1x RTX 6000 Ada 8.2 GB 3h06m run
RNN 0.8292 0.9765 0.9784 0.7600 0.8284 1x RTX 6000 Ada 8.2 GB 2h04m run
Transformer 0.6774 0.9408 0.9665 0.6012 0.6585 1x RTX 6000 Ada 8.2 GB 2h20m run
Bottleneck Transformer 0.7062 0.9578 0.9659 0.6189 0.6847 1x RTX 6000 Ada 8.2 GB 1h55m run
EHRMamba 0.8082 0.9703 0.9764 0.7353 0.7945 1x RTX 6000 Ada 8.2 GB 1h39m run
JambaEHR 0.8276 0.9735 0.9783 0.7517 0.8098 1x RTX 6000 Ada 8.3 GB 1h54m run

Systems

All cells: one GPU each, batch size 32, --use-amp --amp-dtype bf16,
--freeze-encoder, 50 epochs max with patience 5. Runtime is wall clock for
the whole cell including the lab-standardiser fit (~21 min, single-threaded).

Peak VRAM in the tables is torch.cuda.max_memory_allocated, which
understates what a card actually needs. A labs+notes cell reports ~5.9 GB
allocated but the process held 10.19 GiB when it OOMed on a 10.57 GiB
card -- the caching allocator's reserve, plus fp32 master copies on Turing,
which has no native bf16. Size hardware off the process figure, not the
allocated one.

Hardware differs by arm and is recorded per row: labs on RTX 2080 Ti (11 GB),
labs+notes and labs+notes+CXR on RTX A6000 / RTX 6000 Ada (48 GB), seed 5 on
campus-cluster A10s. Within an arm the hardware is consistent; across arms it
is not, so cross-arm runtime is not a like-for-like comparison.

Systems measurements

Metric convention

Field Meaning
peak_pss_mb Primary. Proportional set size across the process tree; shared pages divided among mappers, so the total is additive.
peak_rss_parent_mb Parent process only. Flat across worker counts by construction.
peak_rss_mb Summed RSS over the tree. Not a valid memory figure -- kept as a negative control.

Summed RSS double-counts copy-on-write pages; it reported 1.89 TB on a 480 GB node for the 64-worker PyHealth 1.16 baseline. PSS is the additive figure.

Measurement variability

Repeated runs of identical configurations on shared scavenger nodes differ by up to 1.82x:

Context Impl Workers Spread (max/min)
uncontrolled ph2_ingest 8 1.82x
uncontrolled ph2 16 1.12x
uncontrolled ph116 1 1.06x
uncontrolled ph2 32 1.06x
uncontrolled ph2 8 1.03x

Every ratio below is a median over repeats, not a single run. With n=2 per configuration these are indicative, not tight; 5+ repetitions with randomised order is the standard this should eventually meet.

Strong scaling, thread budget pinned

Workers Wall clock Speedup Efficiency Karp-Flatt e
1 7704.0s - - -
2 3943.7s 1.95x 0.977 0.0238
4 2047.7s 3.76x 0.941 0.0211
8 864.5s 8.91x 1.114 -0.0146 super-linear
16 425.1s 18.12x 1.133 -0.0078 super-linear
32 239.0s 32.23x 1.007 -0.0002 super-linear

Worker sweep, threads unpinned

Workers Wall clock Speedup Efficiency Karp-Flatt e
1 4346.5s - - -
2 2792.5s 1.56x 0.778 0.2849
4 1981.1s 2.19x 0.548 0.2744
8 658.5s 6.60x 0.825 0.0303
16 273.9s 15.87x 0.992 0.0005
32 163.9s 26.52x 0.829 0.0067

No single speedup factor is claimed. Efficiency should fall monotonically at fixed problem size; both sweeps exceed 1.0 and both produce a negative Karp-Flatt serial fraction. Pinning the thread budget (workers x threads = 64, exclusive node) did not remove it, so this is not a measurement artifact: at N=1 the work runs as one process with 64 polars threads, and thread-level parallelism is simply less efficient here than process-level. The 1-worker baseline is therefore intrinsically handicapped and any speedup quoted against it is inflated. What is defensible is the absolute wall clock at each worker count, reported above.

Naive pandas baseline

Pipeline Wall clock Peak PSS Output
Hand-rolled pandas 110.0s 16.5 GB one .npz of trainable arrays
PyHealth 2.0 task layer, 32 workers 163.9s 32.1 GB chunked streaming dataset

A competent hand-rolled pandas pipeline producing the same per-patient lab tensors -- it imports LAB_CATEGORIES from the task, so the two agree on itemid mapping by construction -- is not slower than the framework task layer. We do not claim a speed win over it.

The claim that holds is memory: pandas needs its peak in a single process, so it will not run on a 16 GB machine, while the framework at one worker peaks near 3.9 GB and will. Caveat: the pandas arm produces 140,334 patient records against 180,733 samples, because it does not replicate the admission-selection logic.

labevents ingest: PyHealth 2.0 vs 1.16

Quantity Median ratio n per arm Spread
Wall clock, 2.0 faster by 5.05x 2 / 2 1.06x / 1.82x
Peak PSS, 2.0 lower by 5.15x
Peak parent RSS, 2.0 lower by 5.88x

The 2.0 ingest arm itself varies 1.82x between repeats, so the speed ratio should be read as approximate. Comparison is conservative toward 1.16 on three axes: 1.16 used 64 pandarallel workers against 8, read pre-decompressed CSV while 2.0 paid gzip decompression inside the timed region, and kept 180,733 patients against 299,712 ingested.

Hardware accessibility

Measured on a 2018 consumer card, not extrapolated from allocated-memory
figures. One GPU per cell throughout.

Arm RTX 2080 Ti, 11 GB Result
labs batch 32 runs, ~400 MiB, ~2 min/epoch (all 12 labs cells ran here)
labs+notes batch 32 / 16 / 8 OOM -- process needs 10.19 GiB on a 10.57 GiB card
labs+notes batch 4 crashes at ~10% of epoch 1: RuntimeError: Cannot pack empty tensors
labs+notes+CXR batch 32 OOM

labs+notes does not currently run on an 11 GB card at any batch size, for
two independent reasons.

Memory. The binding constraint is the frozen 108.3M-parameter text encoder,
not the backbone (33K-2.6M). Allocated-memory figures understate this badly:
max_memory_allocated reports ~5.9 GB while the process actually holds
10.19 GiB.

A small-batch bug. 17.2% of samples carry no notes. At batch 4 the
probability that every sample in a batch is empty is 0.172^4, which over 36,147
batches means ~32 expected occurrences per epoch -- effectively certain -- and
the RNN path then calls pack_padded_sequence on an empty tensor. At batch 32
the same probability is negligible, which is why the defect never appears at the
protocol batch size. Lowering the batch size to fit a smaller card is therefore
not currently a workaround.

The structured-EHR arm is fully reproducible on a six-year-old consumer GPU at
the protocol batch size. The text arms need a >=24 GB card, and the empty-batch
defect should be fixed before small-batch operation is advertised.

Frozen-text cache ablation

The frozen Bio_ClinicalBERT [CLS] cache is keyed on token ids and capped by
--max-frozen-text-cache. Same cell (labs+notes / RNN / batch 32), one epoch,
one A6000, cap varied:

Cap Epoch time Speedup vs no cache Peak VRAM
1 (caching effectively off) 2249s 1.00x 6807 MB
200,000 1452s 1.55x 7885 MB
1,000,000 (default) 1404s 1.60x 6469 MB
0 (no cap) 1454s 1.55x 6921 MB

The cache is worth 1.60x. Note the flag's own help text states "200k is too
small for full MIMIC"; measured, a 200k cap recovers 97% of the benefit
(1452s vs 1404s), so that guidance is wrong for this cohort. Raising the cap
past 200k buys 3%, and removing the cap entirely is slightly worse than the
default -- an unbounded cache costs more than it returns.

Batch size vs VRAM

labs+notes / RNN, one A6000, one epoch each. Establishes what a smaller card
needs, since the 2080 Ti runs only produced pass/fail:

Batch Peak VRAM (allocated) Epoch time
8 4124 MB 1927s
16 5592 MB 1908s
32 6469 MB 1404s
64 9827 MB 1370s

VRAM grows 2.4x from batch 8 to 64 while epoch time improves only 29%, so there
is little throughput argument for a large batch here -- useful if memory is the
binding constraint. Allocated VRAM understates the real requirement (see
Hardware accessibility): a cell reporting 6469 MB allocated held 10.19 GiB as a
process.

Cohort characterisation

Measured directly from the cache the models trained on (every 2nd sample,
n=90,367 of 180,733), so it describes the actual cohort rather than a
separately-derived one.

value
Samples 180,733
In-hospital mortality 4.79% (4,331 of 90,367 scanned)
Test split size 18,074 (identical across all three arms)

Modality availability

Modality Samples with >=1 event Mean events p50 p90 p99 max
Labs 76.34% 13.6 3 35 153 1186
Notes 82.80% 5.8 2 14 51 235
CXR 14.71% 0.8 0 2 16 159

Informative missingness

Modality presence is itself prognostic, which bounds how much of a multimodal
gain can be attributed to content:

Modality Mortality when present when absent ratio
Notes 5.71% (n=74,823) 0.37% (n=15,544) 15.30x
CXR 11.45% (n=13,295) 3.64% (n=77,072) 3.14x

For notes, ~98.7% of deaths fall in the has-notes group, so a presence-only
predictor reaches AUROC ~0.58 but only 1.19x precision lift over the 4.79%
base rate. Real, but far too weak to account for the observed notes gain.

Cross-cluster cache determinism

The task cache key is a uuid5 over the task's public fields and schemas, so it is
content-addressed. Building the same tasks independently on two unrelated
clusters -- different OS, filesystem (NFS vs Lustre), CPU count and scheduler --
produced byte-identical keys:

Task Key on both clusters
LabsMIMIC4 0091954e-3d3e-5416-8983-392885bad7d9
NotesLabsMIMIC4 9f184752-4ff5-5998-8a26-c7e5a16ff99b

Neither build saw the other. This is a stronger reproducibility statement than
pinned versions: it shows a cache built elsewhere is the same cache, so a
stale one cannot be silently reused after emitted data changes, and results
transfer between sites without re-deriving the cohort.

Pretraining contamination (BioClinicalBERT / MIMIC-III)

Bio_ClinicalBERT was pretrained on MIMIC-III notes. MIMIC-III and MIMIC-IV
overlap in the 2008-2012 collection window, so some evaluation patients may
appear in the encoder pretraining corpus. MIMIC-IV regenerated every patient
identifier, so that overlap cannot be resolved per patient.

What can be done: anchor_year_group buckets patients by era, and anyone
anchored 2014 onward falls after MIMIC-III coverage ends. 130,559 of 299,712
patients (43.5%) are in that range. Recomputing the labs to labs+notes gain
separately per stratum, from existing predictions, with no retraining:

Stratum Test n Mortality Comparisons Mean notes gain
Overlap era (<=2013), may be in pretraining 10,658 5.06% 15 +0.0912
Post-MIMIC-III (>=2014), cannot be 7,416 4.71% 15 +0.1095

The gain is larger on patients the encoder cannot have seen (+0.1095 vs
+0.0912), and positive in all 30 model x seed comparisons. Contamination would
predict the opposite ordering, so it is not what produces the notes gain.

Caveats: anchor_year_group reflects a patient anchor year rather than every
admission, so this is a strong proxy for non-overlap rather than a proof; and
the two strata differ in era-related ways beyond contamination. The direction
of the difference is the informative part.

Parameter counts

Counted from the shipped best.ckpt, so they describe exactly what trained.

Task Backbone params Frozen text Image enc Total
labs 33,153 (MLP) - 2,641,537 (JambaEHR) 0 - 36,191 - 2,644,575
labs+notes same range 108,310,272 - 108.4M - 111.1M
labs+notes+CXR same range 108,310,272 98,432 108.5M - 111.2M

Per backbone: MLP 33,153 | RNN 99,201 | Transformer 395,649 |
Bottleneck 397,313 | EHRMamba 748,673 | JambaEHR 2,641,537.

In the multimodal arms the backbone under comparison is 0.03%-2.44% of total
parameters; the frozen text encoder is the rest. This is the honest framing
for a benchmark that varies only the backbone -- the varied component is a
small fraction of model capacity, which is consistent with modality effects
dominating architecture effects throughout these tables.

The CXR encoder is 98,432 randomly-initialised parameters (0.09% of the
model) against 14.71% modality coverage, which is the context for the CXR arm's
results.

Training stability

No NaN losses and no skipped optimizer steps in any completed cell, and
every backbone uses identical optimizer settings -- there is no per-model
hyperparameter exception. An ablation with --no-lab-standardization also
trained cleanly (bottleneck transformer, 12 epochs, monotone loss decrease), so
input standardisation is not the mechanism; it is worth +0.0315 val PR-AUC
(0.6231 -> 0.6545) but is not what prevents divergence.

Reproducibility

Every cell writes run_config.json recording the resolved settings plus
source_sha256. All cells in these tables share one source_sha256
(b2d601b4f547), which is what demonstrates the table came from a single
build. git.dirty reads true on every run and is a false positive: the
launcher creates logs/ and wandb writes wandb/ inside the tree, so
git status sees untracked directories while git diff HEAD is empty.

Findings

Computed from 78 cells across seeds [1, 2, 3, 4, 5]. Every modality delta is paired -- same backbone, same seed, same split -- because the spread between backbones is far larger than the modality effect and unpaired means would drown it.

Modality effects against the seed noise floor

Effect n Mean delta SD Positive vs noise floor
labs -> +notes 24 +0.0917 0.0202 24/24 4.93x
+notes -> +CXR 24 -0.0002 0.0181 13/24 0.01x

Noise floor is the mean absolute difference between seeds for the same backbone and arm: labs 0.0160, notes_labs 0.0186, notes_labs_cxr 0.0185.

Notes help, unanimously. 24 of 24 paired comparisons positive, range +0.0578 to +0.1590, and the effect is ~4.93x the seed noise floor. Positive for every backbone at every seed.

Seed n Mean All same sign
1 6 +0.1126 yes (+)
3 6 +0.0961 yes (+)
4 6 +0.0752 yes (+)
5 6 +0.0829 yes (+)

CXR contributes nothing measurable. Mean -0.0002, positive in 13 of 24 -- a coin flip -- and 0.01x the seed noise floor. The sign is set by the seed, not the model:

Seed n Mean All same sign
1 6 -0.0246 yes (-)
3 6 +0.0114 yes (+)
4 6 +0.0063 no
5 6 +0.0062 no

A single-seed ablation would have reported a confident direction here, and which direction would have been luck of the split. Context: the image encoder is 98,432 randomly-initialised parameters against 14.71% modality coverage.

Early stopping behaviour

Backbone Median epochs run Median best epoch Mean test PR-AUC
MLP 21 16 0.5436
Transformer 15 10 0.6519
Bottleneck Transformer 13 8 0.6749
RNN 12 7 0.8114
EHRMamba 8 3 0.7805
JambaEHR 7 2 0.7933

Training length runs opposite to accuracy: the weakest backbone trains longest (MLP, median 21 epochs) and the strongest stop earliest (EHRMamba 8, JambaEHR 7). With patience 5 on val PR-AUC that is consistent with the weaker models never finding a good optimum rather than converging to one -- so equal epoch budgets do not mean equal optimisation difficulty.

Logiquo and others added 30 commits April 12, 2026 12:19
@Logiquo
Fix Interpretability Methods target_class_idx (sunlabuiuc#926)
39b86e4 
* rename arg name for chefer

* Initial attempts to fix the interpretability target_class_idx

* Support negative prediction for interpretability metric.

* Fix tests

* Fix more tests

* Revert "Support negative prediction for interpretability metric."

This reverts commit fe8c8ad.

* Reapply "Support all samples for interpretability metric"

* Initial attempt for the filter

* Fixup

* Fix sample_class handling

* fixup

* fix test

* Fix arg name

* Add example

* fix docs
@jhnwu3
small fix + bump to pyproject.toml ver. for bug fixed release on pypi (…
f7dd848 
…sunlabuiuc#927)

* small fix + bump to pyproject.toml ver. for bug fixed release on pypi

* We don't really have someone qualified for a second review, and this broken CI is leading to a lot of issues here.

Will revert if it doesn't resolve here.
@haoyu-haoyu
test: add unit tests for RNN and MultimodalRNN models (sunlabuiuc#936)
8004398 
Add test_rnn.py with 12 test cases covering:

TestRNN (8 tests):
- Model initialization with correct attributes
- Forward pass output structure and shapes
- Backward pass gradient propagation
- Embedding extraction via embed=True
- Custom hyperparameters (embedding_dim, hidden_dim)
- LSTM cell type variant
- Vanilla RNN cell type variant
- Bidirectional RNN variant

TestMultimodalRNN (4 tests):
- Initialization with correct sequential/non-sequential classification
- Forward pass with mixed modalities (sequence + multi_hot + tensor)
- Backward pass gradient propagation
- Embedding extraction with correct mixed-modality dimensions

Follows the established test pattern from test_mlp.py and test_tcn.py
using create_sample_dataset with synthetic data.

Ref sunlabuiuc#425
@lehendo
Conformal Methods and Scripts (sunlabuiuc#942)
8c0f157 
* Fixed repo to be able to run TUEV/TUAB + updated example scripts

* Args need to be passed correctly

* Minor fixes and precomputed STFT logic

* Fix the test files to reflect codebase changes

* Args update

* test script fixes

* dataset path update

* fix contrawr - small change

* divide by 0 error

* Incorporate tfm logic

* Fix label stuff

* tuab fixes

* fix metrics

* aggregate alphas

* Fix splitting and add tfm weights

* fix tfm+tuab

* updates scripts and haoyu splitter

* fix conflict

* Remove weightfiles from tracking and add to .gitignore

Weight files are large binaries distributed separately; untrack all
existing .pth files under weightfiles/ and add weightfiles/ to
.gitignore so they are excluded from future commits and the PR.

Made-with: Cursor

* normalization = 95%

* temporarily re-add weight files

* 16 workers

* tuab sanity check

* consistent log outputs

* test tuab

* change back to multiclass

* update conformal scripts

* remove weightfiles

* oops

* fix tests
@capccode @lookman-olowo
@christyanamarie @ddhangdd Colton Loew lookman-olowo christiana-beard ddhangdd
Dev/grasp full pipeline (sunlabuiuc#905)
7d95dea 
* feat: migrate GRASP model from PyHealth 1.0 to 2.0 API

Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* feat: add GRASP mortality prediction notebook and fix cluster_num

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* Restore code_mapping support in SequenceProcessor for PyHealth 2.0

Adds optional code_mapping parameter to SequenceProcessor that maps
granular medical codes to grouped vocabularies (e.g. ICD9CM→CCSCM)
before building the embedding table. Resolves the functional gap
from the 1.x→2.0 rewrite where code_mapping was removed. Ref sunlabuiuc#535

Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>

* Add RNN baseline and code_mapping comparison notebooks for MIMIC-III

Two identical notebooks for A/B testing code_mapping impact on mortality
prediction. Only difference is the schema override in Step 2. Both use
seed=42 for reproducible splits.

Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>

* fix(tasks): extract NDC codes instead of drug names for prescription mapping

event.drug returns drug names (e.g. "Aspirin") which produce zero matches
in CrossMap NDC→ATC; event.ndc returns actual NDC codes enabling 3/3
feature mapping for mortality and readmission tasks.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* test(tasks): add tests verifying NDC extraction in drug tasks

Checks that mortality and readmission task processors build vocabulary
from NDC codes (numeric strings) rather than drug names (e.g. "Aspirin"),
confirming the event.drug -> event.ndc fix works correctly.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* fix(tasks): fix missed MortalityPredictionMIMIC4 event.drug and update docs

- Fix event.drug -> event.ndc in MortalityPredictionMIMIC4 (line 282)
- Update readmission task docstrings to reflect NDC extraction

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* fix(tasks): fix DrugRecommendationMIMIC3 to extract NDC codes

DrugRecommendationMIMIC3 used prescriptions/drug (drug names) via Polars
column select; changed to prescriptions/ndc to match MIMIC-4 variant and
enable NDC->ATC code mapping.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* fix(models): guard RNNLayer and ConCare against zero-length sequences

RNNLayer: clamp sequence lengths to min 1 so pack_padded_sequence
does not crash on all-zero masks, matching TCNLayer (tcn.py:186).

ConCare: guard covariance divisor with max(n-1, 1) to prevent
ZeroDivisionError when attention produces single-element features.

Both edge cases are triggered when code_mapping collapses vocabularies
and some patients have all codes map to <unk>, producing all-zero
embeddings and all-zero masks.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* docs: add docstrings to SequenceProcessor class and fit method

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* docs: add docstrings, type hints, and fix test dims for GRASP module

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* feat: add GRASP mortality prediction notebooks for baseline and code_mapping

Baseline notebook runs GRASP with raw ICD-9/NDC codes. Code_mapping
notebook collapses vocab via ICD9CM→CCSCM, ICD9PROC→CCSPROC, NDC→ATC
for trainable embeddings on full MIMIC-III.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* fix(models): guard ConCare and GRASP against batch_size=1 crashes

- ConCare FinalAttentionQKV: bare .squeeze() removed batch dim when
  batch_size=1, causing IndexError in softmax. Use .squeeze(-1) and
  .squeeze(1) to target only the intended dimensions.
- ConCare cov(): division by zero when x.size(1)==1. Guard with max().
- GRASP grasp_encoder: remove stale torch.squeeze(hidden_t, 0) that
  collapsed [1, hidden] to [hidden] with batch_size=1. Both RNNLayer
  and ConCareLayer already return [batch, hidden].
- GRASP random_init: clamp num_centers to num_points to prevent
  ValueError when cluster_num > batch_size.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* feat: add GRASP mortality prediction notebooks for baseline and code_mapping

Baseline notebook runs GRASP with raw ICD-9/NDC codes. Code_mapping
notebook collapses vocab via ICD9CM→CCSCM, ICD9PROC→CCSPROC, NDC→ATC
for trainable embeddings on full MIMIC-III.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* Add code_mapping as task __init__ argument

Allow tasks to accept a code_mapping dict that upgrades input_schema
entries so SequenceProcessor maps raw codes (e.g. ICD9CM) to grouped
vocabularies (e.g. CCSCM) at fit/process time. This avoids manual
schema manipulation after task construction.

- Add code_mapping parameter to BaseTask.__init__()
- Thread **kwargs + super().__init__() through all task subclasses
  with existing __init__ methods (4 readmission tasks, 1 multimodal
  mortality task)
- Add 17 tests covering SequenceProcessor mapping and task-level
  code_mapping initialization

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* Update code_mapping notebook to use task init argument

Replace manual task.input_schema override with the new
code_mapping parameter on MortalityPredictionMIMIC3().

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* feat(examples): add ConCare hyperparameter grid sweep script

Mirrors the GRASP+ConCare mortality notebook pipeline exactly
(same tables, split, seed, metrics) but sweeps 72 configurations
of embedding_dim, hidden_dim, cluster_num, lr, and weight_decay.

Results are logged to sweep_results.csv. Supports --root for
pointing at local MIMIC-III, --code-mapping, --dev, and --monitor.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* chore(sweep): increase early stopping patience from 10 to 15 epochs

Smaller ConCare configs (embedding_dim=8/16) may learn slower and
need more epochs before plateauing.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* Initial plan

* fix: filter falsy NDCs, guard None tokens in process(), fix NDC regex

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-authored-by: ddhangdd <43976109+ddhangdd@users.noreply.github.com>

* refactor(sweep): rename and generalize sweep script for all backbones

Rename sweep_concare_grasp.py → sweep_grasp.py. Now supports
--block GRU|ConCare|LSTM with per-backbone default grids, --resume
for crash recovery, --grid JSON override, auto-dated output dirs
(sweep/{BLOCK}_{YYYYMMDD}_{HHMMSS}_{mapping}/), and config.json
saved alongside results for reproducibility.

Co-Authored-By: Colton Loew <colton.loew@gmail.com>
Co-Authored-By: lookman-olowo <lookmanolowo@hotmail.com>
Co-Authored-By: christiana-beard <christyanamarie116@gmail.com>
Co-Authored-By: ddhangdd <dfung2@wisc.edu>

* test(sweep): add unit and integration tests for sweep_grasp utilities

Covers grid building, combo hashing, CSV resume parsing, output
directory naming, and end-to-end single-config runs for GRU and ConCare
on synthetic data (13 tests, all passing).

Co-Authored-By: Colton Loew <loewcx@illinois.edu>
Co-Authored-By: lookman-olowo <lookman-olowo@github.com>
Co-Authored-By: christiana-beard <christiana-beard@github.com>
Co-Authored-By: ddhangdd <ddhangdd@github.com>

* docs(sweep): add tmux copy-paste instructions for each paper run

Co-Authored-By: Colton Loew <loewcx@illinois.edu>
Co-Authored-By: lookman-olowo <lookman-olowo@github.com>
Co-Authored-By: christiana-beard <christiana-beard@github.com>
Co-Authored-By: ddhangdd <ddhangdd@github.com>

* chore(examples): adds cleans examples, removes util script

* Delete tests/core/test_grasp.py

we removed grasp script from examples, dropped test

* Revert "Delete tests/core/test_grasp.py"

This reverts commit 0d95758.

* fix: remove orphaned sweep test, restore grasp tests

* feat(grasp): add static_key support for demographic features with tests

* fix(test): add valid NDC to test prescriptions so readmit test produces both labels

---------

Co-authored-by: lookman-olowo <lookmanolowo@hotmail.com>
Co-authored-by: christiana-beard <christyanamarie116@gmail.com>
Co-authored-by: ddhangdd <dfung2@wisc.edu>
Co-authored-by: Lookman Olowo <42081779+lookman-olowo@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ddhangdd <43976109+ddhangdd@users.noreply.github.com>
Co-authored-by: ddhangdd <desmondfung123@gmail.com>
Co-authored-by: Colton Loew <loewcx@illinois.edu>
Co-authored-by: lookman-olowo <lookman-olowo@github.com>
Co-authored-by: christiana-beard <christiana-beard@github.com>
Co-authored-by: ddhangdd <ddhangdd@github.com>
Co-authored-by: lookman-olowo <lookman-olowo@users.noreply.github.com>
@mtmckenna
Add PhysioNet De-Identification dataset, NER task, and TransformerDeI…
5f63039 
…D model (sunlabuiuc#981)
@KobeGuo99
dl4h final project kobeguo2 - CaliForest (sunlabuiuc#999)
144c06a 
* dl4h final project kobeguo2 - CaliForest

* Update CaliForest to require explicit fit before inference

* Remove unused logit_scale from CaliForest
@Logiquo
Fix drug recommandation drug code and padding. (sunlabuiuc#1138)
1262a76 
* Fix Drug Recommandation NDC/ATC3 code

* Fix padding behaviour

* remove .codex file

* Change test from FakePatient to demo dataset
@Logiquo
Fix sparsemax in AdaCare (sunlabuiuc#1139)
3fcf8d2
@jhnwu3
reinitialize tutorials for documentation as old tutorials died with U…
cb75b5c 
…IUC purge (sunlabuiuc#1143)

literally just updating the examples/ no need to waste reviewer time.
@janezdu
Colab Tutorial re-link (sunlabuiuc#1146)
acaf7b7 
* add back backups of original tutorials

* Backup lost tutorials

* generate new tokenizer tutorial

* update with pip install d4rl install pyhealth and rename

* update colab references
@chufangao @claude
Add synthetic-EHR generative evaluation metrics (sunlabuiuc#1148)
53fa513 
* Add synthetic-EHR generative evaluation metrics

Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic
EHR data along privacy, utility, and statistical-fidelity axes:

- privacy.py: NNAAR, membership inference attack, discriminator privacy
- utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence
  similarity (R2, Pearson, RMSE)
- utils.py: shared data prep, an LSTM classifier, and a random-forest
  baseline
- evaluate_synthetic_ehr(): convenience orchestrator for the full suite

These functions are ported from a standalone evaluation script. The
MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any
flat EHR dataframe. Public functions are re-exported from
pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py
and Sphinx docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add synthetic-EHR generative evaluation metrics

Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic
EHR data along privacy, utility, and statistical-fidelity axes:

- privacy.py: NNAAR, membership inference attack, discriminator privacy
- utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence
  similarity (R2, Pearson, RMSE)
- utils.py: shared data prep, an LSTM classifier, and a random-forest
  baseline
- evaluate_synthetic_ehr(): convenience orchestrator for the full suite

These functions are ported from a standalone evaluation script. The
MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any
flat EHR dataframe. Public functions are re-exported from
pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py
and Sphinx docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* add baselines

* removed halo save file and updated promptehr to be more paper accurate

* update docs

* update docs

* Update pyhealth.models.HALO.rst

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jhnwu3 @claude
transfer FHIR pipeline to branch (sunlabuiuc#1155)
fb481d9 
* transfer FHIR pipeline to branch

* fix

* fix unit test using fast json readers

* Replace editdistance with rapidfuzz for Python 3.13 compatibility

editdistance 0.8.1 only ships cp311 wheels and has no Python 3.13
binary, causing CI installs to fail on Linux. rapidfuzz>=3.0.0
ships wheels for all major platforms including cp313 and provides
an equivalent Levenshtein.distance() API.

https://claude.ai/code/session_01L5qHpvAZQSgmZyc6tMTX6d

* copilot fixes

* revert ignore error change

---------

Co-authored-by: Claude <noreply@anthropic.com>
@seannian
[DL4H Sp26] Add MedFuse model for multi-modal EHR + chest X-ray fusion (
17338b4 
sunlabuiuc#1003)

* Add MedFuse multimodal model for EHR+CXR fusion

* Address PR feedback: docstring, mask doc, test cleanup, paper citations
@fbonc
Add attention rollout interpretability method (Abnar & Zuidema 2020) (s…
820e5ee 
…unlabuiuc#1158)

* Attention Rollout skeleton

* attention_rollout.py done

* tests/core/test_attention_rollout.py done

* attention rollout integrated into example scripts

* attention rollout docs

* attention rollout docs/interpret/pyhealth.interpret.methods.attention_rollout.rst added

* Stip trailing whitespace and rename example keys to rollout

* attention rollout: doc and style changes

* attention rollout: module docstring header
@jhnwu3 @claude
Add CI gate enforcing PR contribution rules for pyhealth/ changes (su…
c6a63ee 
…nlabuiuc#1176)

Any PR touching pyhealth/**/*.py must also update docs/ and examples/,
keep added/modified lines free of ruff lint violations, and give new or
modified top-level public classes/functions a '>>>' docstring example.

Co-authored-by: Claude <noreply@anthropic.com>
@vihaan101
Add EEGBCI dataset and tasks (sunlabuiuc#1177)
546c1ad 
* feat: add EEGBCI helper functions

* feat: add EEGBCI dataset

* feat: add EEGBCI tasks

* test: add opt-in EEGBCI real-data smoke test

* docs: add EEGBCI pattern discovery example

* docs: add EEGBCI API docs

* chore: record EEGBCI verification

* docs: refine EEGBCI moment report design

* Add EEGBCI moment report constants

* Add EEGBCI rest baseline helpers

* Add EEGBCI state scoring helpers

* Add EEGBCI task state quality helpers

* Add EEGBCI moment row annotation

* Add EEGBCI representative windows

* Render EEGBCI moment summary

* Wire EEGBCI moment report main flow

* Document EEGBCI moment report outputs

* Fix EEGBCI moment report review findings

* Polish EEGBCI report artifact

* Exclude EEG pattern discovery notes

* fix: address EEGBCI review feedback

* fix: address EEGBCI review feedback

* fix: satisfy PR contribution rules
@lehendo
Covariate Shift Conformal Prediction Fixes (sunlabuiuc#1180)
132c9fb 
* Covariate CP fixes

* small edits to pass checks
@lehendo
Add missing citations in PyHealth code (sunlabuiuc#1181)
ef48918 
* Add missing citations for TFMTokenizer, EHRMambaCEHR, CEHR embeddings, comprehensiveness/sufficiency metrics, MLE, and NNAAR

* to pass the pr checks
@AxelNoun @cursoragent
Add MEDS dataset support (MEDSDataset + typed Parquet scan path) (sun…
841be53 
…labuiuc#1179)

* feat(datasets): add Parquet scan path to BaseDataset

Route .parquet/.pq files, globs, and directories through a typed
_scan_parquet scanner; keep CSV/TSV(.gz) on the existing path. Add a
datetime fast-path in load_table that skips the string round-trip and
casts to datetime64[ms], preserving NaT for static events.

* feat(datasets): add MEDSDataset for the Medical Event Data Standard

Declarative YAML wrapper over the shared Parquet scan path, with
split_source subset selection (metadata or directory layout), distinct
processing caches per subset, and a construction-time Parquet footer
schema guard that rejects missing, non-timestamp, or timezone-aware
time columns.

* test(datasets): add MEDS synthetic and demo smoke tests

Deterministic sharded Parquet fixtures cover nested splits, subset
filtering, cache isolation, set_task smoke, and construction-time
schema-guard TypeErrors. Demo smoke stays skip-gated behind
MEDS_DEMO_ROOT / test-resources/meds_demo (gitignored).

* docs(examples): add MEDS example and API docs

Document MEDSDataset in the API reference and add an end-to-end
examples/meds_demo.py against the public PhysioNet MIMIC-IV MEDS demo.

* feat(tasks): add InHospitalMortalityMEDS

MEDS-native in-hospital mortality task: one sample per completed stay,
reconstructed by joining HOSPITAL_ADMISSION/HOSPITAL_DISCHARGE events on
hadm_id. Half-open [admit, prediction_time) observation window (full_stay
default; first_hours early-warning variant), label from the
HOSPITAL_DISCHARGE//DIED discharge code. Discharge and MEDS_DEATH events
are excluded from features to prevent label leakage.

hadm_id is dataset-specific (not part of the core MEDS schema), so it is
exposed via a bundled configs/meds_with_hadm.yaml rather than the default
config.

Verified on the public MIMIC-IV demo in MEDS: 12 positive / 238 stays
(rate 0.0504); set_task sample count 238.

- pyhealth/tasks/in_hospital_mortality_meds.py (+ __init__ export)
- pyhealth/datasets/configs/meds_with_hadm.yaml
- tests/core/test_in_hospital_mortality_meds.py
- examples/verify_meds_mortality.py
- docs/api/tasks/pyhealth.tasks.InHospitalMortalityMEDS.rst (+ tasks.rst toctree)

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tasks): re-export InHospitalMortalityMEDS with explicit alias

Satisfies ruff F401 on the newly added __init__ line under
tools/check_pr_rules scoped lint (same pattern as eegbci).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: link MEDS schema docs for subject_splits mapping

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor: rename _reconstruct_stays to _group_stays and clarify summarize docstring

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: add end-to-end RNN training to MEDS demo

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: link subject_splits in MEDSDataset API page; qualify demo metrics output

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor: drop summarize helper from InHospitalMortalityMEDS

Co-authored-by: Cursor <cursoragent@cursor.com>

* style: modernize typing annotations, drop unused noqa (UP006/UP035/UP045/RUF100)

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
@will-pang
Initial Push
a92c25a
@will-pang
Updates
bc05304
William Pang
New Changes
5483dee
William Pang
New updates
a0f1422
William Pang
New Updates
b80f7ef
William Pang
New Updates
f0187f9
William Pang
New Updates
b91ec9e
William Pang
New Updates
2b7b2c5
@lehendo
Implements real APS & dynamic scoring for Conformal Prediction (sunla…
0a75f99 
…buiuc#1189)

* Implement real APS and add dynamic score_type to conformal methods

Adds pyhealth/calib/predictionset/scores.py, a shared score module
implementing genuine Adaptive Prediction Sets (Romano, Sesia, and
Candes 2020): nonconformity score = cumulative sum of predicted
probabilities for classes ranked above the target, plus a randomized
U*p(target) term (U ~ Uniform(0,1), one draw per example, shared
across all candidate classes). Previously, BaseConformal's
score_type="aps" was silently just an alias for "threshold" and did
not implement APS at all.

Threads a new score_type parameter ("threshold" [default, unchanged
behavior] or "aps") through BaseConformal, LABEL, ClusterLabel,
CovariateLabel, and NeighborhoodLabel, each with an optional
random_state for reproducible APS randomization. SCRIB and FavMac are
intentionally excluded since their calibration isn't a score-then-
quantile pattern.

Verified via numpy-only synthetic tests: both score types hit ~90%
empirical coverage at alpha=0.1, for marginal and class-conditional
coverage, in both nonconformity and conformity sign conventions.

* Add scores.py doctests, aps usage examples, tests, and docs

Adds >>> usage examples to the 4 public functions in
pyhealth/calib/predictionset/scores.py (verified against real computed
output). Adds a score_type="aps" usage example to the docstrings of
BaseConformal, LABEL, ClusterLabel, CovariateLabel, and
NeighborhoodLabel.

Adds tests/core/test_scores.py covering both score types: threshold
backward-compatibility, the APS formula's hand-computable non-randomized
case, monotonicity, reproducibility under a seeded RNG, nonconformity/
conformity complementarity, and empirical marginal coverage at the
target alpha. Extends test_cluster_label.py, test_covariate_label.py,
and test_neighborhood_label.py with score_type="aps" end-to-end cases.

Documents the score_type argument and adds the previously-missing
BaseConformal entry to docs/api/calib/pyhealth.calib.predictionset.rst.

* Fix ruff lint violations flagged by CI (UP/RUF rules)

CI's ruff install (pip install 'ruff~=0.15') resolves to the latest
0.x release under PEP 440 compatible-release semantics, which enabled
more pyupgrade/ruff-specific default rules than the older cached ruff
used for local verification. Fixes all 13 flagged violations:
Optional[X]/Union[X, Y] -> X | Y, typing.Dict -> dict (including the
now-modernized pre-existing forward() return annotations this forced),
an unused unpacked variable, and an unsorted __all__.

Verified by reproducing the CI's exact environment: a clean venv with
`pip install 'ruff~=0.15'` (which also resolves to 0.16.3), confirming
`tools/check_pr_rules.py` now passes.
@will-pang
Stop emitting fake missing-event placeholders (9782aca)
a7852f4
will-pang and others added 23 commits August 26, 2026 15:58
@will-pang
Default to a full stay and stamp admission-context notes at admit.
4b78f28
@will-pang
Add scripts for lamba labs and update condor scripts
32ca3a6
@will-pang
Merge branch 'sunlabuiuc:master' into ml4h-merge-tranche-1-wp-20260825
448d93a
@will-pang
Create pyhealth2_environment.yml
b4f88f9
@will-pang
Merge branch 'ml4h-merge-tranche-1-wp-20260825' of https://github.com…
ba3b7a0 
…/Multimodal-PyHealth/PyHealth into ml4h-merge-tranche-1-wp-20260825
@will-pang
Update pyhealth2_environment.yml
731c8c6
@rianfyi
Add rian/scripts: notes_labs seed-1 paper-cell launchers (MLP, JambaE…
3ae1a2a 
…HR, BottleneckTransformer) with results.

Claude-Session: https://claude.ai/code/session_01U3rt6DmCQ2NiPmfwujzoje
@rianfyi
rian/scripts: add cache-500k and unfrozen-BERT launchers; note the ov…
0d8907d 
…erflow-path fix.

Claude-Session: https://claude.ai/code/session_01U3rt6DmCQ2NiPmfwujzoje
@will-pang
Add MLP in _build_model
deacd03
@will-pang
Merge branch 'ml4h-merge-tranche-1-wp-20260825' of https://github.com…
fb7fda4 
…/Multimodal-PyHealth/PyHealth into ml4h-merge-tranche-1-wp-20260825
@will-pang
Default the frozen [CLS] cache cap to 1e6 and log cache growth.
fb272d2
@will-pang
Bound trainable text-encoder memory with gradient checkpointing and c…
659610e 
…hunked note rows.
@will-pang
Encode a frozen-cache miss once per forward, even when the cache is f…
4e158a6 
…ull.
@will-pang
Add Will's Run Scripts
be65739
@will-pang
Update pyhealth environment file and will run scripts
ef7430c
@will-pang
Avoid ambiguous -1 reshape when a batch has no note or image slots.
f8fefc0 
Replace the trailing -1 in .view()/.reshape() calls for the text and
  image branches with the known embedding dim (or an explicit empty
  mask), since torch can't infer -1 for a 0-element tensor and raises
  "cannot reshape tensor of 0 elements ... dimension size -1 is
  ambiguous" when a batch has zero note or image slots.
@will-pang
Create Will's CXR Labs Notes variant scripts
fc7050a
@will-pang
Add labs only scripts
de7c37b
@will-pang
Unify event timestamps in pyhealth/tasks/multimodal_mimic4.py onto a …
cdeb3e0 
…single per-sample clock.

  Previously, time offsets were computed inconsistently across modalities within a multi-admission sample:
  - ICD codes: hours since the previous admission (a delta, reset to 0 for the first admission).
  - Labs, notes, CXR: hours since that admission's own start (reset to 0 at every admission).

  Since labs, admission_note_times, and cxr_image_times concatenate events from every admission into one sequence per patient,
  the per-admission reset caused values from different admissions to collide — e.g., a lab drawn 6h into stay 2 sorted
  identically to one drawn 6h into stay 1, even though the two are actually days apart.
@rianfyi
Wire lab z-scoring, run provenance, and a named eval split.
d45e3cc 
Five small changes on top of cdeb3e0, plus the paper launchers.

1. lab_standardizer.py + wiring. Per-feature z-score fit on the training
   split only; missing values stay missing. UnifiedMultimodalEmbeddingModel
   already accepted numeric_standardizers, so this is only the fit + the
   hand-off. --no-lab-standardization runs raw labs as an ablation, so the
   default is a choice you can turn off rather than a commitment.
   Measured on EHRMamba/labs+notes/seed 1: 0.7473 without, 0.8011 with.

2. write_run_config. metrics_history.json records what a run scored but not
   the conditions that produced it, and it records resolved values rather
   than raw flags -- that distinction is what surfaced a per-model optimizer
   override where run_config stored adam_eps: null while the optimizer used
   1e-6. Also records source_sha256 so a table can be shown to come from one
   build.

3. eval_split. The inference loader fell back test-or-val-or-train, so a run
   without a test split reported TRAINING metrics as test with nothing saying
   so. The split is now named, warned about, and recorded.

4. exp_name includes the task. It was {model}_seed{seed}, so labs and
   notes_labs at one seed wrote to the same directory and the second run
   silently destroyed the first. This matters immediately: the plan is 36
   paired cells.

5. Restore emitted_data_version. cdeb3e0 removed it. It is part of vars(task),
   which is what the task-cache uuid5 key is built from, so without it a cache
   built before an emitted-data change is silently reused -- and cdeb3e0
   changes every event timestamp, which is exactly when the bump is needed.

Deliberately not included: the time_origin fix. cdeb3e0 already does it, and
by inspection it is identical to ours (same _hours_since helper, same anchor
on admissions_to_process[0].timestamp). No need to revert it.

scripts/paper: common.sh holds the protocol; will.sh and rian.sh add only the
data roots and the CPU tuning for their machine. rian.sh pins OMP threads and
uses loader workers because those nodes run several cells at once -- unpinned,
four concurrent cells put ~800 threads on 128 cores and epoch time went 191s
to 8600s with the GPUs at 0-1%. will.sh keeps num_workers=4 and no pinning.

Verified on a dev split before and after: output dir goes mlp_seed1 ->
notes_labs_mlp_seed1, run_config.json appears with eval_split=test, and the
fitted mean/std/count buffers land in the checkpoint with --no-lab-
standardization correctly removing them.
@will-pang
Revert "Unify event timestamps in pyhealth/tasks/multimodal_mimic4.py…
6ad4726 
… onto a single per-sample clock."

This reverts commit cdeb3e0.
@will-pang
Revert "Revert "Unify event timestamps in pyhealth/tasks/multimodal_m…
2c93199 
…imic4.py onto a single per-sample clock.""

This reverts commit 6ad4726.
@will-pang
Wire lab z-scoring, run provenance, and a named eval split (#53)
d1c81d4
@will-pang will-pang changed the title Ml4h merge tranche 1 wp 20260830 ML4H Submission Final: Code + Experiment Run Results Sep 4, 2026
rianfyi and others added 4 commits September 4, 2026 16:43
@rianfyi
paper/rian.sh: drop flags the runner does not define, pass extras thr…
c526a51 
…ough

--loader-num-workers and --persistent-workers do not exist on this branch, so
every rian cell died at argparse before doing any work. This runner has no
dataloader-worker control at all: --num-workers feeds the dataset build only,
and thread pinning is what actually keeps concurrent cells off each other.

Also forward "$@" so callers can add flags (--wandb, --observation-window-hours)
without editing the launcher.
@rianfyi
Persist test metrics, log CPU usage, and make run identity window-aware
f4b3707 
Four small fixes found while running the Tranche 1 sweep.

1. Test evaluation was gated on wandb. `if wandb_logger.enabled and
   test_loader is not None` meant a run without --wandb never computed test
   metrics at all -- not merely unlogged, never calculated. Ungated.

2. Test metrics are now written to test_metrics.json. metrics_history.json
   carries validation only and log.txt has no test lines, so the numbers that
   go in a paper previously lived nowhere on disk: only in stdout and W&B,
   recoverable afterwards only by re-scoring predictions_*.csv by hand.

3. Per-epoch CPU accounting alongside the existing VRAM and epoch_time_s:
   train_cpu_seconds and train_cpu_util_pct. Counts dataloader workers, since
   self-only time badly understates a data-loading-bound run. psutil is
   already present via wandb, with a resource fallback.

4. exp_name and the W&B run name now include the observation window. An
   observation-window arm is a different experiment from the full-stay run at
   the same task/model/seed, but both resolved to the same name -- so they
   shared an output directory and collided in W&B. Runs also now set W&B
   group (arm) and job_type (backbone) so a many-cell sweep is navigable.

Also: create_directory used `if not exists: makedirs`, which two processes
importing pyhealth for the first time can both pass, leaving one to die on
FileExistsError. Seen on a shared cluster home with two concurrent jobs.
@will-pang
Persist test metrics, log CPU usage, and make run identity window-awa…
0b83af8 
…re (#54)
@will-pang
Update Will's scripts for Wandb naming
a2e49df
@will-pang

will-pang commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

For some reason my experiments are not persisting in the PR description:

Seed 2

Labs (Will)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Compute VRAM Runtime WandB Link
MLP 0.48 0.89 0.96 0.37 NVIDIA A100-SXM4-40GB 0.19 GB 1 hr 28 min Link
RNN 0.75 0.94 0.98 0.70 NVIDIA A100-SXM4-40GB 0.20 GB 0 hr 40 min Link
Transformer 0.57 0.90 0.96 0.55 NVIDIA A100-SXM4-40GB 1.51 GB 0 hr 45 min Link
Bottleneck Transformer 0.59 0.91 0.96 0.53 NVIDIA A100-SXM4-40GB 0.72 GB 0 hr 46 min Link
EHRMamba 0.65 0.89 0.97 0.65 NVIDIA A100-SXM4-40GB 0.86 GB 0 hr 21 min Link
JambaEHR 0.64 0.87 0.97 0.67 NVIDIA A100-SXM4-40GB 2.97 GB 0 hr 36 min Link

Labs + Notes (Will)

Backbone Val PR-AUC Val ROC-AUC Val Accuracy Val F1 Compute VRAM Runtime WandB Link
MLP 0.58 0.94 0.96 0.49 NVIDIA A100-SXM4-40GB 0.71 GB 1 hr 47 min Link
RNN 0.84 0.98 0.98 0.77 NVIDIA A100-SXM4-40GB 0.72 GB 1 hr 31 min Link
Transformer 0.68 0.94 0.97 0.63 NVIDIA A100-SXM4-40GB 2.53 GB 1 hr 44 min Link
Bottleneck Transformer 0.70 0.96 0.97 0.64 NVIDIA A100-SXM4-40GB 1.36 GB 1 hr 34 min Link
EHRMamba 0.71 0.92 0.97 0.70 NVIDIA A100-SXM4-40GB 1.46 GB 1 hr 6 min Link
JambaEHR 0.76 0.97 0.97 0.72 NVIDIA A100-SXM4-40GB 4.17 GB 1 hr 9 min Link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.