Calibration-Aware Lightweight Model Selection for Trustworthy Skin Lesion Classification

This repository contains the research code for the accepted MIWAI submission "Calibration-Aware Lightweight Model Selection for Trustworthy Skin Lesion Classification".

The project compares compact skin-lesion classifiers on HAM10000, ISIC 2019, and PAD-UFES-20 using multi-class performance, calibration, binary malignant/pre-malignant risk, selective referral, corruption robustness, cross-dataset transfer, Grad-CAM auditing, and deployment profiling. The proposed UAX-Distill-Lite is a compact distillation-based student evaluated as a deployment-oriented operating point rather than as an accuracy-dominant replacement for larger EfficientNet-style backbones.

Repository structure

configs/                  YAML configuration files for each dataset and experiment setting
scripts/                  Entry points for full runs, post-hoc audits, TTA, and cross-dataset evaluation
src/skinlesion_uax/       Dataset registry, transforms, metrics, calibration, TTA, and model utilities
tests/                    Lightweight unit tests for registry and metrics behavior
results_summary/          Paper-level CSV summaries only; raw prediction exports and checkpoints are omitted
run_v8_full_sequential.sh Sequential full-run helper for memory-constrained workstations
run_v8_resume_safe.sh     Resume-safe runner for interrupted experiments

Dataset layout

The data/ folder is intentionally not included. Prepare datasets as:

data/
  ham10000/
    HAM10000_images_part_1/
    HAM10000_images_part_2/
    HAM10000_metadata.csv
  ISIC2019/
    ISIC_2019_Training_Input/
    ISIC_2019_Training_GroundTruth.csv
    ISIC_2019_Training_Metadata.csv
  PAD-UFES-20/
    images/imgs_part_1/
    images/imgs_part_2/
    images/imgs_part_3/
    metadata.csv

Environment

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

The experiments used PyTorch/timm models with 224 x 224 inputs, ImageNet normalization, AdamW, learning rate 3e-4, weight decay 1e-4, maximum 80 epochs, patience 8, seeds 42, 52, 62, validation-fitted temperature scaling, and 15-bin ECE.

Smoke test

python scripts/validate_dataset_layout.py --data-root data
python scripts/run_smoke_test.py --dataset ham10000 --data-root data --artifact-root artifacts_smoke

Full experiment sequence

bash run_v8_full_sequential.sh
python scripts/run_v8_posthoc_suite.py --artifact-root artifacts_v8/ham10000
python scripts/run_v8_posthoc_suite.py --artifact-root artifacts_v8/isic2019
python scripts/run_v8_posthoc_suite.py --artifact-root artifacts_v8/pad_ufes20
python scripts/run_cross_dataset_evaluation.py   --source-artifact-root artifacts_v8/ham10000   --source-name ham10000   --target-datasets isic2019,pad_ufes20   --data-root data   --output-dir artifacts_v8/cross_dataset_from_ham10000

Citation

If you use this codebase, please cite the associated manuscript and the dataset papers.

@inproceedings{oladele2026uax_distill_lite,
  title     = {LiteMedRag: Selective Retrieval-Augmented Compact Medical Visual Question Answering,
  author    = {Daniel Ayo Oladele, Lateef Adesola Akinyemi, Mbuyu Sumbwanyambe, Yifan Luo, Ernest Mnkandla and Malusi Sibiya},
  booktitle = {The 19th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI) 2026},
  series    = {Lecture Notes in Artificial Intelligence},
  publisher = {Springer},
  year      = {2026}
}