TinyFedPrompt

Federated CoOp prompt tuning of a frozen BioMedCLIP ViT for privacy-preserving pneumonia and COVID-19 detection, running under 1.5 s on a $70 Raspberry Pi 4B.

This repository is the official technical implementation of the paper:

TinyFedPrompt: Federated Prompt Tuning of BioMedCLIP for Privacy-Preserving Pneumonia Detection on Edge Hardware M N Aditya, Kartikeya A, Keerthivasan K, Dr. Jansi R — Department of Electronics & Communication Engineering, SRM Institute of Science and Technology, Kattankulathur, India.

The Problem

Pneumonia is one of the leading infectious causes of death worldwide, with the burden falling disproportionately on regions with limited diagnostic infrastructure.

Radiologist scarcity — Rural primary health centres in India see dozens of suspected pneumonia cases weekly with no on-site specialist and no reliable broadband for remote review.
No cloud connectivity — Most high-performing deep learning models run on data-centre hardware and assume internet access that often does not exist in rural clinical settings.
Patient data privacy — Centralising raw patient images raises serious legal and ethical concerns under India's Digital Personal Data Protection Act and equivalent frameworks globally.

Our Solution

TinyFedPrompt combines three ideas to address all three problems simultaneously:

Federated Learning — Raw images never leave the client device. Only a compact set of pruned prompt tensors (kilobytes per round) is transmitted to the server — no patient images, no full model gradients.
CoOp Prompt Tuning of BioMedCLIP — Rather than fine-tuning BioMedCLIP's full ~100M-parameter ViT backbone (impractical on edge hardware), only a small set of learnable context vectors is optimised and aggregated across clients.
AdaFedPrune — A two-level hardware-aware pruning mechanism that dynamically adjusts per-client sparsity based on real-time RAM availability, scaling from workstation-class metro nodes down to Raspberry Pi 4B rural nodes.

The result: a system that runs on $70 hardware, requires no cloud connection at inference time, and returns a clinically actionable diagnosis within 1.5 seconds of image upload.

Performance vs Baselines

Chest X-Ray — Pneumonia Detection

Model	AUC	Accuracy	F1
TinyFedPrompt (Ours)	0.9509	0.878	0.880
BioMedCLIP (vanilla)	0.7054	0.659	0.648
CLIP	0.4399	0.812	0.805
MedCLIP	0.4399	0.812	0.805

CT Scan — COVID-19 Detection

Model	AUC	Accuracy	F1
TinyFedPrompt (Ours)	0.9419	0.881	0.881
BioMedCLIP (vanilla)	0.8057	0.680	0.646
CLIP	0.5149	0.515	0.470
MedCLIP	0.5149	0.515	0.470

System Capabilities

Capability	TinyFedPrompt	BioMedCLIP	CLIP	MedCLIP
Runs on Raspberry Pi 4B	✅	❌	❌	❌
No raw image transmission	✅	❌	❌	❌
Under 1.5 s edge inference	✅	❌	❌	❌
Hardware-adaptive pruning	✅	❌	❌	❌
Federated training	✅	❌	❌	❌

The bottom line: TinyFedPrompt is the only system in this comparison that achieves AUC > 0.94 while running entirely on edge hardware with no patient data leaving the device.

Tech Stack

Training Stack (Laptop / Dev Machine)

Technology	Version	Role
Python	3.10+	Runtime
PyTorch	≥ 2.0.0	Gradient computation, federated training loop
open_clip_torch	≥ 2.20.0	BioMedCLIP backbone loading
timm	≥ 0.9.0	ViT model utilities
NumPy	≥ 1.24.0	Preprocessing and tensor ops
SciPy	≥ 1.10.0	Statistical utilities
Pillow	≥ 9.0.0	Image loading and resizing
ONNX Runtime	≥ 1.16.0	Post-export parity verification

Edge Inference Stack (Raspberry Pi)

Technology	Version	Role
ONNX Runtime	≥ 1.16.0	Model inference — no PyTorch required
NumPy	≥ 1.24.0	Preprocessing
Pillow	≥ 9.0.0	Image loading

Hardware & OS

Component	Spec	Role
Raspberry Pi 4B	4 GB RAM, ARM64	Edge inference node (Rural tier)
QNX OS	—	Real-time OS for deterministic behaviour on rural nodes
Raspberry Pi OS	64-bit	Alternative OS for town/metro Pi nodes

System Architecture

Federated Training Loop

flowchart TD
    S([Server\nWeighted FedAvg]) -->|Push global prompt θ_p| C1
    S -->|Push global prompt θ_p| C2
    S -->|Push global prompt θ_p| C3

    C1[Rural Client\nPi 4B · 4 GB RAM\n~500 imgs · imbalanced]
    C2[Town Client\nMid-tier hardware\n~1500 imgs · moderate]
    C3[Metro Client\nWorkstation\n~3000 imgs · balanced]

    C1 --> A1[AdaFedPrune\nSparsity ρ ≈ 0.50]
    C2 --> A2[AdaFedPrune\nSparsity ρ ≈ 0.40]
    C3 --> A3[AdaFedPrune\nSparsity ρ ≈ 0.30]

    A1 -->|Sparse update Δθ_p| S
    A2 -->|Sparse update Δθ_p| S
    A3 -->|Sparse update Δθ_p| S

    style S fill:#4F46E5,color:#fff,stroke:none
    style C1 fill:#B45309,color:#fff,stroke:none
    style C2 fill:#0369A1,color:#fff,stroke:none
    style C3 fill:#065F46,color:#fff,stroke:none

Inference Pipeline (Edge)

flowchart LR
    A([Image Upload]) --> B[Preprocessing\nGrayscale → 224×224\nImageNet Normalisation]
    B --> C[Frozen BioMedCLIP\nViT-B/16 Encoder]
    C --> D[CoOp Learnable Prompts\nContext Vectors V₁...V_M]
    D --> E{Softmax\nClassification}
    E --> F([Diagnosis\nConfidence %\nUrgency Flag])

    style A fill:#4F46E5,color:#fff,stroke:none
    style F fill:#059669,color:#fff,stroke:none

System Pipeline Figure

Fig 1. TinyFedPrompt end-to-end pipeline. Client zones handle preprocessing and local prompt tuning; the server aggregates pruned updates via Weighted FedAvg.

AdaFedPrune

Fig 2. AdaFedPrune two-level pruning. Level 1 ranks tokens by gradient magnitude; Level 2 adjusts sparsity ratio according to available RAM.

File Structure

TinyFedPrompt/
│
├── .gitignore                          # Excludes .pth, .onnx, .onnx.data, __pycache__
│
├── xray/                               # X-ray pneumonia detection module
│   ├── XRAY_DATASET_FINAL_PHASE_2_&_3.ipynb   # Federated training notebook (Phase 2 & 3)
│   ├── export_to_onnx.py               # Export .pth → .onnx for Pi deployment
│   ├── detect.py                       # Laptop-side inference (requires PyTorch)
│   ├── detect_pi.py                    # Pi-side inference (ONNX Runtime only)
│   ├── deploy.sh                       # SCP model files to Raspberry Pi
│   ├── requirements.txt                # Laptop deps (PyTorch, open_clip, etc.)
│   ├── requirements_pi.txt             # Pi deps (onnxruntime, numpy, Pillow only)
│   └── samples/                        # Sample chest X-ray images for testing
│       ├── your_xray.jpeg
│       ├── your_xray_1.jpeg
│       └── your_xray_2.jpeg
│
├── ct_scan/                            # CT scan COVID-19 detection module
│   ├── CT_SCAN_DATASET_PHASE_2_&_3.ipynb      # Federated training notebook (Phase 2 & 3)
│   ├── export_ct_to_onnx.py            # Export .pth → .onnx for Pi deployment
│   ├── detect_ct_pi.py                 # Pi-side inference (ONNX Runtime only)
│   ├── deploy_ct.sh                    # SCP model files to Raspberry Pi
│   ├── requirements_pi.txt             # Pi deps (onnxruntime, numpy, Pillow only)
│   └── samples/                        # Sample CT scan images for testing
│       ├── your_ct_1.jpeg
│       ├── your_ct_2.jpeg
│       ├── your_ct_3.jpeg
│       └── your_ct_4.jpeg
│
├── models/                             # Download model weights here from Releases
│   └── .gitkeep
│
└── docs/
    └── assets/                         # Figures extracted from the paper

Model weights are not committed to this repo. Download them from the Releases page and place them in the models/ folder.

Prerequisites

Laptop (Training + Export)

Requirement	Minimum Version	Check Command
Python	3.10	`python3 --version`
PyTorch	2.0.0	`python3 -c "import torch; print(torch.__version__)"`
Git	2.x	`git --version`

Install all laptop dependencies:

pip install -r xray/requirements.txt

Raspberry Pi (Inference Only)

Requirement	Minimum Version	Notes
Raspberry Pi OS	64-bit	Or QNX OS for deterministic real-time behaviour
Python	3.10	Pre-installed on Pi OS
ONNX Runtime	1.16.0	CPU-only build

Install Pi dependencies:

pip3 install -r requirements_pi.txt

No PyTorch, no open_clip, no HuggingFace cache required on the Pi.

Installation & Setup

Path A — Raspberry Pi (Inference Only)

# 1. Clone the repo
git clone https://github.com/MNADITYA05/TinyFedPrompt.git
cd TinyFedPrompt

# 2. Install Pi dependencies (choose the module you need)
pip3 install -r xray/requirements_pi.txt
# or
pip3 install -r ct_scan/requirements_pi.txt

# 3. Download model weights from Releases and place in models/
#    https://github.com/MNADITYA05/TinyFedPrompt/releases/tag/v1.0
#    X-Ray:   federated_biomed_coop.onnx
#    CT Scan: federated_biomed_ct_coop.onnx

Path B — Laptop (Training + Export)

# 1. Clone the repo
git clone https://github.com/MNADITYA05/TinyFedPrompt.git
cd TinyFedPrompt

# 2. Install full training dependencies
pip install -r xray/requirements.txt

# 3. Download the .pth weights from Releases
#    https://github.com/MNADITYA05/TinyFedPrompt/releases/tag/v1.0

# 4. Export to ONNX for Pi deployment
cd xray && python3 export_to_onnx.py --weights ../models/federated_biomed_coop.pth
cd ct_scan && python3 export_ct_to_onnx.py --checkpoint ../models/clinical_focal_hardened_coop.pth

# 5. Deploy to Pi
bash deploy.sh        # X-ray
bash deploy_ct.sh     # CT scan

Usage

X-Ray Pneumonia Detection

cd xray

# Single image
python3 detect_pi.py --model ../models/federated_biomed_coop.onnx samples/your_xray.jpeg

# Multiple images
python3 detect_pi.py --model ../models/federated_biomed_coop.onnx samples/your_xray.jpeg samples/your_xray_1.jpeg samples/your_xray_2.jpeg

# Entire folder with CSV export
python3 detect_pi.py --model ../models/federated_biomed_coop.onnx --folder samples/ --csv results.csv

# Quiet mode (one line per image — for scripting)
python3 detect_pi.py --model ../models/federated_biomed_coop.onnx samples/your_xray.jpeg --quiet

Expected output (single image):

==================================================
       PNEUMONIA DETECTION — RESULT
==================================================
  File      : your_xray.jpeg
  Diagnosis : PNEUMONIA
  Confidence: 99.97%
  Inference : 1394 ms
--------------------------------------------------
  NORMAL    : 0.03%
  PNEUMONIA : 99.97%
==================================================

[URGENT] Pneumonia patterns detected.
          Immediate radiologist review is recommended.

CT Scan COVID-19 Detection

cd ct_scan

# Single image
python3 detect_ct_pi.py --model ../models/federated_biomed_ct_coop.onnx samples/your_ct_1.jpeg

# Entire folder with CSV export
python3 detect_ct_pi.py --model ../models/federated_biomed_ct_coop.onnx --folder samples/ --csv results.csv

Expected output (single image):

[ct] Loading model: federated_biomed_ct_coop.onnx
[ct] Running inference on 1 image(s)...

[1/1] samples/your_ct_1.jpeg
        Prediction:  COVID
        Confidence:  97.61%
        P(COVID):    97.61%
        P(non-COVID):2.39%
        Time:        1332 ms

CLI Reference

`detect_pi.py` (X-Ray)

Flag	Default	Description
`images`	—	One or more image paths (positional)
`--model`	`federated_biomed_coop.onnx`	Path to the ONNX model file
`--folder`	—	Recursively scan a folder for images
`--csv`	—	Write per-image results to a CSV file
`--threads`	CPU count	CPU threads for ONNX inference
`--quiet`	False	Print only verdict per image (for scripting)

`detect_ct_pi.py` (CT Scan)

Flag	Default	Description
`images`	—	One or more image paths (positional)
`--model`	`federated_biomed_ct_coop.onnx`	Path to the ONNX model file
`--folder`	—	Recursively scan a folder for images
`--csv`	—	Write per-image results to a CSV file
`--quiet`	False	Print only verdict per image (for scripting)

Results

X-Ray Pneumonia — Inference Output

Single chest X-ray classified as PNEUMONIA with 99.97% confidence in 1,394 ms with clinical urgency alert.

Batch inference across three X-rays: two PNEUMONIA (99.97%), one NORMAL (92.51%), mean 1,322 ms/image.

Automatic CSV export: file, diagnosis, confidence, NORMAL/PNEUMONIA percentages, and inference time per image.

CT Scan COVID-19 — Inference Output

Single CT scan classified as COVID with 97.61% confidence in 1,332 ms.

Batch CT inference: two COVID (97.61%, 99.88%) and two non-COVID (73.11%, 86.97%), mean 1,323 ms/image.

CSV log of CT results: per-class probabilities and inference time per scan.

ROC Curves

ROC curves for chest X-ray pneumonia classification. BioMedCLIP + CoOp achieves AUC 0.9509 vs 0.7054 for vanilla BioMedCLIP.

ROC curves for COVID-19 CT classification. BioMedCLIP + CoOp achieves AUC 0.9419 vs near-random CLIP and MedCLIP baselines.

Confusion Matrices

Confusion matrices for all compared models on the X-ray test set.

Confusion matrices for all compared models on the CT-scan test set. Near-symmetric error distribution: 29 false negatives, 30 false positives.

Known Limitations & Roadmap

Current Limitations

Rural shard noise — The rural client's small dataset (~500 images) and skewed label distribution produce noisier gradient signals even after Harden Augmentation, leading to slightly higher variance in that node's prompt updates.
Three-tier simulation only — The federated setup currently simulates Rural / Town / Metro tiers; real multi-hospital deployment across physical network boundaries has not been tested.
Single modality per run — The X-ray and CT pipelines are independent; a unified multi-modal inference interface does not yet exist.

Roadmap

Status	Item	Target
✅ Done	Federated CoOp training — X-ray pneumonia	v1.0
✅ Done	Federated CoOp training — CT COVID-19	v1.0
✅ Done	AdaFedPrune hardware-aware pruning	v1.0
✅ Done	ONNX export + Pi inference pipeline	v1.0
📋 Planned	Multimodal acoustic biomarker integration	v2.0
📋 Planned	Adaptive few-shot personalisation per client	v2.0
💡 Exploring	Unified X-ray + CT inference interface	Future

Citation

If you use TinyFedPrompt in your research, please cite:

@article{aditya2026tinyfedprompt,
  title     = {TinyFedPrompt: Federated Prompt Tuning of BioMedCLIP for
               Privacy-Preserving Pneumonia Detection on Edge Hardware},
  author    = {Aditya, M N and Kartikeya, A and Keerthivasan, K and Jansi, R},
  year      = {2026},
  institution = {Department of Electronics and Communication Engineering,
                 SRM Institute of Science and Technology, Kattankulathur, India}
}

License

This project is licensed under the MIT License. See the LICENSE file for full details.

Built by M N Aditya · Kartikeya A · Keerthivasan K · Dr. Jansi R

Department of Electronics & Communication Engineering, SRM Institute of Science and Technology

⭐ Star this repo · 🐛 Report an Issue

Name		Name	Last commit message	Last commit date
ct_scan		ct_scan
docs/assets		docs/assets
models		models
xray		xray
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation