PriLoRA: Prior-Conditioned Low-Rank Adapters for Medical Imaging

Overview

Adapting large foundation models to medical imaging is costly due to limited curated data, dense expert-annotation requirements, and the expense of full fine-tuning. Parameter-efficient fine-tuning (PEFT) methods such as LoRA mitigate these costs, but linear low-rank updates are expressivity-limited. Recent sinusoidal variants such as SineLoRA improve capacity, yet depend on a fixed frequency hyperparameter that is difficult to transfer across modalities and tasks.

We propose Prior-Conditioned Low-Rank Adapters (PriLoRA), an input-conditioned sinusoidal adapter that leverages input-derived prior features to modulate the adapter response per block and per sample, while preserving LoRA's modularity and parameter efficiency. For a frozen weight $W_0$ with LoRA factors $B, A$, PriLoRA forms the low-rank update $BA$ and applies an input-conditioned sinusoidal transform:

$$\Delta W_{\text{Pri}} = \alpha \sin\big(\beta (\omega_0 BA) + \gamma\big) + \delta$$

where a lightweight modulator predicts $(\alpha, \beta, \gamma, \delta)$ from prior features — $\alpha$ controls amplitude, $\beta$ rescales the base frequency $\omega_0$ into an input-conditioned effective frequency, $\gamma$ sets the phase, and $\delta$ adds an offset. This enables adaptive expressivity without increasing the adapter rank or requiring a task-specific frequency search.

We evaluate PriLoRA across multiple medical backbones, modalities, and tasks — generation, classification, segmentation, and CT reconstruction — and observe consistent improvements in robustness and transfer over standard LoRA and representative PEFT baselines.

Method

Comparison of LoRA, SineLoRA, and PriLoRA. (a) LoRA adapts a frozen pretrained weight $W_0$ with a low-rank update $\Delta W = BA$ (rank $r$), producing $y$ via a residual path. (b) SineLoRA increases expressivity by applying an element-wise sinusoidal transform to the low-rank update, e.g. $\sin(\omega_0 BA)$. (c) PriLoRA conditions the sinusoidal update on input-derived prior knowledge using a lightweight modulator that predicts $(\alpha, \beta, \gamma, \delta)$, where $\beta$ rescales a fixed base frequency $\omega_0$ into an input-conditioned effective frequency.

Qualitative Results

Qualitative comparison across tasks. The first two rows show generation results for two radiology-report prompts; the third and fourth rows present segmentation results (ISIC 2016 skin lesions); the fifth row shows Grad-CAM++ attribution for Choroidal Neovascularization (ChN) classification; and the last row reports CT reconstruction results. Across all tasks, PriLoRA produces sharper, more faithful, and better-localized results than the competing PEFT baselines.

Repository Structure

PriLoRA/
├── Classification/   # Classification experiments (Grad-CAM++ attribution, LoRA variants)
├── Generation/       # Text-to-image generation experiments
├── Segmentation/     # Segmentation experiments (SAM-based backbones, LoRA variants)
├── assets/           # Figures used in this README
└── README.md

License

This project is released under the MIT License.