Sparse Spectral LoRA: Routed Experts for Medical VLMs

1. Problem Statement

Large Vision-Language Models (VLMs) excel in general domains but struggle in medical imaging due to three primary challenges:

• Cross-Dataset Interference: Medical datasets are heterogeneous (different modalities, anatomies, and supervision signals). Naively mixing these data or fine-tuning on one dataset often degrades performance on others.
• Catastrophic Forgetting: In realistic clinical workflows, tasks and data arrive sequentially. Standard fine-tuning or even standard Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA suffer from severe performance drops on previously learned tasks when new data is introduced.
• Cost and Efficiency: Full fine-tuning of large medical VLMs is computationally prohibitive. Existing PEFT methods (e.g., standard LoRA) often fail to match the performance of full fine-tuning while maintaining robustness across diverse medical tasks.

The authors identify that current LoRA implementations are highly sensitive to the data regime (how data is mixed) and lack the structural capacity to specialize for distinct medical domains without interfering with one another.

2. Methodology: MedQwen

The authors propose MedQwen, a parameter-efficient medical VLM that integrates a Spectrally Routed Mixture-of-Experts (MoE) with a theoretically grounded scaling mechanism. The core components are:

A. Sparse Spectral LoRA Initialization

Instead of initializing LoRA adapters with random weights or zero, MedQwen leverages the Singular Value Decomposition (SVD) of the pre-trained weight matrix $W^{(0)} = USV^\top$.

• Non-Overlapping Segments: The spectral components (singular values) are partitioned into $N$ non-overlapping segments.
• Expert Assignment: Each expert in the MoE is initialized from a distinct SVD segment. This creates "spectral priors," where each expert specializes in a specific subspace of the pre-trained knowledge.
• Adaptive Routing: A lightweight router dynamically selects the top-$k$ experts for a given input. This allows the model to activate only the experts whose spectral priors are most relevant to the specific medical task or modality, reducing interference.

B. Theoretical Optimization Alignment

To ensure stable training and prevent the model from diverging from the pre-trained weights, the authors derive a theoretical framework to align the dynamics of the LoRA-MoE with a full-rank, fully fine-tuned MoE.

• Residual Compensation: A residual term ($W_{res}$) is introduced to compensate for the initial mismatch caused by activating only a subset of experts. This ensures the effective weight at initialization equals the pre-trained weight.
• Theoretical Scaling Factor: The authors derive an optimal scaling factor $s^*$ that aligns the gradient geometry of the low-rank updates with full fine-tuning.
  $$s^* = \sqrt{\frac{3n\eta}{r}}$$
  Where $n$ is the input dimension, $r$ is the rank, and $\eta$ is the learning rate ratio. This scaling is crucial for maintaining gradient magnitude and convergence speed, especially when using low ranks.

C. Architecture

MedQwen is built upon the Qwen2.5-VL 7B base model. It replaces standard dense layers with sparse MoE layers where the experts are the spectrally initialized LoRA adapters. The architecture remains unchanged from the base model, ensuring compatibility and efficiency.

3. Key Contributions

1. Sparse Spectral LoRA: A novel initialization strategy that partitions pre-trained weights into non-overlapping spectral segments, assigning each to a specific expert. This reduces cross-dataset interference by allowing experts to specialize in distinct spectral subspaces.
2. Optimization Alignment via Scaling: A theoretical derivation of a scaling factor and a residual compensation scheme that aligns low-rank MoE dynamics with full-rank fine-tuning, ensuring stable convergence without changing the base optimizer or architecture.
3. Unified Medical VLM: A single model that achieves State-of-the-Art (SOTA) performance across discriminative (classification, VQA) and generative (report generation) medical tasks while being highly parameter-efficient.
4. Robustness to Sequential Learning: The method significantly mitigates catastrophic forgetting, a critical requirement for continuous clinical learning.

4. Experimental Results

The authors evaluated MedQwen on 23 medical datasets covering VQA, report generation, radiology classification, and hallucination mitigation.

• Performance vs. Full Fine-Tuning:
  • MedQwen approaches the performance of full fine-tuning on zero-shot classification tasks while using 339$\times$ fewer trainable parameters (2.24% vs. 100% parameters).
  • On medical VQA benchmarks (VQA-RAD, SLAKE, PathVQA), MedQwen outperforms strong baselines like LLaVA-Med and Med-Flamingo by significant margins (e.g., +23.2% over Med-LLaVA).
• Catastrophic Forgetting:
  • In a sequential learning scenario (training on Harvard-FairVLMed then PathVQA), standard LoRA lost >50% of its original accuracy, and MoE-LoRA lost >20%.
  • MedQwen lost only ~5%, demonstrating superior retention of prior knowledge.
• Efficiency:
  • MedQwen achieves SOTA results with a parameter count of roughly 0.81% of the full model (for the LoRA-MoE components).
  • It outperforms other PEFT methods (PiSSA, MiLoRA, HydraLoRA) across various ranks and expert counts.
• Hallucination Mitigation:
  • MedQwen significantly reduces hallucination rates in both closed-ended and open-ended medical report generation compared to general and medical-specific VLMs.

5. Significance

MedQwen addresses the critical gap between the high performance of full fine-tuning and the efficiency of PEFT in the medical domain. By structuring the adaptation process around the spectral properties of pre-trained weights and enforcing theoretical gradient alignment, it solves the issues of data interference and catastrophic forgetting that plague current medical AI systems.

The work demonstrates that:

1. Spectral Structure Matters: The singular value spectrum of pre-trained weights contains task-dependent information that can be exploited for specialization.
2. Scaling is Critical: Proper theoretical scaling is necessary to make low-rank MoE behave like full-rank MoE, ensuring stability and convergence.
3. Clinical Viability: The ability to learn new medical tasks sequentially without forgetting old ones makes MedQwen a viable candidate for real-world, continuous clinical deployment where data arrives over time.

In summary, MedQwen provides a robust, efficient, and theoretically grounded framework for adapting large vision-language models to the complex, heterogeneous, and sequential nature of medical imaging tasks.