Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysis

This paper introduces the Mixture of Low-Rank Experts (MoLRE) framework, a parameter-efficient fine-tuning method that significantly enhances the performance of diverse foundation models on comprehensive multi-label head CT diagnosis by employing specialized low-rank adapters and unsupervised soft routing without requiring explicit pathology supervision.

The Gist

Imagine you have a super-smart, all-knowing chef (this is the "Foundation Model"). This chef has tasted every dish in the world and knows the general rules of cooking. However, if you ask this chef to run a busy hospital cafeteria where they need to instantly spot 75 different specific medical problems in head CT scans (like tiny bleeds, old scars, fractures, or tumors), they might get confused. They know everything, but they aren't specialized enough to catch every tiny, specific detail without making mistakes.

Usually, to fix this, we try to "teach" the chef a new trick. But teaching the whole chef new habits is expensive and slow. So, scientists usually use a shortcut called LoRA (Low-Rank Adaptation). Think of LoRA as giving the chef a single, small notepad to write down new rules.

The Problem:
The old method (LoRA) gives the chef just one notepad. It tries to write rules for everything on that single page.

• If the chef is looking for a skull fracture (a hard, sharp problem), they write a rule about "hard edges."
• If they are looking for a tiny bleed (a soft, subtle problem), they try to write a rule about "soft spots."
• The Conflict: These rules fight each other on the same page. The chef gets confused, and the performance isn't perfect. It's like trying to write a recipe for a steak and a recipe for a delicate soufflé on the same sticky note; the instructions get messy.

The Solution: MoLRE (Mixture of Low-Rank Experts)
The authors of this paper invented a smarter system called MoLRE. Instead of one notepad, they give the chef a team of 6 specialized assistants (the "Experts"), each with their own tiny notepad.

Here is how it works, using a simple analogy:

1. The Smart Manager (The Router): When a new patient's scan comes in, a "Smart Manager" looks at the image first.
2. The Decision: The manager doesn't know exactly what's wrong yet, but they can tell if the image looks "trauma-heavy" or "subtle."
3. The Teamwork:
   • If the scan looks like it might have a fracture, the manager hands the task to Assistant A (the Trauma Expert).
   • If the scan looks like it might have a tiny bleed, the manager hands it to Assistant B (the Bleed Expert).
   • If the scan is a mix, the manager might ask Assistant A to do 70% of the work and Assistant B to do 30%.
4. The Result: The chef (the main model) doesn't have to learn everything at once. They just listen to the right assistant for the right job.

Why is this a big deal?

• It's Cheap: This team of assistants adds less than 0.5% more "brain power" (parameters) to the system. It's like adding a few extra pages to a massive encyclopedia, not rewriting the whole book.
• It Needs No Teacher: The manager learns how to pick the right assistant on its own by practicing. It doesn't need a human to say, "This is a fracture, use Assistant A." It figures it out by looking at the patterns.
• It Works Everywhere: The researchers tested this on 6 different types of "chefs" (AI models), ranging from small ones to massive ones.
  • The Surprise: The biggest improvements happened with the "Generalist" chefs (models trained on all kinds of images, not just medical ones). By giving them this specialized team, they became almost as good as the "Medical Specialist" chefs.
  • The Best Combo: The best result came from combining a powerful general model (MedGemma) with this new team system, achieving a 91.7% accuracy in spotting these 75 different head problems.

The Takeaway:
This paper shows that you don't need to build a giant, expensive AI from scratch to diagnose head CT scans. Instead, you can take a smart, general AI and give it a specialized team of mini-experts that know exactly when to step in. It's like turning a general practitioner into a super-specialist team without hiring a whole new hospital.

This method is fast, cheap, and makes AI much better at spotting the tricky, subtle things that doctors need to see.

Technical Summary

1. Problem Statement

Foundation models pre-trained on massive datasets have shown strong zero-shot and few-shot capabilities in medical imaging. However, adapting these models to complex, multi-label diagnostic tasks—specifically comprehensive non-contrast head CT (NCCT) analysis involving dozens of heterogeneous findings—remains challenging.

• Limitations of Standard Fine-Tuning: Current parameter-efficient fine-tuning (PEFT) methods, such as standard Low-Rank Adaptation (LoRA), apply a uniform adaptation across all inputs. This assumes a single set of low-rank weights suffices for all pathology types.
• Knowledge Interference: In comprehensive head CT analysis, features required to detect acute hemorrhage, chronic ischemia, trauma, and subtle structural abnormalities often compete for the same limited adaptation capacity. This "uniformity" can lead to knowledge interference, limiting performance on diverse medical findings.
• Lack of Supervision: Explicit pathology supervision (labeling specific features for routing) is often unavailable or impractical in large-scale clinical datasets.

2. Methodology: Mixture of Low-Rank Experts (MoLRE)

The authors propose MoLRE, a framework that extends LoRA by introducing conditional, input-dependent specialization via unsupervised soft routing.

Core Architecture

MoLRE replaces the single LoRA adapter with $K$ specialized low-rank experts and a router network.

• Adapted Output: For an input feature $x$, the output $h$ is computed as:
  $$h = W_0x + \sum_{i=1}^{K} g_i(x) \cdot \Delta W_i x$$
  Where $W_0$ is the frozen pre-trained weight, and $\Delta W_i = B_i A_i$ represents the $i$-th expert's low-rank adaptation.
• Unsupervised Soft Routing: A router network $g(x)$ (a two-layer MLP with softmax) computes mixing weights for the experts based on the input features.
  • The routing is learned fully unsupervised via the task loss (multi-label classification), meaning the model learns to route specific pathologies to specific experts without explicit pathology labels for the routing mechanism.
  • The softmax ensures $\sum g_i(x) = 1$, allowing varying degrees of expert contribution.

Integration Strategies

The implementation differs based on the model architecture (2D vs. 3D):

• 2D Foundation Models (e.g., DINOv3, MedGemma):
  • Volumetric scans are processed slice-by-slice.
  • The router operates on individual slice features, enabling slice-specific expert selection. This is crucial for head CT where pathologies are spatially localized.
  • Features are aggregated using attention-weighted pooling before final classification.
• 3D Foundation Models (e.g., Pillar0-HeadCT):
  • MoLRE is applied to spatially pooled volumetric features.
  • Routing is based on holistic volume-level representations rather than individual slices.

Training Protocol

• Dataset: Over 70,000 NCCT scans with 75 annotated neurological findings (hemorrhage, infarction, trauma, masses, etc.). Labels were generated automatically using an LLM (GPT-4-mini) from radiology reports, validated by neuroradiologists.
• Optimization: End-to-end training using AdamW.
  • Loss: Multi-label focal loss with class weights to handle rare findings.
  • Parameters: MoLRE adds <0.5% additional trainable parameters.
  • Hyperparameters: $K=6$ experts, rank $r=8$, router hidden dimension $d_h=256$.

3. Key Contributions

1. Novel Framework: Introduction of MoLRE, a conditionally routed, low-rank adaptation framework that enables unsupervised specialization of foundation models.
2. Large-Scale Benchmark: A comprehensive evaluation across six state-of-the-art foundation models (ranging from 7M to 431M parameters) covering 2D, 3D, general-domain, and medical-domain architectures.
3. State-of-the-Art Performance: Achievement of a new benchmark with an average detection AUC of 0.917 using MedGemma + MoLRE.
4. Empirical Insights: Discovery that adaptation benefits are not linear with model size but depend on a complex interaction between pre-training domain, architecture, and model scale.

4. Results and Analysis

The study evaluated performance on 75 neurological findings across 72,756 scans.

• Overall Performance: MoLRE consistently improved all compatible models.
  • Gains: Absolute AUC improvements ranged from +0.2% to +4.6%.
  • Best Result: MedGemma + MoLRE achieved the highest average AUC of 0.917.
• Impact by Model Type:
  • General/Medical 2D Models: Showed the largest gains (e.g., DINOv3-Base: +4.6%, MedGemma: +4.3%). This suggests that models with limited capacity or generative pre-training objectives benefit most from conditional specialization to recover task-specific discrimination.
  • 3D/Domain-Specific Models: Showed more modest gains (e.g., Pillar0-HeadCT: +0.2–1.3%). The authors attribute this to the fact that 3D models collapse spatial heterogeneity into a single volume representation before routing, reducing the benefit of slice-level specialization.
• Stratified Analysis:
  • MoLRE primarily shifts findings from the "moderate" performance regime ($0.8 \le \text{AUC} < 0.9$) to the "high-confidence" regime ($\text{AUC} \ge 0.9$).
  • Specific Improvements: The largest gains were observed for visually subtle, heterogeneous, or underrepresented findings (e.g., early ischemic signs, venous sinus thrombosis, occult bone lesions).
  • Saturation: Findings with very high baseline performance (e.g., major hemorrhages) showed minimal improvement, indicating the method targets weak or fragmented features.

5. Significance and Conclusion

• Efficiency: MoLRE achieves significant performance gains with less than 0.5% additional parameters, making it highly suitable for resource-constrained clinical deployment.
• Mechanism Validation: The results prove that conditional adaptation is superior to uniform adaptation for multi-label medical tasks. It allows a single model to dynamically specialize for different pathologies without explicit supervision.
• Strategic Insight: The paper highlights that simply scaling up model size or using domain-specific pre-training is not a silver bullet. The interaction between pre-training strategy (generative vs. discriminative) and the adaptation method (MoLRE) is critical.
• Clinical Impact: By elevating borderline detections to high-confidence regimes, MoLRE enhances the reliability of foundation models for automated radiology report generation and comprehensive head CT screening.

In summary, MoLRE represents a practical, scalable strategy to unlock the full potential of foundation models in complex clinical imaging tasks by enabling dynamic, pathology-aware specialization.