UniPrompt-CL: Sustainable Continual Learning in Medical AI with Unified Prompt Pools

The paper proposes UniPrompt-CL, a medical-oriented continual learning framework that utilizes a minimally expanding unified prompt pool and a novel regularization term to achieve superior stability-plasticity trade-offs and reduced inference costs compared to existing methods.

The Gist

The Big Problem: The "Forgetful Doctor"

Imagine a brilliant medical student who is learning to diagnose diseases.

• The Old Way: In the past, this student studied a textbook full of static pictures. Once they finished the book, they were done. But the real world changes! New types of equipment appear, new hospitals have different lighting, and diseases evolve. If the student tries to learn a new disease today, they often accidentally forget how to diagnose the old ones. This is called "Catastrophic Forgetting."
• The Current Solution (Continual Learning): Scientists tried to teach the student to keep learning new things without forgetting the old. However, most of these methods were designed for natural images (like cats, dogs, and cars).
  • The Mismatch: A photo of a cat can be taken from any angle, in any weather, with any fur color. It's chaotic. But a medical X-ray or eye scan is very different. It's taken under strict rules, with consistent angles. The differences between diseases are often tiny and subtle (like a faint red spot vs. a dark red spot), not huge changes like "cat vs. dog."
  • The Result: When you apply the "cat-and-dog" learning methods to medical data, the student gets confused. They try to learn broad, messy patterns when they should be focusing on tiny, precise details.

The Solution: UniPrompt-CL (The "Smart Note-Taker")

The authors propose a new system called UniPrompt-CL. Think of it as giving the medical student a specialized, organized notebook instead of a messy stack of sticky notes.

Here is how it works, broken down into three simple concepts:

1. The "Unified Notebook" (Unified Prompt Pool)

• The Old Way: Imagine the student had a separate notebook for every single chapter of their training. If they learned about "Eye Disease A" in Chapter 1, they wrote notes in Notebook 1. If they learned "Eye Disease B" in Chapter 2, they wrote in Notebook 2.
  • The Problem: They kept writing the same basic anatomy notes in every notebook. This was a waste of space and made it hard to find the specific details later.
• The UniPrompt Way: The student now has one single, master notebook.
  • Instead of writing the same basic anatomy notes over and over, they write them once.
  • When a new disease comes along, they just add a few new pages to this same notebook.
  • The Benefit: This stops the student from wasting brainpower on redundant information. It forces them to focus only on the new and subtle details that distinguish one disease from another.

2. The "Tiny Add-Ons" (Minimal Prompt Expansion)

• The Old Way: When learning a new disease, some systems would throw out the old notes and start a whole new, massive notebook. This is expensive and slow.
• The UniPrompt Way: When the student encounters a new disease, they only add 20% new pages to their existing notebook.
  • They don't rewrite the whole book. They just add the specific "clues" needed for the new disease.
  • The Benefit: This is incredibly efficient. It's like adding a single sticky note to a recipe card rather than rewriting the whole cookbook. It saves time and computing power.

3. The "Focus Filter" (Regularization)

• The Problem: Sometimes, when you add new notes, you might accidentally scribble over the old ones, or the new notes might be so loud that you can't hear the old ones.
• The UniPrompt Way: The system uses a special "filter" (a mathematical rule called regularization).
  • This filter acts like a noise-canceling headphone. It ensures that when the student learns a new disease, they don't accidentally "turn down the volume" on the diseases they already know.
  • It forces the new notes to be distinct and clear, so the old knowledge stays safe.

Why is this a Game-Changer?

The paper proves that this method is a winner in three ways:

1. It's Smarter: Because it focuses on the tiny details of medical images (like subtle color changes in an eye scan) rather than broad patterns, it diagnoses diseases more accurately. The paper shows it improved accuracy by 1–3% (which is huge in medicine) and up to 10% in some cases.
2. It's Faster and Cheaper: Many other methods require the computer to run the diagnosis twice (like reading a book, then reading it again to check your notes). UniPrompt-CL only needs to run once. It's like reading a book once and understanding it perfectly, rather than reading it twice. This saves a massive amount of computer energy.
3. It's Private: Some older methods require saving old patient photos to "replay" them later. This is a privacy nightmare in hospitals. UniPrompt-CL doesn't need to store old patient data; it just remembers the lessons (the prompts).

The Bottom Line

UniPrompt-CL is like upgrading a medical AI from a student who tries to memorize everything by shouting (loud, messy, and inefficient) to a student who keeps a clean, organized, single notebook.

By realizing that medical images are different from photos of cats and dogs, the authors built a system that learns new diseases without forgetting the old ones, uses less computer power, and keeps patient data private. It's a more sustainable, efficient, and accurate way for AI to help doctors in the real world.

Technical Summary

1. Problem Statement

Context: Modern AI models are typically trained on static datasets, limiting their ability to adapt to evolving real-world environments. Continual Learning (CL) addresses this but faces significant challenges when applied to Medical AI.
Key Challenges:

• Domain Bias & Data Characteristics: Most existing CL methods are designed for natural images (e.g., ImageNet, CIFAR), which exhibit high intra-class variability in viewpoint and shape. In contrast, medical images are acquired under standardized protocols with consistent anatomical structures but contain subtle, critical variations (e.g., device-specific color shifts, fine-grained lesions).
• Inefficiency of Existing PCL: Current Prompt-based Continual Learning (PCL) methods often learn widely dispersed prompts to accommodate broad feature differences. This leads to cross-layer redundancy (repeatedly encoding similar low-level features) and inefficient use of prompt capacity in medical settings.
• Computational Overhead: State-of-the-art PCL methods (e.g., OS-Prompt++) often require multiple Vision Transformer (ViT) inferences (e.g., a second forward pass or extra query generation) to function effectively, creating prohibitive computational costs for medical deployment.
• Catastrophic Forgetting: Medical datasets often suffer from severe class imbalance and long-tailed distributions, exacerbating the forgetting of previous tasks as new domains are introduced.

2. Methodology: UniPrompt-CL

The authors propose UniPrompt-CL, a medical-oriented PCL framework designed to achieve a superior stability-plasticity trade-off with minimal computational cost. The method relies on three core components:

A. Unified Prompt Pool

• Concept: Instead of maintaining independent prompt pools for each layer (which causes redundancy), UniPrompt-CL integrates all prompts into a single, unified pool shared across all layers of the backbone.
• Mechanism: The [CLS] token embedding from each layer serves as a query ($q_l$) to select prompts from this unified pool. The selected prompts are then applied to the subsequent layer transformation.
• Benefit: This design forces the model to learn fine-grained, complementary features rather than redundant representations, maximizing the utility of limited prompt capacity for subtle medical details.

B. Minimal Prompt Expansion (Few Prompt Expansion)

• Strategy: When transitioning to a new domain (e.g., from APTOS to DDR datasets), the model does not retrain all prompts. Instead, it freezes previously learned prompts and expands the pool by adding only a small subset of new prompts (empirically set to 20% of the original pool size).
• Regularization: To ensure these newly added prompts effectively capture new information without being overshadowed by frozen prompts, a novel consistency-enforcing regularization term ($\mathcal{L}_s$) is introduced. This term encourages the new prompts to have distinct representations, preventing them from collapsing into existing clusters.

C. Single-Backbone, Single-Pass Inference

• Architecture: The method utilizes a strong foundation model backbone (DINOv2-base) and performs only one forward pass through the ViT.
• Contrast: Unlike methods like OS-Prompt++ that require dual ViT inferences (one for query generation, one for feature extraction), UniPrompt-CL eliminates the need for a second inference, drastically reducing FLOPs.

3. Key Contributions

1. Unified Prompt Pool Design: Consolidates layer-wise prompt pools into a single integrated pool, reducing cross-layer redundancy and improving fine-grained feature tracking specific to medical data.
2. Efficient Single-Pass Architecture: Achieves state-of-the-art performance using a single ViT backbone and single inference, significantly lowering computational costs compared to multi-pass PCL baselines.
3. Novel Regularization & Expansion Strategy: Introduces a "Few Prompt Expansion" strategy (adding only 20% new prompts) coupled with a specific loss function ($\mathcal{L}_s$) to mitigate catastrophic forgetting and handle class imbalance effectively.
4. Medical-Specific Adaptation: Demonstrates that prompt learning strategies for natural images do not transfer well to medical data; tailored designs (compact, fine-grained distributions) are essential for medical CL.

4. Experimental Results

The method was evaluated on Domain-Incremental Learning (DIL) settings using Diabetic Retinopathy (APTOS $\to$ DDR $\to$ DRD) and Skin Cancer (ISIC $\to$ HAM $\to$ DERM7) datasets.

• Accuracy & F1-Score: UniPrompt-CL outperformed all baselines (including PCL, Rehearsal, Regularization, and Architecture-based methods).
  • On Diabetic Retinopathy, it achieved 84.9% Accuracy and 76.1% F1-score, surpassing the previous best PCL method (OS-Prompt++) by a significant margin.
  • Improvements of 1–3 percentage points in AvgACC were observed across settings.
• Computational Efficiency:
  • GFLOPs: UniPrompt-CL required 44.17 GFLOPs, whereas the strong baseline OS-Prompt++ required 66.42 GFLOPs (due to dual inference).
  • CARA (Cost-Adjusted Retained Accuracy): UniPrompt-CL achieved the highest CARA score (0.116), indicating the best balance between accuracy, stability, and computational efficiency.
• Ablation Studies:
  • Backbone: Using DINOv2 improved performance over standard backbones, but the proposed method still outperformed DINOv2-enhanced baselines, proving the value of the prompt design.
  • Expansion Ratio: A 20% expansion ratio was found to be optimal; 30% led to parameter bloat without performance gains, while 10% was insufficient.
  • Loss Term: The proposed regularization term ($\mathcal{L}_s$) was critical for preventing the new prompts from being ineffective.

5. Significance

• Sustainability in Medical AI: UniPrompt-CL addresses the critical need for sustainable, low-cost continual learning in healthcare, where data privacy often prevents rehearsal-based methods and computational resources are limited.
• Paradigm Shift: The paper challenges the assumption that "wider" prompt distributions are better. It establishes that for medical data, compact, unified, and fine-grained prompt distributions are superior due to the standardized nature of medical imaging.
• Practical Deployment: By reducing inference cost (single pass) while improving accuracy, UniPrompt-CL makes the deployment of adaptive medical AI systems in real-world clinical environments more feasible.

In summary, UniPrompt-CL successfully bridges the gap between theoretical Continual Learning and practical Medical AI by rethinking prompt pool architecture to suit the specific constraints and characteristics of medical imaging data.