Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning

The paper proposes Residual SODAP, a continual learning framework for domain-incremental scenarios that combines sparse prompt selection, residual aggregation, and data-free distillation to achieve state-of-the-art performance while mitigating catastrophic forgetting without task identifiers or stored data.

The Gist

Imagine you are a chef trying to learn how to cook dishes from different regions of the world, one after another.

• The Problem (Catastrophic Forgetting): Usually, when you learn to cook spicy Thai food, your brain gets so used to the new flavors that you start forgetting how to make your grandmother's Italian pasta. In the world of AI, this is called "catastrophic forgetting." The AI learns the new thing but deletes the old thing.
• The Constraint (No Rehearsal): In the real world (like hospitals), you often can't keep old patient photos in a filing cabinet to practice on later due to privacy laws. You have to learn the new data without being able to look back at the old data. This is called "Rehearsal-Free Learning."

The paper introduces a new AI system called Residual SODAP. Here is how it works, using simple analogies:

1. The "Smart Menu" (Prompt Selection)

Most AI systems try to learn new things by adding new "notes" or "prompts" to their brain.

• The Old Way: Imagine a chef who, when asked to cook, randomly grabs a handful of notes from a giant stack. Sometimes they grab the right note, but often they grab notes for "Spicy Curry" while trying to make "Pasta." This creates confusion (noise).
• Residual SODAP's Way: This system uses a Smart Menu (called α-entmax). Instead of grabbing a random handful, it looks at the order (the new data) and picks only the 2 or 3 most relevant notes. It ignores the rest completely. This keeps the kitchen clean and focused.

2. The "Anchored Recipe Book" (Residual Learning)

• The Old Way: When learning a new cuisine, the chef might rewrite their entire recipe book, accidentally erasing the old recipes in the process.
• Residual SODAP's Way: Imagine the chef keeps their original, trusted recipe book frozen (the "Frozen Prompts"). When they learn a new style (like Thai), they don't rewrite the book. Instead, they write a small sticky note (the "Residual") that says, "Add a little chili to the pasta."
  • The AI keeps the old knowledge safe and intact.
  • It only adds the difference needed for the new task.
  • This ensures that even after learning 100 new things, the original 100 recipes are still perfect.

3. The "Memory of Shapes" (Statistical Knowledge Preservation)

Since the chef can't look at old photos of dishes (privacy rules), how do they remember what a "perfectly cooked steak" looks like?

• The Old Way: They try to remember every single steak they ever cooked.
• Residual SODAP's Way: Instead of remembering every steak, the chef remembers the average shape and color of a perfect steak. They store a "statistical ghost" of the old data.
  • When learning a new dish, the chef generates a "ghost steak" from their memory stats and practices on that.
  • This tricks the brain into thinking it's still practicing the old stuff, keeping the old skills sharp without needing the real photos.

4. The "Drift Detector" (PUDD)

How does the chef know when the customers have suddenly switched from ordering Italian food to ordering Japanese sushi?

• The Old Way: The chef keeps guessing until they fail miserably.
• Residual SODAP's Way: The system has a Drift Detector. It watches how the chef is using their notes.
  • If the chef suddenly starts using a completely different set of notes than usual, the system says, "Whoa, the world has changed! We need new notes!"
  • It automatically expands the menu to add new notes for the new style, ensuring the chef is never caught off guard.

5. The "Auto-Balancer" (Uncertainty Weighting)

The chef has to balance many things: cooking speed, taste, and not burning the food.

• The Old Way: The chef manually decides, "Today I will focus 50% on taste and 50% on speed." This is hard to get right.
• Residual SODAP's Way: The system has a Smart Manager that listens to the "noise" of the kitchen.
  • If the "taste" signal is very noisy (uncertain), the manager turns down the volume on that instruction.
  • If the "speed" signal is clear and strong, the manager turns it up.
  • It automatically finds the perfect balance without the chef needing to guess.

The Result

When the researchers tested this "Smart Chef" (Residual SODAP) on difficult tasks like diagnosing eye diseases (Diabetic Retinopathy) and skin cancer, it didn't just learn the new diseases; it didn't forget the old ones.

• Other AIs: Learned the new disease, but forgot how to diagnose the old one.
• Residual SODAP: Learned the new disease, kept the old one perfect, and did it all without needing a giant storage room of old patient photos.

In short: Residual SODAP is an AI that learns new things by adding small, precise notes to a frozen, perfect foundation, remembers the "shape" of old data without storing it, and automatically knows when to expand its knowledge base. It's the ultimate student that never forgets.

Technical Summary

1. Problem Definition

The paper addresses Domain-Incremental Learning (DIL) under strict rehearsal-free constraints.

• Context: In DIL, a model must learn from a sequence of domains (e.g., different medical imaging datasets) where the data distribution shifts over time.
• Constraints:
  • No Task IDs: The model does not know which domain a test sample belongs to during inference.
  • No Data Storage: Past data cannot be stored or replayed (rehearsal-free) due to privacy regulations (e.g., GDPR) or storage limits.
• Core Challenge: Catastrophic Forgetting. Existing Prompt-based Continual Learning (PCL) methods often fail in DIL settings due to two specific limitations:
  1. Suboptimal Prompt Selection: Hard selection (Top-k) lacks expressiveness, while soft selection (Softmax) introduces noise by allowing irrelevant prompts to influence the output.
  2. Classifier Instability: Existing PCL methods focus heavily on adapting the backbone representation via prompts but neglect the classifier head. The authors observe that even with good representation adaptation, the decision boundaries in the classifier degrade significantly over time, leading to performance drops.

2. Methodology: Residual SODAP

The proposed framework, Residual SODAP, jointly optimizes prompt-based representation adaptation and classifier-level knowledge preservation. It consists of four core components:

A. $\alpha$-Entmax-Based Residual Prompt Selection

To address the limitations of standard prompt selection:

• Input Enhancement: The model uses a Memory Bank (learnable key-value pairs) shared across Transformer layers. The current query is enhanced by retrieving signals from this memory bank and concatenating them with the global context (initial CLS token) and the current layer's query.
• Sparse Selection ($\alpha$-Entmax): Instead of Softmax, the method uses $\alpha$-entmax (with $\alpha=1.5$). This allows for sparse selection, assigning exact zeros to low-scoring prompts, thereby reducing noise from irrelevant prompts while maintaining gradient flow.
• Residual Fusion: The prompt pool is split into a Frozen Set ($\mathcal{F}$) and an Active Set ($\mathcal{A}$).
  • Prompts in $\mathcal{F}$ are frozen to preserve prior knowledge.
  • Prompts in $\mathcal{A}$ are trainable and act as a residual adaptation for the new domain.
  • The final prompt is a weighted sum: $p_{out} = p_{\mathcal{F}} + \lambda_r p_{\mathcal{A}}$.

B. Statistical Knowledge Preservation (Pseudo-Feature Replay)

To mitigate forgetting without storing raw data, the method preserves class-wise feature statistics:

• Knowledge Assets: At the end of each stage, the model saves:
  1. A frozen Teacher Classifier Head.
  2. Class-wise Feature Statistics (Mean $\mu_c$ and Diagonal Variance $\sigma_c^2$) computed via Welford's online algorithm.
• Distillation & Replay: During the next stage's training:
  • Real Feature Distillation: The student head learns from the teacher head using real current-batch features.
  • Pseudo-Feature Replay: The model generates synthetic features by sampling from the stored Gaussian distributions ($\mathcal{N}(\mu_c, \sigma_c^2)$). These pseudo-features are passed through the frozen teacher and trainable student heads to enforce knowledge distillation, effectively replaying past decision boundaries without raw data.

C. Prompt Usage-based Drift Detection (PUDD)

To dynamically manage the prompt pool size:

• Drift Signals: The system monitors two signals:
  1. Selection Entropy: Changes in the uncertainty of prompt selection weights.
  2. Usage Set Shift: The Intersection over Union (IoU) between the current set of active prompts and the set used in recent iterations.
• Adaptive Expansion: A drift score is calculated. If the score exceeds a threshold, the prompt pool is expanded proportionally to the magnitude of the drift. Newly added prompts join the Active Set, while previous active prompts are moved to the Frozen Set.

D. Uncertainty Weighting (UW)

To balance the multiple loss terms (Cross-Entropy, Real Distillation, Pseudo Distillation, Diversity, and Norm regularization) without manual tuning:

• The method adopts homoscedastic uncertainty weighting. It learns a log-variance parameter ($s_i$) for each loss term.
• Losses with higher uncertainty (noisier) are automatically down-weighted, while reliable losses are up-weighted, ensuring stable joint optimization.

3. Key Contributions

1. Dual-Level Preservation: The first framework to explicitly address classifier-level instability in PCL by combining prompt adaptation with statistical knowledge preservation (pseudo-replay).
2. Sparse Prompt Selection: Introduction of $\alpha$-entmax to achieve sparse, noise-free prompt selection, overcoming the limitations of Top-k and Softmax.
3. Drift-Aware Self-Organization: A mechanism (PUDD) that automatically detects domain shifts and expands the prompt pool capacity dynamically, ensuring the model has sufficient resources for new domains without over-provisioning.
4. Rehearsal-Free Efficiency: The method achieves state-of-the-art performance without storing any past data or requiring task identifiers.

4. Experimental Results

The method was evaluated on three benchmarks under DIL settings:

• Diabetic Retinopathy (DR): APTOS $\to$ DDR $\to$ DRD.
• Skin Cancer: ISIC $\to$ HAM $\to$ DERM7.
• General Domain: CORe50 (11-stage stream).

Performance Highlights (AvgACC / AvgF - Lower is better):

• DR: 0.850 / 0.047 (SOTA). Outperforms previous PCL methods (e.g., OS-Prompt++, Dual-Prompt) and Rehearsal-based methods (DER++).
• Skin Cancer: 0.760 / 0.031. Achieves the highest accuracy while maintaining low forgetting, outperforming Dual-Prompt which had low forgetting but poor accuracy.
• CORe50: 0.995 / 0.003. Demonstrates exceptional generalization to non-medical domains.

Ablation Studies:

• Removing the Query Enhancer caused the largest drop in accuracy (-4.2%), highlighting the importance of memory-augmented queries.
• Removing Pseudo Replay or Distillation significantly increased forgetting, validating the necessity of classifier-level preservation.
• Removing Uncertainty Weighting led to suboptimal trade-offs between accuracy and forgetting.

5. Significance

• Practical Applicability: Residual SODAP is highly relevant for real-world medical AI applications where data privacy prevents data storage and domain shifts (e.g., new hospital scanners) are frequent.
• Theoretical Insight: The paper provides empirical evidence that in continual learning, classifier instability is a primary driver of forgetting, often overlooked by methods focusing solely on feature representation.
• Robustness: By integrating sparse selection, statistical replay, and dynamic capacity expansion, the method offers a robust solution that balances the trade-off between retaining old knowledge and adapting to new domains, setting a new benchmark for rehearsal-free continual learning.