One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning

The paper proposes SMoPE, a novel framework for prompt-based continual learning that integrates task-specific and shared prompt strategies via a sparse Mixture of Experts architecture to mitigate knowledge interference, balance expert utilization, and significantly reduce computational costs while achieving state-of-the-art performance.

The Gist

Imagine you are a master chef who has just learned how to make a perfect Spaghetti Bolognese. You've got the recipe down pat. Now, your boss asks you to learn how to make Sushi. Then, Tacos. Then French Pastries.

In the world of Artificial Intelligence, this is called Continual Learning. The big problem is "Catastrophic Forgetting." If you try to learn Sushi by just rewriting your entire brain, you might accidentally forget how to make Spaghetti.

The Old Ways: Too Many Cooks or One Overworked Chef

Before this paper, AI researchers tried two main ways to solve this:

1. The "Specialized Chef" Approach (Task-Specific Prompts):
   Imagine giving your chef a completely new, separate recipe book for every single dish. One book for Spaghetti, one for Sushi, one for Tacos.

   • Pros: The chef never forgets anything because the recipes are kept in separate, safe books.
   • Cons: It's a nightmare to manage. If you have 1,000 dishes, you need 1,000 books. It takes up too much space (memory) and it takes forever to find the right book when a customer orders food (computational cost).

2. The "One Magic Book" Approach (Shared Prompts):
   Imagine giving the chef just one recipe book that they use for everything. They just flip to the right page.

   • Pros: Super efficient! Only one book to carry.
   • Cons: The pages get messy. The instructions for Sushi might accidentally rub out the instructions for Spaghetti. The chef gets confused, and the food tastes bad (knowledge interference).

The New Solution: SMoPE (The "Smart Kitchen" System)

The authors of this paper, SMoPE, came up with a brilliant middle ground. They asked: "What if we had one recipe book, but inside it, we had a team of specialized sous-chefs, and we only woke up the ones we needed for the current order?"

Here is how SMoPE works, using simple metaphors:

1. The "Prompt Experts" (The Sous-Chefs)

Instead of one giant recipe book, imagine the "One Magic Book" is actually a team of 25 specialized sous-chefs (called "Prompt Experts").

• Chef #1 is great at Italian.
• Chef #2 is great at Japanese.
• Chef #3 is great at Mexican.
• ...and so on.

When you order Spaghetti, the system doesn't wake up the whole kitchen. It only wakes up Chef #1 (and maybe a few helpers). The other 20 chefs go back to sleep. This means the "Italian" knowledge stays safe and isn't messed up by the "Sushi" instructions.

2. The "Smart Score" (The Head Chef's Intuition)

How does the system know which chefs to wake up?
In the old "One Magic Book" method, the chef had to read the entire book to figure out what to do. That's slow.
In SMoPE, the Head Chef takes a quick look at the order (the input) and calculates a "Score" for every sous-chef.

• "Spaghetti?" -> Chef #1 gets a score of 99/100. Chef #2 gets 5/100.
• The system then picks the Top 5 chefs with the highest scores and lets them work.
• The Magic Trick: They figured out a way to calculate this score super fast, without reading the whole book first. This saves a ton of time and energy.

3. The "Adaptive Noise" (The Gentle Nudge)

There's a common problem in teams: If Chef #1 is really good at everything, the Head Chef might keep calling only Chef #1, even for Sushi. Chef #1 gets tired, and Chef #2 (the Sushi expert) never gets a chance to practice.

• The Fix: SMoPE uses a clever trick called "Adaptive Noise." Imagine the Head Chef gently nudging the popular Chef #1 and saying, "Hey, take a break, let Chef #2 try this one."
• This ensures that all the chefs get a chance to practice and stay sharp, preventing the system from forgetting how to make Sushi just because it's been making Spaghetti all week.

4. The "Memory Prototypes" (The Sticky Notes)

Sometimes, when learning a new dish, you might accidentally erase the notes for an old one.

• The Fix: SMoPE keeps "Sticky Notes" (called Prototypes) of the most important features of the old dishes. Before the chefs start cooking the new Tacos, they check the Sticky Notes to make sure they haven't forgotten the secret sauce for the Spaghetti. This acts like a safety net against forgetting.

Why is this a Big Deal?

• Efficiency: It uses a single "book" (shared prompt) but acts like it has many. It saves massive amounts of computer memory.
• Speed: It doesn't need to scan the whole database to find the right tool; it picks the right tools instantly.
• Performance: In tests, this method was better than the "Specialized Chef" approach (which used too much memory) and much better than the "One Magic Book" approach (which was too messy).

In a nutshell: SMoPE is like a smart kitchen where you have one central menu, but a team of specialists who only wake up when needed. They help each other, don't step on each other's toes, and remember how to cook everything perfectly, no matter how many new dishes you add to the menu.

Technical Summary

1. Problem Statement

Continual Learning (CL) aims to train neural networks on a sequence of tasks without forgetting previous knowledge. A major challenge is catastrophic forgetting, where learning new tasks degrades performance on old ones.

Prompt-based CL has emerged as a solution, using learnable "prompts" (soft instructions) to guide pre-trained models (like Vision Transformers) on new tasks. However, existing approaches face a fundamental trade-off:

• Task-Specific Prompts: Assign distinct prompt parameters to each task. While effective at preventing interference, they suffer from linear parameter growth (memory inefficiency) and high computational overhead at inference (requiring a full forward pass to determine which task prompt to use).
• Single Shared Prompt: Uses one set of prompts for all tasks (e.g., OVOR). This is highly efficient but suffers from severe knowledge interference, as the single prompt is constantly updated, overwriting task-specific information.

The Core Question: Can we achieve the parameter efficiency of a single shared prompt while maintaining the performance and interference resistance of task-specific prompts?

2. Methodology: SMoPE

The authors propose SMoPE (Sparse Mixture of Prompt Experts), a framework that integrates a Sparse Mixture of Experts (MoE) architecture into the Prefix Tuning paradigm.

Key Architectural Components:

1. Prompt Experts as MoE:

   • Instead of treating the prompt as a monolithic block, SMoPE views the prompt parameters (Prefix Keys $PK$ and Values $PV$) as a collection of $N_p$ distinct "prompt experts."
   • These experts are added to the pre-trained attention heads of the Vision Transformer (ViT).
   • For any given input, only a sparse subset ($K$) of the most relevant experts is activated, rather than updating all prompt parameters simultaneously.

2. Prompt-Attention Score Aggregation:

   • Standard MoE requires computing a score for every expert-token pair, which is computationally expensive in the context of attention mechanisms.
   • SMoPE introduces a unified proxy score ($\tilde{s}_{j'}$) by aggregating individual attention scores across all tokens in a sequence.
   • It computes the average token representation ($\tilde{x}$) and uses it to calculate a single score per prompt expert. This reduces complexity from $O(N \cdot d_k)$ to $O(d_k)$, enabling efficient dynamic selection without a full model forward pass.

3. Adaptive Noise Mechanism:

   • Problem: In sparse MoE, a few "popular" experts often dominate, leading to imbalanced utilization and underutilization of other experts.
   • Solution: SMoPE introduces an adaptive noise penalty ($\epsilon_{j'}$) applied to the scores of frequently activated experts.
   • If an expert's activation frequency exceeds the average, its score is penalized. This forces the model to explore and utilize underused experts for new tasks while protecting the "important" experts that encode critical prior knowledge.

4. Prototype-Based Loss ($L_{proto}$):

   • To prevent the "forgetting" of specialized expert roles, the authors treat the Prefix Keys of frequently activated experts from previous tasks as prototypes (implicit memory).
   • A prototype loss function ensures that the current prompt keys remain close to these historical prototypes, preserving the specialization of experts learned in earlier tasks without needing access to past data.

5. Router Loss ($L_{router}$):

   • Encourages distinct specialization among experts by maximizing the scores of selected experts and suppressing unselected ones, ensuring clear decision boundaries for routing.

3. Key Contributions

1. SMoPE Framework: A novel method that unifies the efficiency of single-prompt methods with the performance of task-specific methods by structuring a shared prompt as a sparse MoE.
2. Efficient Selection Mechanism: A prompt-attention score aggregation technique that enables dynamic, sparse expert selection with minimal computational overhead (no full forward pass required for routing).
3. Stability Mechanisms:
   • Adaptive Noise: Solves the expert imbalance problem in continual learning without overwriting critical knowledge.
   • Prototype Loss: Leverages prefix keys as implicit memory to preserve task-specific specializations.
4. Theoretical Analysis: Proves that the score aggregation technique maintains the same sample complexity for estimating prompt experts as standard dense prefix tuning.

4. Experimental Results

The authors evaluated SMoPE on standard CL benchmarks: ImageNet-R, CIFAR-100, and CUB-200 (using class-incremental learning scenarios).

• Performance: SMoPE consistently outperformed state-of-the-art task-specific prompt methods (e.g., HiDe-Prompt, NoRGa, VQ-Prompt) and significantly surpassed the single-shared-prompt baseline (OVOR).
  • Example (ImageNet-R): SMoPE achieved 79.32% FAA, beating VQ-Prompt (78.71%) and OVOR (75.25%).
• Efficiency:
  • Parameters: SMoPE uses significantly fewer learnable parameters (0.38M) compared to task-specific methods (e.g., Deep L2P++ uses 4.78M).
  • Computation: By avoiding full forward passes for prompt selection, SMoPE reduces inference computational cost by up to 50% compared to task-specific methods.
• Ablation Studies: Confirmed that each component (Score Aggregation, Sparse Selection, Adaptive Noise, Prototype Loss) contributes meaningfully to the final performance.

5. Significance

SMoPE represents a significant advancement in Continual Learning by resolving the long-standing trade-off between efficiency and performance.

• Scalability: It offers a path to lifelong learning where the model does not need to grow its parameter count linearly with the number of tasks.
• Interference Mitigation: By dynamically routing inputs to specialized sub-networks (experts) within a shared parameter space, it effectively isolates knowledge interference.
• Practicality: The reduction in both memory footprint and inference latency makes it highly suitable for deployment in resource-constrained environments or edge devices requiring continual adaptation.

In summary, SMoPE demonstrates that a single, well-structured shared prompt can outperform complex task-specific architectures, provided it is managed through a sparse, adaptive, and memory-aware expert selection mechanism.