Adapt before Continual Learning

Technical Summary: Adapt before Continual Learning (ACL)

1. Problem Statement: The Stability-Plasticity Dilemma in PTM-based CL

Continual Learning (CL) aims to enable models to acquire new knowledge incrementally while retaining old knowledge. Current state-of-the-art approaches leverage Pre-trained Models (PTMs) to provide a strong foundation. However, these methods face a fundamental trade-off:

• Stability-focused methods: Typically freeze the PTM backbone and only train lightweight modules (e.g., prompts or adapters). While this prevents catastrophic forgetting (high stability), it limits the model's ability to adapt to new data distributions that diverge from the pre-training data (low plasticity).
• Plasticity-focused methods: Fine-tune the entire PTM for each new task. While this allows for better adaptation to new concepts, it often leads to the degradation of foundational knowledge (catastrophic forgetting).

The authors identify that the rigid feature space of a frozen PTM becomes a bottleneck when incoming data distributions differ significantly from the pre-training domain (e.g., ImageNet-A or ImageNet-R).

2. Methodology: The ACL Framework

The proposed Adapting PTMs before Continual Learning (ACL) is a plug-and-play framework that introduces a dedicated Adaptation Phase before the core CL learning process for each new task.

The Two-Phase Process:

1. Adaptation Phase: Before learning task $k$, the PTM backbone ($\phi$) and lightweight modules ($\Theta$) are updated using a specialized ACL Loss. This phase aims to realign the feature space to be more discriminative for the current task's data ($D_k$) without destroying existing knowledge.
2. Core Learning Phase: The adapted backbone is then frozen. The classification head and lightweight modules are fine-tuned to learn the specific task. This ensures that the improved feature representations are utilized while maintaining stability.

The ACL Loss Function:
The core innovation is a loss function based on prototypical alignment and contrastive principles:
$$\mathcal{L}_{ACL}(x_i, y_i) = -\log \frac{\exp(\cos(\phi^*(x_i), p_{y_i})/\tau)}{\sum_{j} \exp(\cos(\phi^*(x_i), p_j)/\tau)}$$

• Mechanism: It encourages the adapted embedding ($\phi^*$) to move closer to its corresponding class prototype ($p_{y_i}$) while pushing it away from irrelevant class prototypes ($p_j$).
• Theoretical Justification:
  • Plasticity: Minimizing this loss reduces the upper bound on the current task's classification error.
  • Stability: The loss acts as an implicit regularizer. By anchoring adapted features to the original class prototypes (the most representative points in the original space), it constrains the total feature deviation, preventing the backbone from drifting too far from its foundational knowledge.

3. Key Contributions

• Identification of the Plasticity Gap: The paper demonstrates that the primary limitation in current PTM-based CL is not stability, but insufficient plasticity due to frozen backbones.
• Novel Plug-and-Play Framework: ACL can be integrated into existing CL methods (like L2P, DualPrompt, etc.) without requiring fundamental architectural changes.
• Theoretical Grounding: The authors provide mathematical proofs showing how the ACL loss simultaneously optimizes for task-specific discriminability (plasticity) and feature deviation constraints (stability).
• Versatility: The method is shown to work across different architectures (ViT, CLIP) and various CL paradigms.

4. Experimental Results

The authors evaluated ACL on challenging benchmarks with significant domain gaps (ImageNet-R and ImageNet-A):

• Integration with SOTA Methods: When integrated with existing methods (e.g., L2P, DualPrompt, RanPAC, MOS), ACL consistently improved performance. For example, it provided up to a 7.87% boost in Last Accuracy (LA) for some methods.
• Comparison with Aper: ACL outperformed "Aper" (a method that adapts the PTM only on the first task) across all tested datasets and task configurations.
• Ablation & Visualization:
  • Ablations confirmed that adapting the entire backbone is superior to only adapting lightweight modules.
  • t-SNE visualizations showed that ACL produces more discriminative and clustered feature representations.
  • Grad-CAM visualizations demonstrated that ACL helps the model focus on class-specific object features rather than irrelevant background noise.
• Cross-Model Validation: Success was demonstrated on ViT-B/16-IN21K and CLIP, proving the framework's generalizability.

5. Significance

ACL shifts the focus of PTM-based Continual Learning from merely "protecting" weights to "intelligently realigning" them. By introducing a brief, controlled adaptation step, it breaks the rigid bottleneck of frozen backbones. This provides a scalable and effective way to deploy large-scale pre-trained models in dynamic, real-world environments where data distributions are constantly shifting.