A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

1. Problem Statement

Incremental Learning (IL) aims to enable AI models to learn new tasks continuously while retaining knowledge from previous tasks, addressing the issue of catastrophic forgetting. While recent approaches have integrated Vision-Language Models (VLMs) like CLIP (Contrastive Language-Image Pre-training) to leverage their zero-shot capabilities, existing methods face three critical challenges:

1. Training Efficiency: Many methods require fine-tuning large portions of the model or extensive computational resources.
2. Memory Dependency: State-of-the-art methods often rely on large memory banks (replay buffers) to store past data, which is impractical for privacy-sensitive or resource-constrained applications.
3. Backbone Limitations: The performance of IL methods is heavily dependent on the pre-trained backbone and dataset, yet there is a lack of systematic exploration into optimizing these components for IL.

2. Methodology: The SimE Framework

The authors propose SimE (Simple and Efficient), a framework that combines a pre-trained VLM (specifically CLIP) with lightweight Adapters.

Core Architecture

• Frozen Backbone: The pre-trained CLIP image encoder parameters are frozen during the incremental learning process to preserve zero-shot capabilities and prevent catastrophic forgetting.
• Trainable Adapters: Only lightweight adapter modules inserted into the transformer blocks are updated. This drastically reduces the number of trainable parameters.
• No Replay Buffer: SimE does not store past data. Instead, it computes class prototypes (average feature vectors) for new classes and updates a dynamic classifier based on these prototypes and the frozen encoder.

The Multi-Adapter Innovation

To explore better adaptation strategies, the authors introduce Multi-Adapter, which expands upon the standard AdaptFormer architecture. It incorporates three types of sub-modules within transformer blocks:

1. AdaptMLP: Adapts the Multi-Layer Perceptron branch.
2. AdaptAtten: Adapts the Self-Attention branch.
3. AdaptAll: Adapts both branches simultaneously.

Key Theoretical Insight: Nonlinear Correlation

Through extensive experimentation, the authors discovered a nonlinear correlation between the number of adaptive connections and IL performance:

• Between Blocks: Increasing the number of adapters inserted between transformer blocks consistently improves performance.
• Within Blocks: Adding multiple adaptive connections within the same transformer block (e.g., using AdaptMLP + AdaptAtten simultaneously) does not improve performance during small incremental steps (few classes per step). In fact, it can degrade performance due to overfitting to the current task and redundancy. This negative effect is only mitigated during larger incremental steps (more classes per step) where the model requires higher flexibility to capture significant distribution shifts.

Optimization of CLIP

The paper conducts a systematic study to maximize the zero-shot potential of CLIP for IL:

• Pre-training Datasets: Models pre-trained on larger, more diverse datasets (e.g., LAION-2B) significantly outperform those trained on smaller datasets (e.g., WIT-400M).
• Backbone Size: Larger architectures (e.g., ViT-L/14) yield superior feature extraction and IL performance compared to smaller variants (ViT-B/16).

3. Key Contributions

1. SimE Framework: A simple, efficient IL framework that achieves state-of-the-art results without a memory bank, using only a fraction of the trainable parameters compared to full fine-tuning.
2. Multi-Adapter & Nonlinear Discovery: The proposal of Multi-Adapter and the identification of a critical phenomenon: more adapters are not always better. Specifically, dense adaptive connections within transformer blocks are detrimental in small-step IL scenarios but beneficial in large-step scenarios.
3. Systematic CLIP Optimization: A comprehensive analysis recommending the use of LAION-2B pre-trained CLIP models with ViT-L/14 backbones for optimal IL performance.

4. Experimental Results

The authors evaluated SimE on standard benchmarks including CIFAR-100, TinyImageNet, and various other datasets (CUB200, ImageNet-R, etc.).

• Performance Gains:
  • On TinyImageNet, SimE outperforms traditional IL methods by 9.6%.
  • On CIFAR-100, it surpasses other CLIP-based methods by 5.3%.
  • Using the optimized configuration (ViT-L/14 + LAION-2B), SimE achieves an average accuracy of 92.27% on CIFAR-100 (20 steps), significantly beating baselines like Continual-CLIP and ZSCL.
• Efficiency:
  • Parameters: SimE requires only thousands of trainable parameters (compared to millions in other methods).
  • Memory: It eliminates the need for a replay buffer, storing only class prototypes.
  • Compute: It uses approximately one-third of the GPU resources required by competing methods.

5. Significance and Impact

This paper addresses the scalability and sustainability of AI systems in real-world scenarios (e.g., autonomous driving, medical diagnosis) where data arrives continuously and storage is limited.

• Sustainability: By removing the need for large memory banks and reducing trainable parameters, SimE lowers the computational and environmental footprint of continual learning.
• Practicality: The framework demonstrates that high-performance IL can be achieved with minimal architectural changes to powerful pre-trained models, making it easier to deploy in resource-constrained environments.
• Theoretical Guidance: The discovery of the nonlinear relationship between adapter density and task step size provides crucial guidelines for designing efficient parameter-efficient fine-tuning (PEFT) strategies, preventing the common mistake of "more adapters = better performance."