Technical Summary: Sparse Orthogonal Parameters Tuning for Continual Learning (SoTU)

Problem Statement

Continual Learning (CL) aims to enable models to learn from a sequence of tasks without suffering from "catastrophic forgetting," where learning new tasks overwrites previously acquired knowledge. While classical CL strategies (rehearsal, regularization, dynamic expansion) have been developed, they often rely on models trained from scratch. Recently, methods based on Pre-trained Models (PTMs) have emerged, leveraging the strong generalization of models like Vision Transformers (ViT) trained on large datasets (e.g., ImageNet-21K). However, existing PTM-based CL methods typically face challenges such as:

1. Complex Structural Designs: Many approaches require intricate adapter modules, prompt pools, or ensemble strategies that are difficult to implement in real-world applications.
2. Catastrophic Forgetting: Despite freezing pre-trained weights, methods often struggle to maintain performance across diverse downstream tasks, especially those with large domain gaps (e.g., ImageNet-R, ImageNet-A).
3. Classifier Dependency: Many methods rely on elaborate classifier designs or memory-intensive exemplar storage to achieve optimal performance.

The authors investigate whether merging parameters from multiple fine-tuned tasks can mitigate forgetting, specifically exploring the role of sparsity and orthogonality in these parameter updates.

Methodology: SoTU (Sparse Orthogonal Parameters Tuning)

The proposed SoTU method is a "Plug-and-Play" approach that transforms knowledge from multiple domains into a fusion of orthogonal delta parameters. It operates in three distinct stages:

1. Fine-Tuning Stage

For each incoming task $k$ in a stream of tasks $\{D_1, ..., D_T\}$, the method initializes a feature extractor using a pre-trained ViT ($\theta_{pre}$). It then fine-tunes the model to learn task-specific features ($\theta_{ft}^k$) and a classification head ($W^k$). Crucially, the classification head $W^k$ is used only to assist in learning the feature representation and is discarded afterward; it does not participate in the final merging process.

2. Delta Masking Stage

The method calculates the delta parameters ($\Delta \theta^k = \theta_{ft}^k - \theta_{pre}$) representing the knowledge learned for task $k$. To induce sparsity and orthogonality, these deltas are randomly masked using a binary mask matrix $m$ generated from a Bernoulli distribution:
$$\Delta \hat{\theta}^k = \Delta \theta^k \odot m$$
where $m \sim \text{Bernoulli}(p)$. The authors empirically set the masking probability $p = 0.9$ (retaining only 10% of parameters) to ensure high sparsity. This step is critical: the authors hypothesize that high sparsity reduces parameter collisions between tasks, effectively making the delta updates orthogonal.

3. Delta Merging Stage

Instead of maintaining separate models or complex ensembles, SoTU merges the sparse deltas from all seen tasks into a single cohesive model:
$$\theta^k = \theta_{pre} + \sum_{i=1}^{k} \Delta \hat{\theta}^i$$
This merged model serves as the feature extractor for all tasks. For classification, the method employs a Nonlinear Random Projection mechanism (similar to RanPAC) to transform the mixture features into a linearly separable space, followed by a Nearest Class Mean (NCM) classifier. Notably, the method achieves optimal feature representation without requiring elaborate classifier designs specific to the CL process.

Key Contributions

The paper outlines four primary contributions:

1. Discovery of High-Sparsity Merging: The authors found that merging multiple high-sparsity delta parameters can achieve competitive or superior performance on downstream tasks compared to individual fine-tuned models, offering a promising direction for combating catastrophic forgetting.
2. Orthogonality Insight: They demonstrated that the effectiveness of this merging strategy stems from the orthogonality of parameters across multiple tasks. High sparsity minimizes parameter conflicts (collisions), allowing task-specific knowledge to coexist.
3. SoTU Framework: A novel CL method is proposed that achieves optimal feature representation for streaming data without complex classifier designs, functioning as a Plug-and-Play solution.
4. Empirical Validation: Extensive experiments on six benchmarks show that SoTU outperforms existing State-of-the-Art (SoTA) methods, particularly in feature representation and on datasets with large domain gaps.

Experimental Results

The authors evaluated SoTU on six benchmarks: CIFAR100, ImageNet-R, ImageNet-A, CUB200, VTAB, and Cars196. Each dataset was split into 5 or 10 tasks. The method was compared against 12 baselines, including rehearsal methods (iCaRL, DER), prompt-based methods (L2P, DualPrompt), representation-based methods (SimpleCIL, ADAM, RanPAC), and model mixture methods (ESN, HiDe-Prompt).

Key Findings:

• Superior Performance: SoTU achieved the highest Average Accuracy ($\bar{A}$) and Final Accuracy ($A_T$) across all datasets. For instance, on the challenging Cars196 dataset, SoTU improved final accuracy by +3.7% over the previous SoTA (RanPAC).
• Robustness to Domain Gaps: The method showed significant superiority on datasets with large domain gaps from ImageNet (ImageNet-R, ImageNet-A, Cars196), where other PTM-based methods struggled.
• Feature Space Effectiveness: When evaluated purely on feature representation (using NCM without task-specific classifiers), SoTU significantly outperformed RanPAC, demonstrating its ability to learn robust, non-forgetting features.
• Sparsity Analysis: Experiments varying the masking rate revealed that:
  • High sparsity (masking 80–90%) maintains competitive performance.
  • Low sparsity leads to severe parameter collisions and performance degradation.
  • Excessive sparsity (masking >90%) removes necessary task-specific knowledge.
  • The optimal delta-sparsity ratio appears to be approximately $1/T$ (where $T$ is the number of tasks).
• Attention Visualization: Visualizations of attention maps confirmed that merging high-sparsity deltas preserves the attention patterns of individual tasks, whereas merging without masking causes severe distortion due to parameter collisions.

Significance and Claims

The paper claims that SoTU offers a simple yet effective solution to the catastrophic forgetting problem by leveraging the sparse orthogonal characteristics of pre-trained parameters.

• Simplicity: Unlike methods requiring complex prompt pools or dynamic architecture expansion, SoTU relies on a straightforward masking and merging mechanism.
• Plug-and-Play: The method does not necessitate elaborate classifier designs, making it adaptable to various real-world streaming scenarios.
• Theoretical Basis: The authors provide theoretical analysis suggesting that random masking of delta parameters retains the attention map structure, allowing the model to perform well on corresponding tasks even with significant parameter reduction.

The authors conclude that merging sparse orthogonal delta parameters is a powerful mechanism for continual learning, transforming the challenge of sequential learning into a problem of managing parameter orthogonality. They intend to extend this approach to other domains, such as natural language generation, in future work.