Technical Summary: Mamba-CL

Problem Definition

Continual Learning (CL) aims to enable AI models to learn a sequence of tasks over time without forgetting previously acquired knowledge, addressing the stability-plasticity dilemma. While State Space Models (SSMs), particularly the Mamba architecture, have achieved notable success in computer vision and sequence modeling, applying them to CL presents unique challenges. Existing CL methods, such as orthogonal subspace projection, have been primarily developed for Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Directly applying these methods to Mamba is difficult due to its higher-order recurrent state-space structure and its non-linear discretization mechanism. Specifically, forgetting in selective SSMs can be understood as changes in the state-update and output-generation processes activated by previous-task inputs. The core problem is how to fine-tune a pre-trained Mamba foundation model for new tasks while mathematically guaranteeing that the outputs for previous tasks remain consistent.

Methodology

The authors propose Mamba-CL, a framework that optimizes selective SSMs in the null space to mitigate catastrophic forgetting. The method focuses on fine-tuning the core SSM blocks within a pre-trained Mamba backbone (specifically De-focus Mamba) while keeping other components frozen.

Theoretical Derivation

The core of Mamba-CL is the derivation of four sufficient consistency conditions for the parameters of the SSM block. The goal is to ensure that the output of the SSM for previous task inputs ($X_t$) remains unchanged after learning a new task. By analyzing the forward process of the SSM, which involves an input-invariant parameter $A$ and input-conditioned parameters $B$, $C$, and $\delta$ (discretization step), the authors decompose the consistency requirement into constraints on the parameter updates ($\Delta$):

1. Discretization Step ($\delta$) and Input Projection ($W_\delta$): The update $\Delta W_\delta$ must be orthogonal to the input feature space $X_t$ ($X_t \Delta W_\delta = 0$).
2. State Transition ($A$): The update $\Delta A$ must be orthogonal to the discretization step parameter $\delta_t$ ($\delta_t \Delta A = 0$).
3. Input Injection ($W_B$): The update $\Delta W_B$ must be orthogonal to the product of the discretization step and input features ($\delta_t X_t \Delta W_B = 0$).
4. Output Readout ($W_C$): The update $\Delta W_C$ must be orthogonal to the input feature space $X_t$ ($X_t \Delta W_C = 0$).

These conditions ensure that the state-update and output-generation processes activated by old-task inputs are preserved.

Optimization Strategy

To implement these constraints efficiently, Mamba-CL employs null-space projection:

• Projection Matrices: The authors compute uncentered covariance matrices for the feature spaces ($X_t$, $\delta_t$, and $\delta_t X_t$) from previous tasks. Using Singular Value Decomposition (SVD), they derive projection matrices ($H_1, H_2, H_3$) that project the gradients of the new task onto the null spaces of these feature subspaces.
• Balance Factor: A hyper-parameter $\eta \in [0, 1]$ is introduced to relax the orthogonality constraints. This allows for a trade-off between stability (retaining old knowledge) and plasticity (learning new tasks). When $\eta=1$, strict null-space projection is applied; lower values allow for more plasticity.
• Implementation: During training, gradients for the SSM parameters ($A, W_B, W_C, W_\delta$) and the subsequent linear projection layer are projected using these matrices before the parameter update step.

Key Contributions

1. Output-Preservation Perspective: The paper formulates the continual adaptation of selective SSMs from an output-preservation perspective, linking the selective scan mechanism with orthogonal subspace constraints.
2. Theoretical Derivation: The authors theoretically derive four sufficient consistency conditions for the SSM block within the Mamba model, addressing the complexities of its recurrent structure and non-linear discretization.
3. Null-Space Approximation: Based on these conditions, a null-space-based approximation is introduced to efficiently implement gradient orthogonal projections, enabling the continual adaptation of large-scale Mamba foundation models without storing previous task samples (exemplar-free).
4. Empirical Validation: Extensive experiments demonstrate the effectiveness of the approach across multiple benchmarks.

Experimental Results

The authors evaluated Mamba-CL on four class-incremental benchmarks:

• Datasets: 10-split and 20-split ImageNet-R, 10-split CIFAR-100, and 10-split DomainNet.
• Baselines: The method was compared against state-of-the-art CL methods (e.g., L2P, DualPrompt, CODA-Prompt, EvoPrompt, EASE) and a Mamba baseline without orthogonal projection (Mamba-Seq).
• Performance:
  • Mamba-CL achieved superior performance compared to existing methods. On the 10-split ImageNet-R, it reached 81.67% accuracy with 4.07% forgetting, outperforming the next best method (CL-LoRA at 78.48% accuracy) by a significant margin.
  • It showed substantial improvements over the Mamba-Seq baseline, increasing accuracy by 18%–45% and reducing forgetting by 24%–61% across benchmarks.
  • The method demonstrated robustness in long-sequence CL scenarios (50 and 100 tasks), maintaining a lead over other methods.
  • Ablation studies confirmed that the projection of the discretization step ($\delta$) is the most critical component for anti-forgetting.
  • The approach was validated on different Mamba variants (MambaVision, Vim, VMamba) and even in a "training from scratch" scenario, confirming its generalizability.

Significance and Claims

The paper claims that Mamba-CL provides an effective way to continually adapt the core selective SSM modules of large foundation models while mitigating interference with previously acquired knowledge. Unlike previous approaches that require episodic meta-training, task-specific backbone expansion, or the retention of exemplars, Mamba-CL operates in an exemplar-free setting with no additional operations during inference.

The authors emphasize that their method is not merely a direct application of existing projection techniques but a necessary adaptation to the specific mathematical structure of SSMs. By deriving specific consistency conditions for the time-invariant parameters of the Mamba model, they bridge the gap between the theoretical requirements of continual learning and the practical implementation of state-space models. The results suggest that explicitly protecting the input-dependent discretization, state transition, input injection, and output readout processes via null-space projection is crucial for achieving strong continual learning performance in Mamba-based architectures.