Technical Summary: Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

1. Problem Statement

Continual Learning (CL) aims to enable models to incrementally acquire new knowledge from a stream of tasks without catastrophic forgetting. While Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) are popular for adapting Large Pre-trained Models (PTMs), standard LoRA applies dense updates that inevitably perturb the representation spaces of previous tasks, leading to interference and forgetting.

Recent approaches have adopted Mixture-of-Experts (MoE) frameworks with LoRA adapters (MoE-LoRA) to isolate task interference by freezing old experts and adding new ones. However, the paper identifies that these existing MoE-LoRA methods operate at a coarse granularity. By treating multi-rank LoRA adapters as single, indivisible expert units, they suffer from three critical issues:

1. Interference: Coarse experts encode low-specialty information, forcing irrelevant subspaces to activate for specific inputs.
2. Redundancy: New experts cannot selectively reuse specific "atoms" of knowledge from old experts, leading to inefficient capacity usage and the need to relearn entire blocks.
3. Routing/Retrieval Collapse: As the number of coarse experts grows, external routers struggle to reliably index the expanding pool, leading to routing ambiguity, drift, and the eventual collapse of old experts' performance.

2. Methodology: MoRAM

The authors propose MoRAM (Mixture of Rank-1 Associative Memory), a framework that redefines continual learning as the incremental expansion of a parametric memory system composed of atomic rank-1 units.

Core Conceptual Shift

Grounded in the view that weight matrices act as Linear Associative Memories (Kohonen, 1972), MoRAM treats the weight matrix $W$ not as a monolithic block, but as a collection of $m$ atomic key-value pairs $\{(k_i, v_i)\}$.

• Key-Value Formulation: A rank-$r$ update is decomposed into $r$ atomic rank-1 pairs. Each pair consists of a Key ($A_{:,i}$) and a Value ($B_{:,i}$).
• Content-Addressable Retrieval: Instead of an external router learning where to send data, the input $x$ automatically triggers relevant memory atoms via an intrinsic relevance score: $s_i = A_{:,i}x$. The output is a weighted sum of values based on these scores.

Key Components

1. Fine-Grained Rank-1 Experts:
   Instead of adding a full rank-$r$ adapter per task, MoRAM introduces $r$ new atomic rank-1 pairs per task. These atoms act as fine-grained MoE experts. The model freezes previous atoms and adds new ones incrementally ("little by little").

2. Self-Activation Mechanism:
   MoRAM eliminates the need for an external, trainable router network.

   • Relevance Scoring: The relevance of the $i$-th memory atom to an input $x$ is computed directly via the dot product of the input and the atom's Key vector ($A_{:,i}$).
   • Normalization: Scores are $\ell_2$-normalized across the entire memory ensemble for stability.
   • Benefit: This removes the optimization instability and forgetting associated with training auxiliary router networks.

3. Sparse Routing and Specialization:
   To prevent interference and computational overhead, MoRAM employs a sparse selection strategy:

   • Top-$k$ Masking: Only the $k$ atoms with the highest activation scores are retained for a given input.
   • Temperature Scaling: A temperature hyperparameter $\tau_{MoRAM}$ is applied to the softmax to sharpen the distribution, concentrating gradients on the most relevant "specialist" atoms and accelerating their adaptation.
   • Threshold-Based Pruning: At inference, atoms with activation scores below a threshold $\delta$ are pruned to further reduce noise and computational cost.

4. Incremental Expansion:
   For each new task, the model introduces a set of new rank-1 pairs, freezes all prior atoms, and allows the self-activation mechanism to route across the union of old and new memories. This enables the model to reuse shared semantics (e.g., "blue sky") while specializing new atoms for task-specific concepts (e.g., "airplane").

3. Key Contributions

1. MoRAM Framework: A novel CL framework that treats learning as the expansion of a parametric memory of rank-1 associative units and inference as test-time memory retrieval. This enables fine-grained knowledge reuse with minimal interference.
2. Self-Activated Sparse Routing: By decomposing updates into rank-1 memory atoms, the model naturally forms a fine-grained MoE. The proposed self-activation mechanism eliminates external routers, enabling sparse, content-based retrieval that avoids routing collapse.
3. Empirical Superiority: Extensive evaluations on vision-language (CLIP) and Large Language Model (LLM) benchmarks demonstrate that MoRAM significantly outperforms state-of-the-art methods, achieving better plasticity-stability trade-offs and preserving pre-trained generalization capabilities.

4. Experimental Results

The paper evaluates MoRAM on several benchmarks, including X-TAIL (for CLIP), TRACE (for LLMs), and standard language classification tasks.

• Vision-Language (CLIP): On the X-TAIL benchmark, MoRAM achieves state-of-the-art performance in "Average" and "Last" accuracy metrics, surpassing methods like MoE-Adapter, CoDyRA, and RAIL-Primal. It effectively enhances representation without requiring external domain IDs or feature banks.
• Large Language Models (LLMs): On the TRACE benchmark using LLaMA and Gemma models, MoRAM demonstrates robust performance with significantly lower Backward Transfer (BWT) scores (indicating less forgetting) compared to SeqLoRA, O-LoRA, and TreeLoRA.
• Generalization vs. Forgetting: In code generation fine-tuning (CodeAlpaca), standard dense LoRA updates degrade performance on semantically distant topics (e.g., World Religions). MoRAM, by isolating new knowledge in independent rank-1 atoms, preserves out-of-domain generalization while improving in-domain code performance.
• Efficiency: MoRAM achieves superior performance using roughly one-third of the active parameters required by a standard rank-32 LoRA. It also maintains low GPU memory usage and inference latency comparable to existing PEFT methods.

5. Significance and Claims

The paper claims that high-performing continual learning does not require complex external routers or rigid structural expansions. Instead, it argues for a paradigm shift toward precise, content-addressable retrieval of low-rank atomic memories.

• Mitigation of Catastrophic Forgetting: By treating updates as independent memory atoms rather than entangled blocks, MoRAM prevents the overwriting of unrelated concepts.
• Scalability: The self-activated, sparse mixture mechanism allows the model to scale to many tasks without the routing ambiguity that plagues coarse-grained MoE-LoRA.
• Efficiency: The method reduces compute and memory requirements, making continual adaptation of large models feasible for resource-constrained settings.

The authors conclude that MoRAM offers a "little by little" approach to memory expansion, balancing plasticity and stability effectively while preserving the pre-trained model's generalized knowledge. Future work is suggested to explore adaptive temperature settings based on data distribution and extending applicability to broader PTMs.