💻 computer science

Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

This paper proposes CoDyRA, a continual learning method that employs rank minimization as an implicit forgetting regularizer within LoRA updates to dynamically balance plasticity and stability, thereby outperforming existing approaches across multiple benchmarks.

Original authors: Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, Dong Gong

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, Dong Gong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Leaky Bucket" of Learning

Imagine you have a very smart, well-read friend (the Pre-trained Model) who knows a lot about the world. You want to teach them new things every day, like how to recognize a specific breed of dog or how to write code in a new language. This is called Continual Learning.

The problem is the "Plasticity-Stability Trade-off":

Plasticity: You want your friend to be flexible enough to learn the new dog breed easily.
Stability: You don't want them to forget how to recognize cats or how to speak French while learning about dogs.

If you try to teach them too hard, they might "overwrite" their old memories (Catastrophic Forgetting). If you try to protect their old memories too strictly, they can't learn anything new.

The Old Ways: Building Add-ons or Using a Library

The paper looks at how people have tried to solve this so far:

The "Add-on" Method (Model Complementation): Instead of teaching your friend new things, you give them a new notebook for every new topic. They keep their original brain untouched, but now they have to carry a backpack full of notebooks and figure out which one to open. This is bulky and requires extra tools.
The "Library" Method (Reference-Constrained): You keep a giant library of old books (past data) in the room. Every time you teach them something new, you force them to look at the old books to make sure they don't forget. This is slow, expensive, and requires storing all that old data.

The New Idea: CoDyRA (The "Minimalist" Approach)

The authors propose a new method called CoDyRA. Instead of adding notebooks or keeping a library, they teach the friend to update their brain directly, but with a very specific rule: "Take only what you need."

They use a technique called LoRA (Low-Rank Adaptation). Think of LoRA like a set of transparent overlays you can place over your friend's brain. These overlays are thin and light.

The Secret Sauce: "Rank Minimization"

The core discovery of the paper is about the size (or "rank") of these overlays.

Big Overlays (High Rank): If you use a thick, heavy overlay, your friend learns the new task very fast (great plasticity). But, because the overlay is so thick, it squishes and distorts the old memories underneath (high forgetting).
Tiny Overlays (Low Rank): If you use a very thin overlay, the old memories stay safe (great stability). But the friend might struggle to learn the new task because the overlay isn't "thick" enough to carry the new information.

The Sweet Spot: The authors found that there is a "Goldilocks" zone—a moderately small size—that allows learning without destroying old memories.

The Innovation: Instead of guessing the right size, CoDyRA uses a special mathematical "shrink ray" (called Rank Minimization).

It starts with a standard overlay.
As the friend learns, the system automatically "prunes" or cuts away the parts of the overlay that aren't strictly necessary.
It forces the update to be as small and simple as possible.

The Analogy: Renovating a House

Imagine your friend's brain is a house filled with furniture (knowledge).

Old Methods: You either build a whole new wing for every new hobby (Add-ons) or you hire a moving crew to constantly check the old furniture every time you add a new chair (Library/Replay).
CoDyRA: You want to add a new chair (new knowledge). Instead of buying a massive, heavy chair that blocks the hallway, you buy a folding chair.
- If the chair is too big, it blocks the path to the kitchen (forgetting old tasks).
- If the chair is too small, it's useless.
- CoDyRA is like a smart carpenter who builds a chair that is exactly the size needed for the new task and nothing more. If the task only needs 3 legs of the chair, the carpenter removes the other 2. This keeps the hallway clear for the old furniture.

Why This Works (The "Why")

The paper proves mathematically that the smaller the update (the lower the rank), the less likely you are to forget.

By minimizing the "rank" (the complexity of the change), the system naturally protects the past.
It doesn't need to remember old data or use extra modules. It just updates the main brain, but only with the bare minimum amount of change required.

The Results

The authors tested this on:

Vision Models (CLIP): Teaching a model to recognize different types of planes, flowers, and pets.
Language Models (LLaMA, Gemma): Teaching a chatbot new skills like coding or math.

The Outcome:

Better Learning: The model learned new tasks well.
Less Forgetting: It forgot much less than previous methods.
No Extra Cost: It didn't need to store old data or carry extra "notebooks." It just updated the main model and merged the changes in, leaving the system clean and efficient.

Summary

CoDyRA is a smarter way to teach AI new things. Instead of forcing the AI to remember everything or adding bulky extra parts, it teaches the AI to make tiny, precise changes. By strictly limiting how "big" a change can be, the AI learns new skills without accidentally deleting its old memories. It's the difference between painting a whole wall red (which hides the old art) and carefully adding a single, small sticker (which adds new info without covering the past).

Technical Summary: Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

Problem Definition

Continual Learning (CL) faces a fundamental tension between plasticity (the ability to acquire new knowledge) and stability (the ability to retain prior knowledge). In the context of Pre-trained Models (PTMs), existing approaches generally fall into two regimes:

Model Complementation: Freezing the PTM and adding external modules (e.g., adapters, prompts) or maintaining orthogonal subspaces. While stable, these methods do not truly improve the backbone's capabilities and often require task-specific routing or inference-time selectors.
Reference-Constrained Learning: Updating the PTM weights directly but constraining updates using stored data (replay buffers) or memory. This incurs high training costs, requires storing past data, and often still suffers from catastrophic forgetting.

The central challenge addressed by this paper is: How can we continually update a pre-trained backbone to absorb new knowledge while preserving existing capabilities, without relying on past data, task identifiers, or external modules?

Methodology: CoDyRA

The authors propose Continual Dynamic Rank-Selective LoRA (CoDyRA), a capacity-controlled approach that treats the effective rank of each parameter update as the primary design knob for balancing plasticity and stability.

Core Insights

Rank as a Control Knob: Through empirical analysis on CLIP, the authors demonstrate that LoRA rank directly governs the plasticity-stability trade-off. Higher ranks facilitate learning new tasks (plasticity) but increase forgetting, while lower ranks mitigate forgetting (stability) but limit adaptation. There exists a "sweet spot" at a moderate rank that varies by module and task.
Theoretical Bound: The paper derives a forgetting bound showing that the forgetting $F(\theta_t)$ on a reference distribution is upper-bounded by terms involving the Frobenius norm of the update $\Delta W$ . Crucially, $\|\Delta W\|_F \leq \sqrt{\rho} \|B\|_2 \|A\|_2$ , where $\rho$ is the effective rank. This establishes that reducing the effective rank $\rho$ tightens the forgetting bound, effectively regularizing forgetting regardless of the specific reference data distribution.

Algorithm Design

CoDyRA integrates rank minimization directly into the LoRA training process:

Architecture: LoRA adapters are inserted into all Attention and MLP modules of both the vision and text encoders.
Dynamic Rank Selection: Instead of a fixed rank $r$ , the update is formulated as a weighted sum of rank-one components:
$\Delta W = \sum_{i=1}^r w_i B_{:,i} A_{i,:}$
where $w_i$ are learnable importance weights.
Regularization Objective: The training objective combines the supervised loss for the current task ( $L_{sup}$ ) with an $\ell_1$ regularization term on the importance weights:
$L_{train} = L_{sup} + \lambda \sum_{m} \|w_m\|_1$
This $\ell_1$ penalty promotes sparsity in the importance weights $w$ . As $w_i$ approaches zero, the corresponding rank-one component is effectively pruned, reducing the effective rank $\rho$ .
Optimization: The non-differentiable $\ell_1$ norm is handled via a proximal gradient method using soft-thresholding. A threshold $\kappa$ is scheduled to increase from zero to $\kappa_{max}$ , allowing dense updates initially before enforcing sparsity.

Key Mechanism

By minimizing the active rank, CoDyRA acts as an implicit forgetting regularizer. It forces the model to retain only the minimal necessary capacity for the current task, thereby confining parameter shifts to a small subspace and preventing interference with the general capabilities encoded in the PTM and knowledge from prior tasks.

Key Contributions

Capacity Control via Rank: The paper identifies the effective rank of parameter updates as a direct, controllable lever for the plasticity-stability trade-off, distinct from model-level capacity (parameter count).
Theoretical Connection: It provides a formal bound linking effective rank to forgetting, proving that rank minimization constrains the magnitude and dimensional reach of updates, thereby protecting prior knowledge without access to past data.
CoDyRA Framework: The proposal of a method that jointly trains LoRA updates with rank minimization, achieving a strong balance between learning new tasks and retaining pre-trained capabilities without replay buffers, task IDs, or external modules.
Empirical Validation: Extensive experiments on CLIP (MTIL, X-TAIL) and LLMs (TRACE) demonstrate that CoDyRA outperforms or competes with state-of-the-art methods in both accuracy and forgetting metrics.

Experimental Results

The method was evaluated on:

CLIP: Multi-domain Task-Incremental Learning (MTIL) and Cross-domain Task-Agnostic Incremental Learning (X-TAIL).
LLMs: TRACE benchmark using LLaMA and Gemma models.

Key Findings:

Superior Plasticity-Stability Balance: CoDyRA achieved the highest "Transfer" accuracy (retention of pre-trained zero-shot capabilities on unseen tasks) and "Last" accuracy (performance on all tasks after the stream) in most settings. For instance, on MTIL, CoDyRA exceeded the zero-shot reference baseline (70.1 vs. 69.4), whereas other methods either fell below it or were capped by it.
Minimal Forgetting: CoDyRA achieved the lowest Backward Transfer (BWT) scores (e.g., 1.87% on CLIP vs. >5% for fixed-rank LoRA and >8% for orthogonal methods), indicating significantly reduced catastrophic forgetting.
Efficiency: Unlike model complementation methods, CoDyRA merges updates into the backbone, incurring no inference overhead and requiring no stored past data or task identifiers. It also uses fewer trainable parameters than many baselines.
Constructive Shift: Analysis of cumulative parameter shifts showed that CoDyRA's updates grow sublinearly over time, whereas fixed-rank LoRA grows near-linearly. Furthermore, the accumulated shift was found to be constructive, slightly improving zero-shot performance on held-out datasets rather than degrading it.

Significance and Claims

The paper claims that rank minimization serves as a universal forgetting regularizer in the continual learning regime. By controlling the effective rank of updates, CoDyRA protects both general pre-trained capabilities and prior-task knowledge under a single, update-based criterion.

The authors emphasize that this approach eliminates the need for:

Replay buffers or stored reference data.
Task identification or routing mechanisms at inference time.
External auxiliary modules that increase inference cost.

The work suggests that "taking only what you need" (minimal rank) is a principled way to update pre-trained models continuously, offering a robust solution to the stability-plasticity dilemma that is both theoretically grounded and empirically effective across vision and language models.