On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

This paper empirically demonstrates that catastrophic forgetting in low-rank decomposition-based parameter-efficient fine-tuning is primarily driven by update subspace geometry, revealing that tensor-based and structurally aligned methods outperform traditional shared matrix approaches in sequential learning scenarios.

The Gist

Imagine you have a brilliant, all-knowing librarian (the Pretrained Model) who has read every book in the world. This librarian is perfect at general knowledge but hasn't yet learned about specific topics like "Birds," "Landscapes," or "Sports."

Your goal is to teach this librarian these new topics without making them forget the old ones. This is called Continual Learning.

However, there's a problem: if you try to teach the librarian too much new information, they might start mixing things up or forgetting what they already knew. This is called Catastrophic Forgetting.

To solve this, researchers use a technique called PEFT (Parameter-Efficient Fine-Tuning). Instead of rewriting the librarian's entire brain (which is huge and expensive), they just add a small, sticky note or a specialized index card to help them learn the new topic.

This paper investigates how the shape and design of these "sticky notes" affect whether the librarian forgets old things.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Shared Hallway" vs. The "Private Office"

Imagine the librarian's brain is a giant building.

• Full Fine-Tuning (FF): You give the librarian a whole new wing of the building to work in. They can move walls, paint rooms, and rearrange everything. They learn the new task perfectly and don't forget the old stuff because they have plenty of space. But, this is expensive and takes up too much room.
• Low-Rank Decomposition (The Sticky Notes): You only give them a tiny hallway or a single desk to work on. They have to squeeze all their new learning into this small space.

The paper asks: Does squeezing into a small space make them forget more?

2. The Experiments: Four Different "Note" Designs

The researchers tested four different ways to design these small learning spaces:

• LoRA (The Flexible Folder):

  • The Analogy: Imagine a folder where you can write notes on any page, but you only have a limited number of pages (a low "rank").
  • The Result: If the folder is too small (low rank), the new notes crowd out the old ones, causing forgetting. But if you give them a slightly bigger folder (higher rank), they can remember everything better. It's all about having enough space to write without cramping.

• PiSSA (The "Main Idea" Trap):

  • The Analogy: This method forces the librarian to only write on the "Main Idea" pages of the book. These pages are already filled with the most important, general knowledge.
  • The Result: This is the worst for remembering. Because the librarian is forced to overwrite the most important general knowledge to learn a specific new task (like "Birds"), they lose their general sense of the world. It's like trying to learn a new recipe by erasing the table of contents.

• WeGeFT (The Aligned Blueprint):

  • The Analogy: Instead of writing on random pages, this method gives the librarian a blueprint that matches the existing structure of the library. They add new notes alongside the old ones, following the same architectural lines.
  • The Result: This works very well! Because the new notes fit perfectly into the existing structure, the librarian doesn't have to scramble or overwrite old memories. They stay organized and remember everything.

• LoRETTA (The 3D Puzzle):

  • The Analogy: Instead of a flat piece of paper (a 2D matrix), this method uses a complex, multi-layered 3D puzzle piece (a tensor). Even though the piece is tiny, it has a lot of hidden depth and structure inside it.
  • The Result: This is a magic trick. Even with a tiny amount of space, the librarian can pack a huge amount of information into that 3D shape. They learn the new task perfectly without forgetting the old one, because the "puzzle piece" holds more information than a flat sheet of paper ever could.

3. The Big Takeaway

The paper concludes that how you design the "learning space" matters more than just how small it is.

• Don't just shrink the space: If you just make the space tiny and force the librarian to overwrite their most important general knowledge (like PiSSA), they will forget everything.
• Two winning strategies:
  1. Give them enough room: Let them have a slightly larger, flexible space (like a bigger LoRA folder).
  2. Make the space smart: Either align the new notes perfectly with the old structure (WeGeFT) OR use a clever 3D shape that packs more info into less space (LoRETTA).

In Summary

If you want an AI to learn new things without forgetting the old, don't just squeeze it into a tiny box. Either give it a slightly bigger box, or build a "smart box" that fits perfectly with what it already knows. The shape of the learning tool is just as important as the size of the tool itself.

Technical Summary

Here is a detailed technical summary of the paper "On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning".

1. Problem Statement

While Parameter-Efficient Fine-Tuning (PEFT) methods based on low-rank decomposition (e.g., LoRA) have become the standard for adapting Large Pretrained Models (LPMs), their behavior in sequential learning scenarios remains poorly understood. Specifically, the phenomenon of catastrophic forgetting—where a model loses performance on previously learned tasks after training on new ones—is not well characterized in the context of re-parameterization-based fine-tuning.

The core research questions addressed are:

• How does constraining updates to low-dimensional subspaces affect forgetting compared to full fine-tuning?
• How do different low-rank decomposition designs influence task interference and knowledge retention?
• What design principles lead to robust PEFT in continual learning settings?

2. Methodology

The authors conducted a controlled empirical study using a Vision Transformer (ViT) pretrained on ImageNet-1K as the base model.

• Experimental Protocol:

  • Sequential Learning: The model was fine-tuned on a sequence of four distinct image classification tasks. After each task, the model was evaluated on all previously seen tasks to measure retention.
  • Frozen Backbone: All backbone parameters were frozen; only PEFT module parameters were updated.
  • Datasets: Four diverse tasks were used to introduce progressive domain shifts:
    1. Bird Species Classification (CUB-200-2011)
    2. Land Use Classification (EuroSat)
    3. Natural Scene Recognition (Intel Image Dataset)
    4. Sports Image Classification (Sports Image Dataset)

• Methods Compared:
  The study compared Full Fine-Tuning (FF) against four state-of-the-art PEFT methods, each with distinct update mechanisms:

  1. LoRA: Decomposes weight updates into rank-constrained matrices ($A \times B$).
  2. PiSSA: Updates only the principal singular values and vectors of the pretrained weights.
  3. LoRETTA: Uses Tensor-Train (TT) decomposition to reshape weights into high-dimensional tensors, capturing richer structural information.
  4. WeGeFT: Constrains updates to a subspace aligned with pretrained weights and shares parameters across layers.

• Metrics:

  • Forgetting ($F_j$): Measured as the performance degradation on task $j$ after training on subsequent tasks (based on Chaudhry et al.).
  • Final Accuracy: Average accuracy across all tasks after the final training stage.
  • Parameter Efficiency: Count of trainable parameters.

3. Key Contributions & Findings

The study reveals that forgetting is strongly influenced by the geometry and parameterization of the update subspace, rather than just the number of parameters.

A. The Trade-off Between Flexibility and Constraint

• Full Fine-Tuning (FF): Exhibits the lowest forgetting because updates are unconstrained, allowing different tasks to occupy distinct directions in the parameter space.
• Low-Rank LoRA: Shows a monotonic relationship between rank and forgetting. Small ranks force tasks to share a limited set of directions, causing high interference. As the rank increases (more flexibility), forgetting decreases.

B. The Impact of Subspace Geometry

• PiSSA (Principal Singular Subspace): Suffers from the most severe forgetting, even at comparable ranks to LoRA.
  • Reason: PiSSA restricts updates strictly to the principal singular subspace of the pretrained weights. These directions encode highly general, cross-task representations. Forcing new task-specific updates into this shared, general subspace disrupts the pretrained representation, leading to catastrophic interference.
• WeGeFT (Aligned Subspace): Achieves consistently low forgetting across a wide range of ranks.
  • Reason: Unlike PiSSA, WeGeFT constrains updates to a subspace aligned with the pretrained weights but allows for a weighting mechanism that preserves the original representation structure. It avoids disrupting the pretrained space while adapting to new tasks.

C. The Power of Tensor Decomposition

• LoRETTA: Demonstrates that tensor-based decompositions can outperform matrix-based methods under extreme parameter budgets.
  • Reason: By using Tensor-Train decomposition, LoRETTA captures richer structural information and dependencies within ultra-compact budgets. It achieves minimal forgetting and competitive accuracy even with very few parameters (e.g., 57K vs. 313K for LoRA on ViT-Base).

4. Quantitative Results (ViT-Base Example)

The table below highlights the performance differences on the ViT-Base model:

Method	Trainable Params	Avg. Forgetting	Final Accuracy (%)
Full Fine-Tuning (FF)	85.8M	0.0740	92.70
LoRA (r=8)	313K	0.1491	86.79
PiSSA (r=8)	313K	0.2089 (Worst)	81.25
LoRETTA (R=5)	57K	0.0717 (Best PEFT)	92.65
WeGeFT (r=16)	54K	0.0752	92.30

• Key Insight: LoRETTA and WeGeFT achieved forgetting levels nearly identical to Full Fine-Tuning while using ~1,500x fewer parameters than FF. PiSSA, despite similar parameter counts to LoRA, performed significantly worse due to its restrictive geometry.

5. Significance and Conclusion

This work provides critical design principles for PEFT in continual learning:

1. Update Subspace Design is Critical: The geometry of the update subspace is more important than the sheer number of parameters.
2. Two Regimes for Mitigating Forgetting:
   • Regime 1 (Flexibility): Allow sufficient update flexibility (higher-rank LoRA) or use tensor decompositions (LoRETTA) to capture rich structural information with minimal parameters.
   • Regime 2 (Alignment): Impose structured constraints that maintain alignment with pretrained representations (WeGeFT) rather than disrupting them.
3. Warning on Principal Subspaces: Methods like PiSSA that restrict updates to the principal singular subspace of pretrained weights may inadvertently cause severe catastrophic forgetting by disrupting the general representations encoded in those directions.

Conclusion: To achieve a favorable balance between new-task adaptation and historical knowledge retention, PEFT methods must carefully consider the size, geometry, and structural efficiency of the update subspace. Tensor-based approaches and alignment-preserving strategies offer the most promising paths for efficient, stable continual learning.