📊 statistics

Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics

This paper investigates catastrophic forgetting in Low-Rank Adaptation (LoRA) by employing a mean-field self-attention model to identify phase transitions between forgetting and non-forgetting behaviors based on perturbation norms and Transformer depth, while validating these theoretical findings with experiments on real models.

Original authors: Hugo Koubbi, Louis Hernandez, Matthieu Boussard

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hugo Koubbi, Louis Hernandez, Matthieu Boussard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Over-Eager Student" Problem

Imagine a brilliant student (a Large Language Model) who has already learned everything about the world. They are an expert at writing, coding, and reasoning. Now, you want to teach them a new, specific skill, like writing legal contracts.

You don't want to re-teach them everything from scratch (which is too expensive and slow). Instead, you use a method called LoRA (Low-Rank Adaptation). Think of LoRA as giving the student a small, lightweight notebook of "cheat sheets" to add to their existing knowledge. You only write in this notebook; you don't touch their original brain.

The Problem: Sometimes, when the student fills out these cheat sheets to learn the new skill, they accidentally erase or overwrite their old knowledge. They might forget how to write a poem or solve a math problem. This is called Catastrophic Forgetting.

This paper asks: Why does this happen, and can we predict exactly when the student will start forgetting?

The Tool: A "Particle Cloud" Simulation

Real AI models are too complex to study directly. They are like a massive, chaotic city with billions of moving parts. To understand the problem, the authors built a simplified toy model.

The Analogy: Imagine the AI's internal thoughts as a cloud of dancing particles (like fireflies).
The Movement: As the AI processes a sentence, these fireflies move around, attracted to each other. Eventually, they naturally group together into tight clusters.
The Meaning: These clusters represent the AI's "understanding." If the fireflies cluster in one specific way, the AI understands the concept correctly. If the clusters shift or break apart, the AI has "forgotten" or misunderstood the concept.

The authors treat the layers of the AI (the depth of the network) as time. As the fireflies move through the layers, they are evolving over time.

The Discovery: Two "Tipping Points"

The authors used math (specifically, equations that describe how fluids and particles move) to find two specific "tipping points" where forgetting suddenly happens.

1. The "Volume of Change" Tipping Point (Norm)

Imagine the "cheat sheets" (LoRA) are being filled with random scribbles.

Small Scribbles: If the scribbles are tiny, the fireflies barely notice. They stay in their original clusters. The AI remembers everything.
The Tipping Point: There is a specific size of scribble where the chaos becomes too much.
The Result: Once the scribbles get too big, the fireflies suddenly scatter and form a new cluster shape. The AI has forgotten its old ways and adopted a new, potentially wrong, way of thinking.
The Finding: The paper proves that forgetting isn't a slow slide; it's a sudden switch. If the "size" of the update is small enough, you are safe. If it crosses a specific threshold, you lose your memory.

2. The "Depth" Tipping Point (Layers)

Imagine the fireflies are walking through a long hallway (the AI's layers).

Short Hallway: If the hallway is short, the fireflies don't have enough time to drift away from their original group, even if there is a little wind (perturbation) pushing them.
The Tipping Point: As the hallway gets longer, the tiny pushes add up.
The Result: At a certain depth (a specific layer number), the fireflies suddenly break away from the original group and rush toward a new destination.
The Finding: The deeper the AI goes, the more likely it is to forget. The authors found a "safe window" where the AI is stable, but once you pass a critical depth, the forgetting becomes inevitable.

The Secret Weapon: The "Spectral Gap"

Why do some AI models forget more easily than others? The authors found a hidden factor called the Spectral Gap.

The Analogy: Imagine the fireflies are in a bowl.
- Wide, Shallow Bowl (Small Gap): If the bowl is flat, it's easy for the fireflies to roll out of the center and wander off. The AI is unstable and forgets easily.
- Deep, Narrow Well (Large Gap): If the bowl is a deep, steep well, the fireflies are trapped at the bottom. It takes a massive force to knock them out. The AI is stable and remembers well.
The Finding: The paper shows that the "shape" of the AI's internal math (specifically the difference between its strongest and second-strongest patterns) acts like this bowl. If the "well" is deep enough, the AI resists forgetting, even when you add new LoRA updates.

What They Tested

The authors didn't just do math on paper. They tested these ideas on real AI models (like Qwen and Llama 2).

They added random "noise" (updates) to the models.
They watched how the "fireflies" (representations) moved.
The Match: The real models behaved exactly like their simplified math model predicted. When the updates got too big or the layers got too deep, the models suddenly started forgetting their original tasks.

Summary

This paper explains that Catastrophic Forgetting in AI isn't random. It happens because of two main triggers:

Too much change at once: If the new learning is too aggressive, the AI's internal structure collapses.
Too many layers: If the AI has to process information through too many steps, small errors accumulate until the memory breaks.

The authors provide a mathematical "speed limit" and a "depth limit" to help us understand when an AI is safe to update and when it is about to forget everything it knew. They also suggest that keeping the AI's internal "bowl" deep and steep (a large spectral gap) is the best way to keep it from forgetting.

Technical Summary: Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics

1. Problem Statement

Low-Rank Adaptation (LoRA) is the dominant method for parameter-efficient fine-tuning of Large Language Models (LLMs) due to its favorable compute-performance trade-off. However, despite freezing the pre-trained backbone and only training low-rank updates, LoRA is susceptible to catastrophic forgetting, where the model degrades on previously learned capabilities while acquiring new ones.

The complexity of full-scale LLMs renders them mathematically intractable for analyzing the mechanisms of forgetting. Consequently, there is a need for a theoretical framework that can characterize how low-rank perturbations to attention matrices alter the forward-pass dynamics and lead to representation drift or collapse.

2. Methodology

The authors propose a tractable mean-field self-attention toy model to study forgetting from a theoretical perspective. The methodology relies on the following core components:

Interacting Particle System Viewpoint: Following recent literature on mean-field Transformers, the forward pass is modeled as the evolution of a cloud of interacting tokens (particles). Depth is treated as continuous time, and token embeddings evolve according to a differential equation driven by attention matrices.
LoRA as Low-Rank Perturbation: The pre-trained attention matrices $(Q, K, V)$ are treated as fixed constants. LoRA is modeled as a low-rank perturbation $\Delta M$ (where $M \in \{Q, K, V\}$ ) added to these matrices.
Two Regimes of Perturbation:
1. Deterministic (Tied) Adapters: The same low-rank update is applied across all layers. This serves as a worst-case scenario for forgetting.
2. Random Adapters: Low-rank updates are drawn independently and identically (i.i.d.) from a Gaussian distribution at each layer. This serves as an "average-case" proxy, allowing for homogenization arguments.
Forgetting Proxy: Forgetting is quantified not by task performance directly, but by the drift in emergent representation geometry. Specifically, the authors measure the displacement of token clusters (using Wasserstein distance) or the divergence of the limiting cluster configuration from the pre-trained state.
Mathematical Tools: The analysis utilizes tools from partial differential equations (continuity equations), dynamical systems (stability of equilibria), and spectral theory (eigenvalue gaps).

3. Key Contributions and Theoretical Results

3.1. Finite-Time Stability (Proposition 3.1)

The paper establishes a quantitative stability bound in the Wasserstein distance for mean-field dynamics under perturbations of $(Q, K, V)$ .

Result: The distance between the pre-trained and perturbed token distributions grows at most doubly exponentially with time (depth).
Implication: The model is robust to small perturbations over short time scales, but the bound suggests that deviations can become significant as depth increases.

3.2. Long-Time Stability and Spectral Control (Proposition 3.3)

Focusing on the Post-LayerNorm setting, the authors identify a spectral condition under which the limiting cluster direction remains stable.

Mechanism: Stability depends on the spectral gap ( $\text{gap} = \lambda_1 - \lambda_2$ ) of the Value matrix $V$ .
Condition: If the perturbation satisfies $\text{gap} + a > 2\|b\| + \|E\|_{op}$ (where $a, b, E$ are components of the perturbation relative to the leading eigenvector), the perturbed dynamics converge to a new stable cluster $\tilde{u}_1$ .
Bound: The deviation between the original and perturbed clusters is bounded by $\frac{2\|b\| + \|E\|_{op}}{\text{gap} + a}$ . This suggests that large spectral gaps enforce tighter confinement of representations, while LoRA updates aligned with unstable directions (small gaps) induce larger forgetting.

3.3. Phase Transitions in Representation Drift (Section 4)

The paper characterizes two distinct phase transitions governing the onset of forgetting:

A. Norm-Controlled Transition (Theorem 4.2)

For random adapters, the authors analyze the scaling of the perturbation norm $\eta_L$ relative to the depth $L$ .

Stable Regime ( $\eta_L \ll \sqrt{L}$ ): The cumulative random effect of LoRA updates vanishes in the large-depth limit; dynamics remain trapped near the pre-trained cluster.
Diffusive Regime ( $\eta_L \simeq \sqrt{L}$ ): The accumulated perturbation has a non-trivial diffusive effect. The tokens synchronize around a common random direction, but the cluster deviates from the pre-trained state. The size of this deviation is controlled by the noise magnitude and the spectral gaps.
Unstable Regime ( $\eta_L \gg \sqrt{L}$ ): Perturbations dominate, potentially leading to synchronization around a strongly random moving direction.

B. Depth-Controlled Transition (Theorem 4.6)

The paper investigates bifurcations with respect to network depth.

Stability Window: For small depths, the perturbed dynamics track the base model.
Bifurcation Time ( $T^*$ ): There exists a critical depth $T^*$ after which the perturbed tokens diverge from the original clustering regime. The paper provides an explicit bound for $T^*$ , showing it depends on the spectral gap, the magnitude of the perturbation, and the "maximality gap" between clusters.
Implication: Even small perturbations can be amplified by depth, leading to catastrophic forgetting once the network exceeds a certain depth threshold.

4. Empirical Results

The theoretical predictions are corroborated by experiments on synthetic data and real LLMs (Qwen 3 0.6B, Llama 2, ALBERT).

Norm-Based Bifurcation: Experiments with random Gaussian matrices show a phase transition in the perturbed loss (next-token prediction on a base dataset) as the Frobenius norm of the perturbation increases. This aligns with Theorem 4.2.
Geometric Alignment: Analysis of trained LoRA updates reveals they are not isotropic. Low-rank adapters tend to align with the dominant spectral directions (principal features) of the pre-trained Value matrices. This "locking on" effect interferes with stable directions, driving representation drift.
Depth-Dependent Drift: In trained LoRA models, the "empirical bifurcation depth" (the first layer where representation drift exceeds a threshold) decreases as fine-tuning progresses. This confirms the theoretical prediction that the stability window of the pre-trained representation shrinks as perturbations accumulate.
Rank Independence vs. Dependence: While the toy model with random adapters predicts rank-independence (due to rotational invariance), experiments on trained models show that rank does play a role, likely because optimization is biased toward unstable directions rather than being isotropic.

5. Significance and Claims

The paper claims to provide the first theoretical characterization of catastrophic forgetting in LoRA using mean-field dynamics. Its significance lies in:

Identifying Mechanisms: It isolates the norm of the perturbation and the depth of the network as primary drivers of forgetting, framing them as phase transitions.
Spectral Insight: It highlights the critical role of the spectral gap in the Value matrix for maintaining representation stability, suggesting that forgetting is exacerbated when LoRA updates align with directions of small spectral gaps.
Predictive Power: The derived bounds on "time-to-deviation" and "bifurcation depth" offer a theoretical basis for understanding why deeper models or larger perturbation norms might suffer more severe forgetting.

The authors modestly note that while their toy model captures the geometric drift correlated with performance degradation, the full complexity of optimization dynamics in real LLMs (where updates are not isotropic) introduces nuances (like rank dependence) that extend beyond the simplified random-adapter assumptions. Nevertheless, the work offers a rigorous mathematical lens through which to view the stability of parameter-efficient fine-tuning.