Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity

This paper investigates the loss of plasticity in deep learning by using dynamical systems theory to identify how activation saturation and representational redundancy create stable manifolds that trap learning trajectories, revealing a fundamental tension where properties beneficial for static generalization hinder adaptability in non-stationary environments.

The Gist

Imagine you are training a dog to perform tricks. At first, the dog is eager, learns quickly, and can pick up new tricks like "sit," "shake," and "roll over" with ease. This is plasticity—the brain's ability to change and learn.

But what happens if you keep training this dog for years, day after day, without ever letting it rest or change its routine? Eventually, the dog might stop learning new tricks entirely. It might get stuck doing the same few moves over and over, even if you try to teach it something completely new. It hasn't forgotten the old tricks (that's "catastrophic forgetting"); it just lost the ability to learn new ones.

In the world of Artificial Intelligence (AI), this phenomenon is called Loss of Plasticity (LoP). It's a major problem for AI agents that need to learn continuously in a changing world, like a self-driving car encountering new weather patterns or a robot learning new tasks on a factory floor.

This paper, published at ICLR 2026, acts like a detective story. Instead of just saying, "The AI is broken," the authors use math to explain why the AI gets stuck and how to get it unstuck.

Here is the breakdown in simple terms:

1. The Trap: The "Muddy Swamp" of Learning

Imagine the AI's brain (its parameters) is a vast, high-dimensional landscape. When the AI learns, it's like a hiker walking down a hill to find the lowest point (the best solution).

The authors discovered that sometimes, the hiker doesn't just stop at the bottom; they get stuck in a muddy swamp. Once the hiker steps into this swamp, the mud is so sticky that no matter which way they try to walk, they just slide back into the same spot.

• The Science: They call this a "LoP Manifold." It's a specific, low-dimensional trap in the AI's brain where the learning process (gradient descent) gets stuck. The AI thinks it's done learning, but it's actually just trapped.

2. How Did the AI Get Trapped? (Two Main Culprits)

The paper identifies two specific ways the AI builds this swamp for itself:

• The "Frozen" Units (The Sleeping Neurons):
  Imagine a classroom where some students have fallen asleep. If a student is "asleep" (their activation is saturated), they stop reacting to the teacher. In an AI, if a neuron gets too "excited" or too "negative," it stops firing. Once it stops, the math says it will never wake up again because the signal to change it is zero. The AI effectively loses a chunk of its brain.

  • Analogy: It's like a light switch that got stuck in the "off" position. No matter how much you push the switch, the light stays off.

• The "Cloned" Units (The Echo Chamber):
  Imagine a choir where everyone starts singing the exact same note at the exact same volume. They aren't adding any new harmony; they are just repeating the same sound. In AI, different neurons can start doing the exact same thing. They become "clones."

  • Analogy: It's like having 100 employees in a company, but they all do the exact same job. You think you have a big team, but you actually only have one worker doing the work 100 times. The AI has lost its diversity.

3. The Big Irony: Success is the Cause of Failure

Here is the most surprising part of the paper. The things that make AI good at a single task are actually the things that trap it for the future.

• The "Compression" Trap: To be smart and efficient, AI tries to compress information. It tries to find the simplest, most elegant way to solve a problem (like folding a map). This is great for generalization (doing well on test questions).
• The Cost: But in doing this, it forces the AI into that "muddy swamp" of low-rank structures (the clones and frozen units). The very mechanism that makes the AI a genius at today's task builds the wall that prevents it from learning tomorrow's task.

4. How to Break the Trap (The Escape Routes)

If the AI is stuck in the swamp, how do we get it out? The paper suggests two main strategies:

• Prevention: The "Normalization" Shield:
  Think of this as giving the AI a thermostat. Normalization layers (like Batch Norm) keep the AI's internal signals from getting too hot or too cold. This prevents the neurons from falling asleep (freezing) or getting stuck in a loop. It keeps the "light switches" working properly.

  • Result: The AI stays flexible and doesn't build the swamp in the first place.

• Rescue: The "Noise" Kick:
  If the AI is already stuck, you need to shake it up. The authors found that adding a little bit of random noise (like a gentle earthquake) or using Dropout (randomly turning off neurons during training) can break the symmetry.

  • Analogy: If you are stuck in mud, sometimes you have to wiggle violently or get a friend to push you from a weird angle to break the suction. The "noise" breaks the perfect clone pattern, waking up the sleeping neurons and forcing the AI to try new paths.

Summary

This paper tells us that AI isn't just "forgetting" things; it's getting physically trapped in a mathematical dead-end caused by its own desire to be efficient.

• The Problem: AI gets stuck in a "low-rank" trap where neurons freeze or clone themselves.
• The Cause: The drive to be efficient and generalize creates this trap.
• The Solution: We need to use "thermostats" (normalization) to prevent the trap and "shakers" (noise) to escape it if we do.

By understanding these mechanics, we can build AI agents that don't just learn once, but can truly learn forever, adapting to a world that never stops changing.

Technical Summary

1. Problem Statement

Deep learning models excel in stationary settings but suffer from Loss of Plasticity (LoP) in non-stationary environments (e.g., continual learning, lifelong learning). While prior literature identifies symptoms of LoP—such as rank collapse of representations, "dead" (saturated) units, and exploding weight norms—it lacks a mechanistic explanation for why gradient descent fails to recover from these states.

The core question addressed is: Why does gradient descent get trapped in these non-plastic states, and why can't the model simply "unlearn" the bad configuration to adapt to new data distributions? The authors argue that LoP is not merely a statistical degradation but a topological entrapment within the parameter space.

2. Methodology and Theoretical Framework

The authors ground their investigation in dynamical systems theory, analyzing the optimization landscape of neural networks.

A. Formal Definition of LoP Manifolds

The paper defines LoP not as a point, but as an invariant sub-manifold ($M \subset \Theta$) in the parameter space.

• Definition: A manifold $M$ induces LoP if the gradient of the loss function is strictly tangent to the manifold at every point on it ($\nabla_\theta L(\theta) \in T_\theta M$).
• Implication: Once the optimization trajectory enters $M$, the gradient flow remains confined within $M$. The model cannot escape to explore other regions of the parameter space necessary for new tasks without external intervention.

B. Identification of Two Primary Trap Mechanisms

The authors theoretically characterize two specific classes of invariant manifolds that act as "traps":

1. Frozen-Unit Manifolds ($M_F$): Caused by activation saturation. If units become persistently saturated (e.g., ReLU units with large negative biases or Tanh units in saturation), their derivatives vanish ($f'(z) = 0$). Consequently, gradients for incoming weights vanish, freezing those parameters. The trajectory becomes trapped in an affine subspace where these weights are constant.
2. Cloned-Unit Manifolds ($M_C$): Caused by representational redundancy. If units within a block become perfectly redundant (cloned), they share identical forward activations and backward errors. The authors prove that under standard gradient descent (and variants like SGD, Momentum, Adam), if the weights satisfy specific equitability constraints (equal row-sums and column-sums across blocks), the gradients remain constant across the block. This creates a "cloning manifold" where the network effectively reduces its capacity, and standard updates cannot break the symmetry to diversify the units.

C. The Rank-Plasticity Tension

A key theoretical contribution is the derivation of a fundamental tension between generalization and plasticity:

• Mechanism: The paper analyzes the dynamics of feature rank. Non-linearities can increase the effective rank of representations (decorrelation), but this process often drives units toward saturation (to maximize decorrelation strength $\gamma_\phi$).
• Neural Collapse: The drive toward low-rank structures (Neural Collapse), which is beneficial for generalization on static datasets, acts as an attractive force toward the LoP manifolds.
• Conclusion: The very mechanisms that optimize performance on the current task (feature compression, low-rank simplicity) inadvertently construct the barriers to adaptability for future tasks.

3. Key Contributions

1. Dynamical Systems Definition: Formalizes LoP as the entrapment of optimization trajectories within invariant sub-manifolds, moving beyond descriptive symptoms to a geometric explanation.
2. Theoretical Characterization of Traps: Proves the existence and invariance of Frozen-Unit and Cloned-Unit manifolds under standard gradient-based optimization.
3. Rank-Plasticity Trade-off: Demonstrates theoretically that maximizing feature diversity (decorrelation) often necessitates pushing units into saturated regimes, creating a direct link between "rich" feature learning and the emergence of dead/frozen units.
4. Modular Cloning Theorem: Extends the cloning concept to modern architectures (ResNets, ViTs) via a modular composition theorem, showing that local cloning certificates propagate globally.

4. Empirical Results

The authors validate their theory across MLPs, CNNs, ResNets, and Vision Transformers (ViTs) using continual learning benchmarks (Tiny ImageNet) and cloning experiments.

• Cloning Experiments:
  • When a model is artificially expanded by cloning units (duplicating weights/activations), standard SGD and Adam fail to escape the manifold; the cloned units remain identical (high $R^2$ score) and the effective rank does not recover.
  • Perturbations: Introducing Dropout or Noisy SGD breaks the symmetry. Dropout creates asymmetric forward/backward passes for cloned units, allowing gradients to diverge and the model to escape the manifold. Noisy SGD similarly provides the necessary perturbation to escape saddle-like structures.
• Continual Learning Symptoms:
  • As training progresses on sequential tasks, the fraction of dead and duplicate units increases, correlating with a drop in training accuracy and effective rank.
  • Normalization: Architectures with Batch Normalization (BN) or Layer Normalization (LN) show significantly fewer dead/duplicate units and maintain higher effective ranks compared to baselines, confirming that normalization prevents the drift into saturated regions.
• Recovery via Perturbation:
  • In "Bit-Flipping" experiments (non-stationary regression), switching from standard SGD to Continual Backpropagation (CBP) or injecting noise allows the model to recover representational diversity and reduce loss after it has degraded, proving that LoP states are often unstable saddles rather than stable minima.

5. Significance and Implications

• Paradigm Shift: The paper shifts the understanding of LoP from a "failure of optimization" to a "geometric property of the loss landscape." It explains why standard optimizers fail: they are mathematically constrained to stay on low-dimensional manifolds once entered.
• Design Principles: It highlights a critical design trade-off. Techniques that promote generalization (like low-rank compression) can be detrimental in dynamic environments.
• Mitigation Strategies:
  • Prevention: Use normalization layers to keep activations in dynamic ranges and avoid saturation.
  • Recovery: Use symmetry-breaking perturbations (Dropout, Noise Injection, CBP) to destabilize the manifold and allow the network to "unlearn" the redundancy.
• Future Directions: The work suggests that truly lifelong learning agents require mechanisms that actively regenerate representational diversity, potentially by integrating noise or dynamic architectural changes to prevent entrapment in these topological traps.

In summary, this paper provides a rigorous mathematical foundation for the "Loss of Plasticity," identifying specific geometric traps in the parameter space and offering theoretical and empirical pathways to prevent or recover from them.