🤖 machine learning

Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization

This paper introduces "local redundancy," an information-theoretic measure of neural network plasticity derived from universal compression theory, which is efficiently approximated by the expected squared gradient norm on a synthetic memorization task and demonstrated to outperform existing metrics in predicting downstream performance and guiding checkpoint selection.

Original authors: Jiaxuan Cheng

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaxuan Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a never-ending video game where the levels keep changing. At first, the robot learns quickly, mastering each new level with ease. But after playing for a long time, something strange happens: the robot gets stuck. It can still remember the old levels perfectly, but it becomes clumsy and slow at learning new ones. In the world of artificial intelligence, this is called a "loss of plasticity." It's like a muscle that has grown so strong at one specific movement that it can no longer bend to learn a new dance. Scientists have been trying to figure out how to measure this "stiffness" in a robot's brain so they can fix it before it happens. They've tried looking at how many neurons are "sleeping" or how heavy the robot's internal weights are, but these methods often fail to predict when the robot will actually struggle with a new task. They are like trying to guess how flexible a rubber band is by just looking at its color; it doesn't tell you the whole story.

This paper introduces a new, smarter way to measure that flexibility, called "local redundancy." Think of a neural network (the robot's brain) as a giant, complex map. Usually, we look at the whole map to see how big it is. But this paper argues that what really matters is how much room there is to move right now, from the exact spot the robot is standing. The authors use a concept from information theory—basically, the science of how much data you need to send a message—to measure this. They imagine a tiny neighborhood around the robot's current settings and ask: "If we nudge the robot just a tiny bit, how many different new worlds can it suddenly understand?" If the answer is "many," the robot is flexible (plastic). If the answer is "very few," it's rigid.

To measure this without actually changing the robot's brain, the researchers came up with a clever trick. They feed the robot a bunch of "fake" data—like a picture of a random jumble of shapes with a completely random label, or a time series of numbers that make no sense. They then ask the robot to try to memorize this nonsense. The key finding is that the "sweat" the robot breaks out while trying to memorize this fake data (measured by how hard it has to work, or its "gradient norm") tells us exactly how flexible it is. If the robot can easily learn the nonsense, it has high plasticity. If it struggles, it's getting stiff.

The paper proves mathematically that this "memorization effort" is a reliable lower bound for the robot's flexibility. In experiments, they tested this on two different challenges: teaching a robot to recognize thousands of different image pairs one after another, and teaching it to predict electricity usage patterns. They found that their new "local redundancy" measure was much better at predicting how well the robot would do on future tasks than the old methods. For instance, in the image task, their measure correlated much more strongly with future success than simply counting how many neurons were asleep. Even more impressively, in the electricity prediction task, they found a moment where the robot's usual "score" (validation loss) stopped improving and looked like it was done learning. But their new measure kept going, showing that the robot still had hidden flexibility. By picking the checkpoint where this flexibility was highest, they got the robot to learn the new task better than if they had just picked the checkpoint with the best old score.

The authors are careful to note that while this method works great as a predictor, it doesn't prove that causing more flexibility would automatically fix the problem—that would need a different kind of experiment. Also, they admit their "fake data" isn't perfect; it only fills about 1 to 2 "bits" of information per parameter, which is far less than the theoretical maximum a robot could hold. But despite these limits, the paper suggests that by using this simple, one-step test of memorizing nonsense, we can spot when an AI is about to lose its ability to learn and choose the best moment to save its progress, ensuring it stays ready for whatever comes next.

Technical Summary: Local Redundancy as an Information-Theoretic Measure of Plasticity

Problem Statement
Neural networks trained over long horizons or across multiple tasks often suffer from a "loss of plasticity," a phenomenon where the network progressively loses the ability to learn new tasks effectively. This issue compromises continual learning and transfer learning pipelines, particularly in stages involving fine-tuning and alignment. Existing metrics used to diagnose this degradation—such as effective rank, the fraction of dormant neurons, weight norms, and distance from initialization—lack a unified theoretical foundation. Empirically, these proxies correlate poorly with future task performance and fail to provide a principled measure of a network's current capacity to adapt.

Methodology: Local Redundancy
The authors introduce local redundancy, an information-theoretic measure of plasticity derived from universal compression theory.

Theoretical Definition: The framework defines plasticity not as the global capacity of an architecture, but as the capacity of a network to learn from its current parameter state ( $\theta_0$ ). The authors define a local model family as the set of distributions reachable within an infinitesimal neighborhood along gradient directions from $\theta_0$ .
Redundancy as Plasticity: Drawing on the theory of minimax redundancy (Shtarkov, 1987; Rissanen, 1984), local redundancy is defined as the worst-case redundancy of this local family. Geometrically, this corresponds to the information radius of the set of distributions the model can express via a single gradient step. A high information radius implies the network can steer its output distribution toward many diverse targets (high plasticity), whereas a low radius indicates the network has collapsed into a rigid region where gradients induce minimal change.
Computable Lower Bound: Since exact computation of worst-case redundancy is intractable, the authors prove that the expected squared gradient norm on a synthetic memorization task provides a computable lower bound on local redundancy (Theorem 3.4).
- Synthetic Data Construction: The method utilizes synthetic inputs (e.g., random shapes for vision, Gaussian noise for time series) paired with labels sampled from the model's own predictive distribution ( $y \sim P_{\theta_0}(y|x)$ ).
- Mechanism: The model is not trained on this data. Instead, a single backward pass computes the gradient norm. The theoretical proof shows that the ability of the current parameters to fit random labels (memorization gain) via a small gradient step is directly related to the worst-case redundancy of the local family.

Key Contributions

Definition of Local Redundancy: The paper formalizes plasticity as the worst-case redundancy of a local parameter neighborhood, grounding the concept in universal compression theory and the information radius.
Efficient Estimation: The authors prove that the expected squared gradient norm on synthetic memorization data serves as a lower bound for local redundancy. This estimator requires only a single backward pass and does not require access to downstream task data.
Empirical Validation: The framework is validated in two distinct settings: continual image classification and time series transfer learning.

Results

Continual Learning (Continual ImageNet): In a benchmark involving 3,000 sequential binary classification tasks, local redundancy was found to decrease steadily as training progressed, tracking the loss of plasticity. Crucially, when predicting future task accuracy (tasks $t+1$ to $t+10$ ) after regressing out the task number, local redundancy demonstrated the highest correlation (Pearson $r \approx 0.089$ ) compared to existing metrics like weight norm, dormant neuron ratio, and training gradient norm.
Transfer Learning (ETT Time Series): In selecting pretraining checkpoints for time series forecasting, the authors compared strategies based on validation loss versus local redundancy. While validation loss often plateaus, local redundancy continued to evolve. Selecting checkpoints with maximum local redundancy yielded better downstream fine-tuning performance (lower MSE) than selecting the checkpoint with the lowest validation loss. Notably, this was the only metric among those tested (including dormant neuron ratio and weight norm) that successfully identified superior checkpoints once validation loss saturated.
Stability-Plasticity Tradeoff: Local redundancy showed a positive correlation with forgetting (the drop in previous task accuracy), consistent with the theoretical tradeoff where higher adaptability implies a greater tendency to overwrite prior knowledge.

Significance and Claims
The paper claims that local redundancy offers a principled, computable measure of plasticity that outperforms existing heuristics in predicting downstream performance. Its primary significance lies in its ability to guide checkpoint selection in scenarios where traditional metrics (like validation loss) fail to discriminate between models, specifically when validation loss has plateaued but the model's capacity to adapt continues to evolve.

The authors maintain a modest stance regarding causality, noting that while local redundancy predicts performance, the study does not prove it causes it; controlled interventions would be required to establish causation. Furthermore, they acknowledge that the current synthetic memorization datasets only saturate a fraction of the theoretical information capacity (approx. 1–2 bits per parameter), suggesting the gradient norm is a proxy rather than a calibrated absolute value. Nevertheless, the framework provides a new theoretical lens for understanding and measuring the "learnability" of neural networks in their current state.

Technical Summary: Local Redundancy as an Information-Theoretic Measure of Plasticity

More like this