🤖 machine learning

Plasticity of Growing and Elastic Neural Networks in Online Continual Learning

This paper investigates the plasticity of growing and elastic neural networks in online continual learning, demonstrating that adaptive approaches which incrementally add new units while retaining existing connections or pruning dead ones can maintain high prediction accuracy and prevent loss of plasticity without catastrophic forgetting.

Original authors: Jeong Min Kong, Richard S. Sutton

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeong Min Kong, Richard S. Sutton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your brain could instantly forget how to ride a bike the moment you learned to juggle, or where a student's mind would get so full of history facts that it couldn't learn a single new math equation. This is the nightmare scenario for artificial intelligence known as "catastrophic forgetting." But there is an even sneakier villain at play: "loss of plasticity." Think of plasticity as the brain's ability to stay flexible and moldable. If a network loses plasticity, it becomes like a hardened statue—stiff, unchangeable, and unable to learn anything new, even if it remembers the old stuff perfectly.

For a long time, scientists tried to fix these problems by building AI with a fixed number of "neurons" (the tiny processing units inside the brain), hoping that if the structure never changed, the AI would be stable. But nature tells a different story. Animals don't have a fixed brain size; they grow new connections and prune old ones as they learn. This paper explores a bold idea: what if we build AI that can actually grow and shrink, just like a living thing? The researchers wanted to see if these "growing" and "elastic" networks could keep their flexibility (plasticity) alive while learning a never-ending stream of new tasks, without getting stuck or forgetting everything.

The Story of the Growing and Shrinking AI

In this study, the researchers set up a digital playground where an AI had to learn a series of image classification tasks, one after another, like a student taking a new test every day. They used two popular image sets: MNIST (handwritten numbers) and FashionMNIST (images of clothes), but with a twist. For each new test, the pixels of the images were scrambled in a unique, random way, forcing the AI to learn from scratch every time without relying on the same visual patterns.

The team tested several different "architectures," or blueprints, for these AI brains. First, they tried the standard approach: a Fixed Network. Imagine a classroom with a set number of desks. No matter how many new students (tasks) arrive, the desks stay the same. The researchers found that as the AI learned more tasks, its performance slowly crumbled. It lost its "plasticity," becoming rigid and unable to adapt, while a growing number of its internal units went "dead"—stopping all activity and becoming useless.

Next, they tried Staged Growing Networks. This was like a classroom that adds one new desk every day, but with a catch: once a desk is added, it gets bolted to the floor and can never be moved or adjusted again. The AI could add new units, but the old ones were frozen in place. The results were disappointing. Because the AI was forced to keep using these frozen, outdated desks, it got cluttered with "noisy" features that didn't help with the new tasks. The AI lost its flexibility, and its accuracy dropped.

Then, the researchers introduced a simple but powerful change: Adaptive Growing Networks (AGNs). This was the same as the previous version—adding a new desk every day—but with one crucial difference: nothing was ever bolted down. Every connection in the network remained flexible and could be adjusted at any time. The results were a revelation. Even though the network kept getting bigger and bigger, and even though many of the old units eventually went "dead" (stopped firing), the AI maintained high accuracy and never lost its plasticity. It seemed that as long as the network could keep tweaking its existing connections while adding new, fresh units, it could keep learning forever.

However, there was a catch. The AGN kept growing infinitely, which would eventually eat up all the computer's memory and slow everything down. It was like a student who keeps adding new notebooks to their backpack but never throws away the old ones; eventually, they can't carry the bag anymore.

To solve this, the team created Adaptive Elastic Networks (AENs). These networks were like the AGNs but with a built-in "spring" mechanism. At the start of every new task, the AI would check its brain and prune (remove) any units it estimated were "dead" or useless. This allowed the network to shrink back down while still adding new units. The result was a "Goldilocks" solution: the AI achieved excellent accuracy, kept its plasticity high, and maintained a compact, near-constant size. It grew when it needed to and shrank when it didn't, staying efficient without losing its ability to learn.

The researchers also tested a "two-layer" version of these networks, where new units only connected to the input and output, skipping the middle layers. This made the learning process faster, though it sometimes resulted in a slightly larger final size.

What This Means for the Future

The paper suggests that the key to keeping AI flexible isn't just about remembering old things or preventing forgetting; it's about keeping the structure of the network alive and adaptable. By allowing networks to grow and shrink dynamically, and by ensuring that connections remain flexible rather than frozen, we can build systems that learn continuously without losing their spark.

The authors are careful to note that these findings come from specific experiments with image tasks and that the results "suggest" these methods are promising. They don't claim to have solved the entire problem of lifelong learning, but they have shown a clear path forward. Future work will need to explore exactly when to add or remove units and how these networks behave with different types of data. But for now, the idea of an AI that can grow like a plant and prune like a gardener seems to be a very strong candidate for the future of intelligent machines.

Technical Summary: Plasticity of Growing and Elastic Neural Networks in Online Continual Learning

Problem Statement

Modern neural networks typically employ a fixed architecture where neuron arrangements and interconnections remain static throughout the learning process. While research into "constructive" networks (those that can add or remove units) exists, prior work has largely focused on offline continual learning settings and the mitigation of catastrophic forgetting (the loss of previously learned knowledge).

This paper addresses two critical gaps:

Online Continual Learning: The setting where data arrives sequentially, one sample at a time, closely resembling natural learning systems.
Loss of Plasticity: A phenomenon distinct from catastrophic forgetting, defined as the progressive decline in a network's ability to learn new tasks. Recent work (Dohare et al., 2024) suggests that the incremental introduction of randomly initialized hidden units can help preserve plasticity.

The core problem investigated is whether foundational growing (networks that only add units) and elastic (networks that can both grow and shrink) architectures can maintain high plasticity and prediction accuracy in online continual learning settings, specifically on permuted image classification tasks.

Methodology

Experimental Setup

The authors evaluate various algorithms on Online Permuted MNIST and Online Permuted FashionMNIST (FMNIST).

Task Structure: A sequence of image classification tasks where each task applies a unique, random pixel permutation to the dataset.
Training Regimen: Online learning where samples are presented one by one. Parameters are optimized immediately after each forward pass using Stochastic Gradient Descent (SGD) with a step size of 0.001 to minimize cross-entropy loss.
Datasets: Two sample sizes per task are tested: $N=10,000$ and $N=40,000$ .
Metrics:
- Accuracy: Correct predictions divided by samples per task.
- Dormancy Percentage: The proportion of "dead" hidden units (units outputting zero for all task samples due to zero/near-zero gradients) relative to the total hidden units at the start of a task.

Algorithms Evaluated

The study compares several architectures, moving from fixed baselines to adaptive constructive networks:

Fixed Fully-Connected Neural Networks (F-FCNNs): Standard dense networks with a static structure serving as the baseline.
Staged Growing Networks (SGNs): A variant of the Cascade-Correlation (CasCor) algorithm adapted for online learning.
- Mechanism: Starts with a linear network. Every $K$ samples, learning pauses, and a new hidden unit is added with randomized weights.
- Constraint: Connections of previously added hidden units are frozen once the new unit is added. Only new connections to the new unit are adaptive.
Adaptive Growing Networks (AGNs): A modification of SGNs.
- Mechanism: Identical to SGNs in adding new units every $K$ samples, but all network connections remain adaptive and are never frozen.
Adaptive Elastic Networks (AENs): An extension of AGNs designed to control network size.
- Mechanism: At the beginning of each new task, hidden units estimated to be dead (based on a 5% random subset of samples) are pruned.
- Note: Pruning estimated dead units may inadvertently remove some active units, altering the learning trajectory compared to AGNs.
Two-Layer Variants: Variants of AGNs and AENs where new hidden units receive connections only from input units, prohibiting connections between hidden units. This enforces a strict two-layer structure to potentially reduce computational overhead.

Key Results

1. Fixed Networks and Staged Growing Networks (SGNs)

F-FCNNs: Exhibit a progressive loss of plasticity, characterized by declining accuracy over successive tasks and a high, saturating dormancy percentage.
SGNs: Despite adding new capacity, SGNs also lose plasticity. The enforced freezing of old features forces the network to rely on features that may no longer be relevant to current tasks. Furthermore, SGNs show a continuous rise in dormancy percentage, indicating that newly added units often die before they can be utilized or frozen.

2. Adaptive Growing Networks (AGNs)

Performance: AGNs maintain high prediction accuracy across all tasks without losing plasticity, outperforming F-FCNNs and SGNs significantly.
Plasticity vs. Dormancy: A key finding is that AGNs maintain high accuracy even as the dormancy percentage continually increases. This contrasts with previous observations where high dormancy correlated with performance drops. The authors hypothesize this is due to the continuous adaptation of existing connections to new tasks combined with the injection of new capacity via random units.
Limitation: The network size grows indefinitely, leading to exponential increases in memory usage and computational time for forward/backward passes.

3. Adaptive Elastic Networks (AENs)

Performance: AENs achieve excellent accuracy and maintain high plasticity comparable to AGNs.
Efficiency: By pruning estimated dead units at the start of each task, AENs maintain a near-constant, compact network size.
Dormancy: Unlike AGNs, the dormancy percentage of AENs hovers consistently around 0%, indicating highly efficient utilization of network parameters.
Scaling: The saturated network size scales approximately linearly with the number of new units added per task for MNIST. Interestingly, for the more complex FMNIST task, the saturated size is sometimes lower than that of MNIST.

4. Two-Layer Variants

Behavior: Two-layer AGNs and AENs (where hidden units do not connect to other hidden units) exhibit similar performance and size trends to their multi-layer counterparts.
Trade-off: While they offer potential computational efficiency by restricting depth, the converged network size (total connections) can be larger than the original variants in some cases (e.g., MNIST).

Significance and Claims

The paper claims that growing and elastic networks represent a promising class of algorithms for preserving high plasticity in online continual learning. Specifically:

Adaptive Growing Networks demonstrate that keeping all connections adaptive while incrementally adding random units allows a network to sustain high accuracy and plasticity, even as the proportion of dead units rises.
Adaptive Elastic Networks extend this capability by introducing a simple pruning strategy (removing estimated dead units at task boundaries). This enables the network to maintain high plasticity and accuracy while simultaneously keeping the model size compact and constant.

The authors conclude that these architectures, which adapt their structure to learning objectives, effectively address the challenge of loss of plasticity in online settings, offering a viable alternative to fixed-structure networks.

Future Directions

The paper outlines two primary directions for future work:

Further Exploration: Investigating the timing of adding and pruning units, and analyzing other metrics associated with plasticity loss, such as the effective rank of representations and average weight magnitude.
New Algorithm Design: Developing networks that can incorporate multiple new units simultaneously (similar to constructive-columnar networks) and architectures that better exploit both network depth and hidden size, bridging the gap between the adaptive networks and their two-layer variants.