Forgetting, plasticity, and co-observation: a third facet of continual learning
This paper argues that beyond the well-known challenges of catastrophic forgetting and loss of plasticity, the inability to co-observe training data simultaneously is a distinct and critical factor limiting continual learning performance, suggesting that mechanisms like memory replay succeed by reintroducing these co-observation benefits rather than merely mitigating forgetting.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Deep neural networks are the powerful engines behind modern artificial intelligence, capable of recognizing faces, translating languages, and diagnosing diseases. These systems are typically trained in a single, massive session where they see all their data at once, allowing them to find patterns that connect different pieces of information. However, in the real world, data often arrives in a steady stream over time, like a river that never stops flowing. To keep these systems useful, they must learn from this new information as it arrives without being able to revisit the old data. This process is known as continual learning. For years, scientists have believed that the main obstacle to making this work is a problem called catastrophic forgetting, where a model learns something new but accidentally erases what it knew before. The prevailing view was that if we could simply stop the model from forgetting and keep it flexible enough to learn new things, it would perform just as well as if it had seen all the data together from the start.
A team of researchers at KU Leuven and the University of Groningen has challenged this long-held assumption. They discovered that even if a model never forgets anything and remains perfectly flexible, it still struggles to reach the same level of intelligence as a model trained on all the data at once. Their work, presented at the 5th Conference on Lifelong Learning Agents, identifies a third, hidden factor that limits how well these systems learn: the loss of co-observation. This term describes the simple fact that when data is learned in separate chunks over time, the model misses the opportunity to see different pieces of information simultaneously. The researchers found that this missing connection prevents the model from building the most robust understanding of the world, a limitation that exists regardless of how well forgetting is managed.
To understand why this matters, imagine trying to assemble a complex puzzle. If you are given all the pieces at once, you can easily spot how the edges of one section connect to the center of another, allowing you to see the bigger picture and fit the pieces together efficiently. This is how standard artificial intelligence is usually trained. In contrast, continual learning is like being handed the puzzle pieces one by one, with no chance to look back at the ones you already placed. You might fit the current piece perfectly into the space you have, but without seeing the surrounding pieces, you might miss a crucial connection that would have helped you understand the whole image. The researchers argue that this inability to see the "whole puzzle" at once creates a fundamental gap in performance that cannot be fixed simply by remembering the past.
The team designed a series of experiments to prove that this gap is real and distinct from the problem of forgetting. They used a method that allowed them to isolate the two issues. First, they created a scenario where a model learned from data in four separate batches, mimicking the real-world stream of information. To ensure that any drop in performance wasn't just due to the model forgetting earlier lessons, they used a special technique called an ensemble. This involved saving a copy of the model's brain after every single learning step and combining all those copies together. This combined model effectively remembered everything perfectly, eliminating the problem of forgetting entirely. They then compared this "perfect memory" model against a standard model that saw all the data at once.
The results were clear and consistent. Even with perfect memory, the model that learned in separate chunks performed worse than the model that saw everything together. This difference held true whether the computer was learning to recognize images in a supervised way, where it was told the correct answers, or in a self-supervised way, where it had to figure out patterns on its own. The researchers tested this on standard image datasets, including CIFAR-100 and ImageNet-100, using different types of neural network architectures. In every case, the model that learned sequentially, even with perfect retention, failed to reach the same level of generalization as the one trained jointly. This proved that the issue was not a failure to remember, but a failure to synthesize information across different time periods.
The study also looked at how current solutions to continual learning handle this problem. One popular method is experience replay, where the model is shown a few old examples alongside new ones to help it remember. The researchers found that this method works, but not just because it stops forgetting. Instead, it works because it artificially recreates the condition of co-observation. By showing old and new data together, even in small batches, the model gets a chance to see the connections between them, which helps it build better representations. Another common technique, known as knowledge distillation, which tries to force the new model to mimic the old one, was found to be effective at preventing forgetting but failed to close the performance gap. This suggests that simply preserving old knowledge is not enough; the model needs the active benefit of seeing data together to truly learn.
These findings suggest that the field of artificial intelligence needs to rethink its approach to lifelong learning. For years, the focus has been almost exclusively on preventing the loss of old knowledge. While that remains important, the researchers argue that we must also address the structural disadvantage of learning in isolation. The performance ceiling for sequential learning is lower than previously thought, not because the model forgets, but because it lacks the context that comes from simultaneous observation. This insight changes how we evaluate progress in the field. It implies that future algorithms must do more than just protect the past; they must find ways to actively synthesize connections between new and old information, perhaps by designing better ways to replay data or by structuring learning tasks to encourage cross-distribution thinking.
The implications extend beyond just image recognition. The researchers note that this phenomenon likely affects large language models and other complex systems that learn from vast streams of data. If a model learns a new concept without seeing it in the context of what it already knows, it may never fully grasp the relationship between the two. The study does not claim that this is the only problem in continual learning, nor does it suggest that forgetting is no longer a major issue. In many practical applications, forgetting remains the most immediate and damaging problem. However, by identifying co-observation as a distinct and fundamental limitation, the researchers have provided a clearer map of the challenges ahead. They show that to build truly intelligent systems that learn throughout their lives, we must solve not just the problem of memory, but also the problem of perspective.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.