🤖 machine learning

Catastrophic Forgetting in Continual Reinforcement Learning

This study investigates the relationship between task similarity and catastrophic forgetting in continual reinforcement learning using Q-learning on graph-based tasks, finding that while forgetting exhibits complex dynamics influenced by both similarity and complexity, there is no statistically significant evidence that task similarity independently drives forgetting.

Original authors: Emma Graham

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Emma Graham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a maze. At first, you show it a simple maze with a few turns, and it learns the perfect path to the treasure. Then, you swap the maze for a slightly different one. If the robot is too eager to learn the new maze, it might completely forget how to solve the old one. This phenomenon is called "catastrophic forgetting," and it's a major headache for scientists trying to build AI that can learn continuously, just like humans do. The big question is: does it matter how different the new maze is? If the new maze looks a lot like the old one, does the robot remember better? Or if it looks totally different, is that actually easier to handle? This paper dives into that mystery using a specific type of robot learning called "Reinforcement Learning," where the agent learns by trial and error to reach a goal in as few steps as possible.

The researchers behind this study decided to test this idea using a clever trick involving geometry. Instead of using messy, real-world mazes, they built their "mazes" out of mathematical shapes called Voronoi diagrams. Think of these as a map where every point on the ground belongs to the nearest "seed" point, creating a patchwork of polygons. To create a new task, they simply nudged the positions of these seed points a tiny bit. This allowed them to create a whole family of mazes that were structurally related but slightly different. They trained a robot agent on one maze, then taught it a second, slightly different maze, and finally checked to see how much it had forgotten about the first one.

The results, however, were a bit of a surprise and not as straightforward as one might hope. The study found that the relationship between how similar the tasks were and how much the agent forgot is incredibly complex and messy. While the researchers saw big swings in how much forgetting happened depending on the task differences, they could not find a clear, consistent rule. In other words, they couldn't prove that being "similar" or "different" is the main reason an agent forgets. The data showed high variability, suggesting that forgetting depends on a tangled mix of task similarity and how complicated the tasks are, rather than just one factor. Ultimately, the paper concludes that while the connection between task similarity and forgetting is fascinating, there is currently no solid statistical evidence that similarity alone drives forgetting in this setting. The story isn't solved yet; it just tells us that the answer is far more complicated than a simple "similar equals safe" or "different equals bad" rule.

Technical Summary: Catastrophic Forgetting in Continual Reinforcement Learning

Problem Statement

Continual learning, the capacity of a system to learn tasks sequentially while retaining knowledge of previous tasks, remains a significant unresolved challenge in machine learning. A primary barrier to this capability is catastrophic forgetting, where a model loses previously acquired knowledge upon learning new information. While extensive research exists on mitigating forgetting in supervised learning and neural networks, the specific relationship between task similarity and the severity of forgetting in Reinforcement Learning (RL) remains under-quantified.

This study investigates the extent to which the similarity of a new task influences performance degradation on a previously learned task within a continual RL setting. The research aims to determine if a quantifiable relationship exists between the degree of task difference and the magnitude of catastrophic forgetting, specifically in interpretable, tabular RL agents.

Methodology

Experimental Framework

The study employs tabular Q-learning on graph-based tasks. The objective for the agent is to minimize the number of steps required to reach a specific goal state. The use of tabular methods ensures interpretability, while the graph-based environment allows for precise control over task structure.

Task Generation

Tasks are generated using Voronoi diagrams derived from a set of sites (points) in a 2D plane.

Structure: The graph nodes correspond to the vertices of the Voronoi diagram, and edges represent the connections between adjacent regions.
Similarity Control: To generate a sequence of related tasks, the internal sites of the Voronoi diagram are perturbed. A base set of sites ( $x_0$ ) and a new set ( $x_1$ ) are sampled from a Gaussian distribution. A new task configuration ( $x_\alpha$ ) is created via linear interpolation:
$x_\alpha = \cos\left(\frac{\pi}{2}\alpha\right)x_0 + \sin\left(\frac{\pi}{2}\alpha\right)x_1$
where $\alpha \in [0, 1]$ controls the degree of perturbation. $\alpha=0$ represents the base task, and $\alpha=1$ represents the fully perturbed task.
Consistency: To ensure the agent can compare tasks, a rotation system is implemented. This assigns actions (back, left, right) based on the relative clockwise ordering of neighbors around a node, rather than absolute coordinates. This allows the agent to interpret the topological structure of the graph independent of spatial distortion.

Measuring Task Similarity

The study defines task similarity based on the topological structural difference between graphs, as the agent optimizes for step count rather than geometric distance.

Metrics: Due to the computational intractability of Graph Edit Distance (NP-hard), the study utilizes a combination of two computationally feasible metrics:
1. Wasserstein Distance ( $W_2$ ): Measures the difference in the spatial distribution of Voronoi cell areas.
2. Hausdorff Distance ( $\delta_H$ ): Measures the maximum distance between the sets of vertices of the two graphs.
Combined Metric: The final similarity measure $\Delta(G_1, G_2)$ is the average of the normalized Wasserstein and Hausdorff distances.

Evaluation Protocol

Phase 1 (Base Learning): An agent is trained on a base graph ( $G_0$ ) until convergence. Performance is measured by the number of steps to reach the goal.
Phase 2 (New Task Learning): The same agent is trained on a new graph ( $G_\alpha$ ) with a different topology but the same goal state definition.
Phase 3 (Forgetting Measurement): The agent is tested on the original base graph ( $G_0$ ) again.
Quantification: Forgetting is quantified as the increase in the number of steps required to solve the base task after training on the new task, compared to the performance before training on the new task.

Key Results

The study conducted one hundred experimental trials across varying levels of task complexity and similarity ( $\alpha$ ).

Complex Dynamics: The results reveal a complex dynamic between task similarity and forgetting, with significant fluctuations in forgetting severity observed across degrees of task similarities and task complexities.
Variability: The data exhibited high degrees of variability in forgetting outcomes, accompanied by an uneven distribution of task similarity measures.
Statistical Significance: Crucially, the study found no evidence of statistical significance that task similarity, independently, has an effect on forgetting in continual reinforcement learning. The relationship between these variables remains unclear, and no specific pattern (linear, monotonic, or otherwise) was confirmed.
Interdependence: The observations are suggestive of an interdependence of forgetting on both the similarity and complexity of tasks, but the specific nature of this relationship was not resolved.

Significance and Claims

The paper positions itself as an exploratory investigation into a gap in the literature: the quantification of how task differences specifically drive catastrophic forgetting in RL.

Modest Claims: The author explicitly states that their observations suggest an interdependence of forgetting on both similarity and complexity, but they do not claim to have proven a definitive causal mechanism or a specific functional relationship.
Negative Result as Contribution: A primary contribution is the finding that, within the constraints of this specific tabular, graph-based setup, task similarity alone does not appear to be a statistically significant predictor of forgetting severity. This challenges the assumption that similarity is a straightforward proxy for forgetting risk in all RL contexts.
Future Directions: The paper concludes that further research is warranted to comprehensively understand the potential interplay between task similarity and catastrophic forgetting, noting that the current results highlight the need for more robust experimental designs or alternative metrics to resolve the observed variability.

The study does not propose new mitigation algorithms or specific applications but rather provides a foundational analysis of the problem space using interpretable models to isolate structural variables.