💻 computer science

NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting

This paper introduces NeuroSynth, a biologically inspired continual reinforcement learning architecture that mitigates catastrophic forgetting through a dual-pathway consolidation mechanism, demonstrating significantly improved retention of prior task knowledge compared to standard PPO and EWC baselines in sequential navigation tasks.

Original authors: Yash Kini

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yash Kini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Brain's Secret to Never Forgetting (And Why Robots Struggle)

Imagine you are trying to learn a new video game. You get really good at the first level, but the moment you start the second level, your brain seems to hit a "reset" button, and you forget how to play the first one. This isn't just a human quirk; it's a massive headache for Artificial Intelligence (AI). In the world of machine learning, this is called "catastrophic forgetting." It happens when an AI learns a new task and accidentally overwrites the code it used for the old task, like writing a new song on a cassette tape and erasing the hit single from the 90s in the process.

Scientists have been looking at how our own brains solve this problem. They believe our brains use a clever two-part system. One part, the hippocampus, is like a super-fast notepad that grabs new information instantly. The other part, the neocortex, is like a slow-cooking library that takes time to organize and store those notes into long-term memory. While we can learn new things without losing old ones, most AI systems are built like a single, giant brain that tries to do everything at once, leading to that frustrating "reset" button effect. The big question is: Can we build an AI that learns like a human, keeping its old skills while mastering new ones?

NeuroSynth: The AI That Remembers Its Roots

Enter NeuroSynth, a new AI architecture created by Yash Kini, a high school researcher, designed to stop this memory loss. Think of NeuroSynth not as a single brain, but as a team of two specialists working together: a "Planner" and a "Habit-Former."

The Planner is inspired by the brain's hippocampus. It's the fast learner. When the AI faces a new challenge, the Planner jumps in, figures it out quickly, and then—crucially—freezes its knowledge. It's like taking a perfect photo of the solution and locking it in a vault so it can never be accidentally erased. The Habit-Former, inspired by the brain's cortex, is the flexible worker. It stays open and keeps learning new things. But here's the magic trick: the Habit-Former doesn't just learn on its own. It constantly looks at the frozen photo in the vault (the Planner) and practices the old moves using a technique called "replay" (imagining the old game) and "distillation" (copying the Planner's style). This way, the AI can learn a new level without deleting the old one.

To test if this brain-like team worked better than standard AI, the researchers put three different models to the test in a digital maze called NeuroMaze-CL. The goal was simple: navigate a grid to find a specific exit. The twist? The exit moved every time. The AI had to learn the first exit, then the second, then the third, without ever being allowed to practice the first two again. This "non-revisitation" rule was designed to force the AI to forget, just like a student who is only allowed to study for the final exam and never looks at their old notes.

The results were a clear victory for the brain-inspired approach. The standard AI model, called PPO, was a total disaster at remembering. After learning the third maze, it completely forgot the first one, achieving a success rate of just 0.33% on Task A. It was as if the AI had never seen the first maze at all. The second model, EWC, tried to protect its old memories by being very careful with its updates, but it was so cautious that it struggled to learn the new mazes effectively.

NeuroSynth, however, found the sweet spot. After the same training, it remembered the first maze with a 18.00% success rate—significantly better than the PPO's near-zero performance. It didn't just remember the first task; it also held onto the second task much better, achieving a 35.33% success rate on Task B compared to PPO's 0.00%. When it came to the final, hardest task (Task C), NeuroSynth performed better than the cautious EWC model (9.00% vs. 2.00%), though this specific difference wasn't statistically strong enough to be called a definitive win.

The study suggests that by separating the "fast learning" from the "slow remembering" and using replay to practice old skills, we can build AI that is both stable and flexible. While this isn't a magic bullet that solves every problem in robotics or medical AI yet, it shows that copying the brain's two-pathway system is a promising way to stop robots from hitting that frustrating "forget" button every time they learn something new. As the author notes, this is a step toward machines that can truly learn a lifetime of skills without losing the ones they already mastered.

Technical Summary: NeuroSynth – A Biologically Inspired Continual Reinforcement Learning Architecture

Problem Statement

Artificial Intelligence systems, particularly in reinforcement learning (RL), frequently suffer from catastrophic forgetting. This phenomenon occurs when a neural network trained sequentially on new tasks overwrites the parameters critical for previously learned behaviors, leading to a rapid degradation of prior knowledge. While biological learning systems mitigate this interference through complementary memory processes—specifically the interaction between rapid hippocampal encoding and slower cortical consolidation—most artificial RL architectures lack explicit memory-system separation. Existing solutions often rely on a single adaptive representation space, making them vulnerable to gradient updates that prioritize new tasks at the expense of old ones.

Methodology

Experimental Framework

The study evaluates NeuroSynth, a brain-inspired architecture, against two standard continual learning baselines:

Proximal Policy Optimization (PPO): A standard actor-critic method with no specific continual learning mechanisms, serving as a reference for catastrophic forgetting.
Elastic Weight Consolidation (EWC): A regularization-based approach that penalizes updates to parameters estimated to be important for prior tasks using Fisher Information matrices.

The evaluation takes place in NeuroMaze-CL, a custom continual RL environment built on the Gymnasium framework. The environment is an 8×8 grid-world where agents navigate to specific goal locations using partially observable, egocentric 7×7 RGB observations. The experiment consists of three sequential tasks (Task A → Task B → Task C) with changing goal locations. Crucially, the setting is non-revisitation: once a task phase ends, the agent never trains on it again, forcing the model to rely on internal retention mechanisms.

NeuroSynth Architecture

NeuroSynth translates Complementary Learning Systems (CLS) theory into a functional algorithmic framework by separating rapid acquisition from long-term consolidation via a dual-pathway mechanism:

Hippocampal-Inspired "Plan" Pathway:
- Rapidly acquires representations for the initial task (Task A) via temporal-difference learning.
- Once Task A training is complete, all planning parameters are frozen to preserve early representations.
- An Exponential Moving Average (EMA) copy of this frozen network acts as a "teacher" for subsequent phases.
Cortical-Inspired "Habit" Pathway:
- Remains plastic during Tasks B and C, allowing the agent to learn new objectives.
- Prevents forgetting of Task A through two mechanisms:
  - Replay: A reservoir sampling buffer stores historical Task A states. During new training, the habit pathway is exposed to a mix of current-task experiences and replayed prior-task experiences.
  - Policy Distillation: The habit pathway (student) is aligned with the frozen plan pathway (teacher) using Kullback–Leibler (KL) divergence losses on replayed Task A states. This stabilizes the policy behavior associated with earlier states without retraining the frozen planner.

Training Protocol

Seeds: Six independent random seeds were used for statistical reliability.
Steps: Task A (100k steps), Task B (100k steps), Task C (200k steps). Task C was extended to increase representational interference.
Evaluation: Deterministic schedules with 50 episodes per task post-training. Success rate is defined as the percentage of episodes reaching the goal.

Key Results

The study reports performance across the three tasks after sequential training, focusing on retention of earlier tasks and acquisition of the final task.

Task A Retention (Post-Task C):
- PPO: Suffered near-total forgetting, achieving a 0.33% success rate.
- NeuroSynth: Achieved an 18.00% success rate.
- Significance: The difference is statistically significant ( $p = 0.014929$ ) with a large effect size (Cohen's $d = 1.49$ ).
Task B Retention (Post-Task C):
- PPO: Achieved 0.00% success rate.
- NeuroSynth: Achieved 35.33% success rate.
- Significance: The difference is highly significant ( $p = 0.002376$ ) with a very large effect size (Cohen's $d = 2.31$ ).
Task C Acquisition (Final Performance):
- EWC: Achieved 2.00% success rate, indicating that parameter constraints limited adaptability to new tasks.
- NeuroSynth: Achieved 9.00% success rate.
- Significance: While NeuroSynth outperformed EWC, the difference was not statistically significant ( $p = 0.226643$ ), though it showed a moderate effect size (Cohen's $d = 0.56$ ).

Key Contributions

Architectural Separation: NeuroSynth introduces a dual-pathway design that explicitly separates rapid acquisition (Plan) from slower consolidation (Habit), mimicking biological hippocampal-cortical interactions.
Hybrid Stabilization: The architecture combines replay (data rehearsal) with policy distillation (behavioral alignment) to stabilize representations without relying solely on static parameter regularization.
Empirical Validation: The study demonstrates that this biologically inspired approach significantly outperforms standard PPO in retaining prior knowledge and offers a moderate improvement over EWC in balancing stability with plasticity.

Significance and Claims

The paper claims that NeuroSynth successfully addresses the stability-plasticity tradeoff in continual reinforcement learning. By decoupling the mechanisms for learning new tasks and retaining old ones, the architecture prevents the gradient updates of new tasks from overwriting critical parameters of previous tasks.

The author concludes that biologically inspired consolidation mechanisms—specifically the separation of memory pathways and the use of replay-guided distillation—offer a promising direction for developing stable lifelong learning systems. The results suggest that such architectures can maintain previously acquired behaviors while continuing to adapt to new environments, a capability that standard gradient-based methods (PPO) and rigid regularization methods (EWC) struggle to achieve simultaneously.

Limitations and Future Directions

The paper modestly acknowledges several limitations:

Environment Complexity: Experiments were conducted in low-dimensional, discrete grid-worlds, which do not fully capture the complexity of high-dimensional sensory inputs or continuous control.
Task Scope: Only three sequential tasks were used; scalability to larger, lifelong learning benchmarks remains untested.
Baseline Comparison: Comparisons were limited to PPO and EWC; broader comparisons with hybrid or transformer-based memory systems are needed.
Biological Fidelity: The model is "biologically inspired" rather than "biologically realistic," simplifying complex processes like sleep-dependent replay and synaptic modulation.

Future work proposed by the author includes testing NeuroSynth in robotic and continuous-control environments, investigating sparse-reward scenarios, and exploring adaptive consolidation mechanisms (e.g., dynamic pathway freezing) rather than static freezing to improve long-term flexibility.