💻 computer science

Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning

This paper proposes Qreg+NWLU, a value-based data rehearsal method for multi-cyclic continual reinforcement learning that overcomes the limitations of actor-centric approaches by introducing continuous Q-value collection and immediate "No-Wait" regularization to effectively mitigate catastrophic forgetting and enhance knowledge transfer.

Original authors: Benjamin Poole, Andrew Quinn, Li Yang, Minwoo Lee

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Poole, Andrew Quinn, Li Yang, Minwoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a series of video games. First, it learns to play Flappy Bird. Then, you switch it to Catcher, and then to a maze game called Room. The goal is for the robot to get good at all of them without forgetting how to play the first one.

In the world of Artificial Intelligence, this is called Continual Learning. The biggest problem robots face here is "catastrophic forgetting." It's like a student who studies for a math test, then immediately starts studying for a history test, only to completely forget how to do math by the time the history exam is over.

The Old Way: The "Actor" vs. The "Critic"

Most previous research tried to solve this by focusing only on the robot's "muscle memory" (called the Actor). They used a technique called Data Rehearsal, which is like giving the robot a small notebook of notes from its past games to glance at while learning new ones.

However, there was a second part to the robot's brain called the Critic (or Value Function). The Critic is like a coach that constantly evaluates: "How good is this move? How much reward will I get?"

Previous researchers found that if they tried to use the "notebook" (rehearsal) to help the Critic remember old scores, the robot actually got worse at learning. So, they stopped trying to help the Critic and only helped the Actor. The authors of this paper say, "Wait a minute. We are ignoring half the brain!" They wanted to see if they could fix the Critic without breaking it.

The New Idea: Qreg+NWLU

The authors tried a new method called Qreg (Q-value Regularization). This is simply using the notebook to remind the Critic what the scores used to be for old moves.

But they found that the standard way of doing this was messy. It was like trying to study for a test by only reading the last page of your textbook, or waiting until you finished the whole semester before looking at your notes. This led to the robot getting confused or forgetting things quickly.

To fix this, they introduced two simple changes, which they combined into a new method called Qreg+NWLU:

Live Updates (The "Live" Strategy):
- The Problem: In the old method, the robot waited until it finished a whole game level before saving any notes. If the robot made a mistake at the start of the level, those notes were never saved.
- The Fix: Now, the robot writes notes continuously while it plays. It's like a student taking notes in real-time during a lecture rather than trying to remember everything after class is over. This ensures the notes are diverse and cover the whole game, not just the end.
No-Wait (The "No-Wait" Strategy):
- The Problem: The old method said, "Don't look at your old notes until you have finished the first game." This meant the robot started learning the second game with a "cold start," forgetting everything immediately.
- The Fix: As soon as the robot has any notes in its notebook, it starts using them to help learn the new game. It's like a student who starts reviewing their old notes the moment they walk into the new classroom, rather than waiting until the teacher finishes the first lesson.

The "Multi-Cyclic" Challenge

The authors also tested this in a special environment called Multi-Cyclic. Imagine the robot plays Flappy Bird, then Catcher, then Room. But then, the cycle starts over: Flappy Bird again, then Catcher again.

In real life, we often face repeating situations (e.g., a robot vacuum cleaning the same rooms every day, or a stock trader seeing the same market patterns every week). The authors found that most robots fail miserably in this loop; they get worse every time the cycle repeats. Their new method, however, allowed the robot to remember the old games even after going through the cycle multiple times.

The Results

When they tested this new Qreg+NWLU method:

It remembered better: The robot didn't forget how to play the first game when learning the second.
It learned faster: Because it was reviewing notes immediately ("No-Wait"), it didn't waste time re-learning basics.
It was more stable: The robot's performance didn't swing wildly between being a genius and being confused.

The Bottom Line

This paper argues that to teach a robot to learn continuously, we shouldn't just train its "muscles" (the Actor); we must also train its "coach" (the Critic). By giving the coach a notebook that is updated live and reviewed immediately, the robot can learn new tasks without forgetting the old ones, even when the tasks keep repeating.

They didn't claim this fixes every problem in AI, but they proved that for a specific type of learning algorithm (called DQN), this simple combination of "Live" and "No-Wait" is a game-changer for preventing forgetting.

Technical Summary: Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning

Problem Statement

Continual Reinforcement Learning (CRL) aims to enable agents to learn a sequence of tasks while retaining performance on previously learned tasks, addressing the challenges of catastrophic forgetting and plasticity loss. While data rehearsal (storing and replaying past experiences) has emerged as a leading approach in CRL, existing research is predominantly confined to policy gradient frameworks (actor-critic methods). In these settings, regularization via behavioral cloning is applied exclusively to the actor network, as extending regularization to the critic has historically been observed to degrade performance. Consequently, the potential of data rehearsal for value function approximation (e.g., Deep Q-Networks or DQN) remains significantly underexplored.

Furthermore, existing CRL evaluations rarely consider multi-cyclic environments, where task sequences repeat over multiple cycles. This setting is critical for real-world applications where environments naturally reoccur. Multi-cyclic settings have been shown to exacerbate forgetting and plasticity loss, yet they remain underutilized in catastrophic forgetting research, which often limits evaluation to single cycles or at most two cycles.

Methodology

The authors investigate data rehearsal for DQN using Q-value regularization (Qreg), the value-based counterpart to behavioral cloning. They identify that standard Qreg exhibits instability and inconsistency in multi-cyclic settings, often suffering from a "primacy bias" where poor initial learning inhibits subsequent cycles. To address this, they propose Qreg+NWLU (No-Wait Qreg with Live-Updates), which introduces two key modifications to the rehearsal framework:

Live-Updates (Continuous Data Collection): Instead of sampling transitions only at the end of a task (as in prior work), the method incrementally adds rehearsal samples to the Rehearsal Replay Buffer (RRB) at a fixed frequency throughout training. This "Live" strategy reduces selection bias toward transitions occurring near the end of training and promotes sample diversity, particularly when the replay buffer size is small relative to the total training steps.
No-Wait Regularization: Standard Qreg typically defers regularization until the first task is complete and RRB samples are available. The "No-Wait" mechanism applies regularization immediately once rehearsal samples become available, rather than waiting for task completion. This aims to stabilize initial task learning, which is crucial in continual settings where each task's knowledge must be efficiently leveraged for subsequent tasks.
Periodic Q-Value Updates: The method periodically updates the stored Q-values in the RRB corresponding to the current task during training. This prevents the regularization from targeting stale or inaccurate Q-values that may result from insufficient initial training or knowledge gained in subsequent tasks.

The framework utilizes a dedicated RRB separate from the standard replay buffer. The RRB acts as long-term, cross-task memory, while the replay buffer serves as short-term, task-specific memory.

Key Contributions

Investigation of Value-Based Data Rehearsal: The paper investigates the application of data rehearsal with Q-value regularization in multi-cyclic CRL settings, filling a gap in understanding how regularization can enable effective value function learning.
Proposal of Qreg+NWLU: The authors identify limitations in standard Qreg (instability and inconsistency) and propose Qreg+NWLU, a modified approach incorporating Live-Updates and No-Wait regularization.
Empirical Validation: The study demonstrates that Qreg+NWLU consistently outperforms standard CRL baselines (including DQN, DDQN, EWC, L2, PackNet, and MER) across multiple environments and metrics.

Results

Experiments were conducted on three task sequences: Minihack Room (featuring hard transitions and observation space changes), Flappy Bird, and Catcher (using the PyGame Learning Environment). The evaluation spanned four cycles with 10 random seeds.

Performance Metrics: The authors utilized Final Transfer ( $F$ ), Worst Transfer ( $W$ ), and Average Return ( $G$ ).
Room Sequence: Qreg+NWLU achieved near-optimal performance by Cycle 2 with notably low standard error. It significantly mitigated the volatility and primacy bias observed in standard Qreg. While standard Qreg showed high variance and failure in some runs, Qreg+NWLU achieved the highest average return ( $G$ ) and positive transfer scores across all tasks.
Flappy and Catcher Sequences: Qreg+NWLU consistently outperformed baselines, achieving the highest average returns. In Flappy Bird, where task difficulty increases, Qreg+NWLU showed improved performance on all tasks compared to DQN, whereas standard Qreg struggled with Task 1. In Catcher, Qreg+NWLU reached near-optimal performance by Cycle 2 and greatly mitigated the forgetting observed in standard Qreg.
Ablation Studies: The study confirmed that both "Live" and "No-Wait" strategies are critical. Removing "No-Wait" led to high variance and instability similar to standard Qreg. The combination of strategies helped smooth Q-value norms ( $\|Q\|_2$ ) across task transitions, reducing the range of Q-norm values over cycles and enhancing robustness.

Significance and Claims

The paper claims that incorporating data rehearsal into value function approximation via Qreg significantly enhances learning efficiency, transfer capabilities, and resilience to multi-cyclic forgetting. However, the authors emphasize that achieving these results requires the proper framework, specifically the combination of No-Wait regularization, continual sampling (Live), and periodic updating of rehearsal samples.

The authors position this work as a first step towards the broader application of value-based data rehearsal regularization in CRL. They acknowledge limitations, including the exclusion of randomized task sequences and hard transitions involving entirely unique tasks, the reliance on task labels for the Updates strategy, and the focus on purely value-based algorithms (DQN) rather than actor-critic frameworks. Despite these limitations, the results suggest that value-based methods, when properly regularized with data rehearsal, can effectively mitigate catastrophic forgetting in multi-cyclic settings, challenging the previous consensus that such regularization harms performance.