💻 computer science

World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays

This paper introduces Recurrent Generative Replay (REGEN), a continual imitation learning framework that leverages World Action Models to synthesize pseudo-replay trajectories from prior task instructions, thereby significantly reducing catastrophic forgetting in robots without the need to store original human demonstrations.

Original authors: Manish Kumar Govind, Dominick Reilly, Smit Patel, Hieu Le, Srijan Das

Published 2026-06-26

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Manish Kumar Govind, Dominick Reilly, Smit Patel, Hieu Le, Srijan Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that learns by watching humans do tasks, like putting a bowl in a cabinet or pushing a plate. This is called "imitation learning." The problem is, if you teach this robot a new trick today, it often forgets the tricks it learned yesterday. This is called "catastrophic forgetting." It's like a student who studies for a math test and immediately forgets how to read because they are so focused on the new material.

Usually, to stop this forgetting, scientists keep a "notebook" of all the old videos showing the robot how to do the old tasks. They show these videos to the robot while teaching it new things. But in the real world, we often don't have those old videos anymore. Maybe the data was private, or the robot was trained by a company that doesn't share its data.

The Big Idea: The Robot's "Imagination"

This paper introduces a new method called REGEN (Recurrent Generative Replay). Instead of needing a physical notebook of old videos, REGEN gives the robot the ability to imagine its past.

Think of the robot's brain as a powerful movie director (called a World Action Model or WAM). This director doesn't just know what to do (the actions); it also knows what the scene will look like in the future.

Here is how REGEN works, using a simple analogy:

The Prompt: You tell the robot, "Remember the time you put the carrot in the bowl?" (This is the old instruction).
The Seed: You give the robot a single, real photo of the current scene (maybe the robot is now in a kitchen with a carrot).
The Dream: The robot's "director" brain starts to hallucinate (generate) the rest of the movie. It predicts: "Okay, I grab the carrot... now I see the carrot in my hand... now I move it... now I see the bowl..."
The Loop: It keeps feeding its own predictions back into itself, step-by-step, creating a full, fake video of the old task from start to finish.
The Practice: The robot then practices this new task alongside this "imagined" video of the old task. By rehearsing the old task in its imagination, it remembers how to do it without needing the original real video.

What They Found

The researchers tested this in two places: a computer simulation and a real robot arm in a lab.

The Result: The robot using REGEN forgot about 50% less than robots that just learned new things one after another without any help.
The Comparison: It worked almost as well as robots that did have the original real videos, even though REGEN had no access to them.
The Catch: The "imagined" videos aren't perfect. Over a long sequence, the images get a little blurry (like a game of "telephone" where the message gets distorted), and sometimes the robot imagines a successful outcome but the actual movement it predicts wouldn't quite work in real life.

Why This Matters

The paper concludes that while the robot's imagination isn't perfect yet, it's a huge step forward. It means robots can keep learning new skills forever without needing a massive, growing library of old videos. As long as the robot can "dream" up a decent version of the past, it can keep its skills fresh.

In short: A robot that can imagine its past can continue learning its future.

Technical Summary: World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays

1. Problem Formulation

The paper addresses the challenge of continual imitation learning for robotic policies, specifically focusing on catastrophic forgetting. As World Action Models (WAMs) emerge as general-purpose robot policies capable of unifying perception, prediction, and control, they must adapt to new tasks sequentially. However, standard sequential fine-tuning causes the model to overwrite previously learned skills.

Traditional solutions rely on Experience Replay (ER), which stores and replays ground-truth human demonstrations from previous tasks. The authors argue this approach is increasingly impractical in modern robotics, where policies are often trained on large-scale, proprietary datasets that are not publicly released, or where storage of past demonstrations is infeasible. The core problem is: How can a robot retain previously learned skills during continual adaptation without access to the original human demonstrations?

2. Methodology: Recurrent Generative Replay (REGEN)

The authors propose Recurrent Generative Replay (REGEN), a framework that leverages the intrinsic generative capabilities of WAMs to synthesize pseudo-replay trajectories, eliminating the need for stored real demonstrations.

Core Mechanism

WAMs are defined as policies that jointly model future actions ( $\tilde{a}$ ) and future visual observations ( $\tilde{o}$ ) conditioned on language instructions ( $\ell$ ) and current observations ( $o$ ). REGEN exploits this by recursively querying the WAM to generate "hallucinated" trajectories for past tasks.

The generation process for a previous task $T_i$ (given only its language instruction $\ell_i$ ) proceeds in three stages:

Initialization: The rollout is seeded with a real observation sampled from the current task's demonstrations ( $D_k$ ) to provide a valid visual context.
Recurrent Generation: The model predicts an action chunk and a future observation. Crucially, the generated future observation ( $\tilde{o}_t$ ) is fed back as the input observation for the next step, rather than using real data. This recursive loop continues until a termination condition is met.
Termination: Generation stops at a maximum horizon or when the model's internal reward head predicts task completion (e.g., reward > 0.99 for consecutive steps).

Training Objective

The policy is updated via behavioral cloning on a combined dataset $D^+_k = D_k \cup R_k$ , where $D_k$ is the current task's real demonstrations and $R_k$ is the set of synthesized pseudo-trajectories for all previous tasks. The loss function minimizes the difference between the policy's predicted actions and the actions within these pseudo-trajectories, conditioned on the respective task instructions.

3. Key Contributions

The paper outlines three primary contributions:

REGEN Framework: The first continual imitation learning framework that uses the WAM itself as a native generative replay mechanism, synthesizing trajectories without storing real past demonstrations.
Empirical Validation: Demonstration that REGEN substantially mitigates catastrophic forgetting in both simulated (LIBERO benchmarks) and real-world (xArm7 manipulator) settings, while preserving forward transfer to new tasks.
Limitation Analysis: Identification of the specific bottlenecks preventing generated replay from matching real experience replay: (i) long-horizon visual degradation in recursive generation, and (ii) inconsistencies between predicted future observations and the corresponding generated actions.

4. Experimental Results

Simulation (LIBERO Benchmarks)

Experiments were conducted on LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal suites.

Performance: REGEN reduced Negative Backward Transfer (NBT) by over 50% compared to naive Sequential Fine-Tuning (Seq-FT). For example, on LIBERO-Goal, NBT dropped from 100 (Seq-FT) to 44.9 (REGEN).
Comparison: While Experience Replay (ER) using real data remains the upper bound, REGEN significantly outperformed other non-replay baselines (Seq-LoRA, EWC, PackNet) and approached the performance of privileged ER methods.
Representation Stability: Analysis of action representation drift showed that REGEN maintained significantly lower drift (0.12) compared to Seq-FT (0.30), preserving the policy's internal structure.

Real-World Manipulation

Tests were performed on an xArm7 robot with tasks involving placing objects (carrots, eggplants) into bowls or on plates.

Results: REGEN reduced forgetting (NBT) from 96.3 to 60.5 and improved Forward Transfer (FWT) from 50 to 80 compared to Seq-FT.
Significance: This confirms the method's viability beyond simulation, offering a practical approach for real robots where re-collecting past data is impossible.

5. Limitations and Future Directions

The authors explicitly note that REGEN is not yet a complete substitute for real experience replay. The performance gap is attributed to two primary factors:

Visual Fidelity Degradation: As the model recursively generates observations, visual artifacts and blurriness accumulate. This degradation correlates with increased forgetting (NBT), as the supervisory signal weakens over time.
Action-Observation Inconsistency: There is a mismatch between the "imagined" success rate (based on generated visuals) and the "grounded" success rate (based on executing actions). The model often predicts visually plausible successful outcomes while generating actions that are physically insufficient.

The paper concludes that improving the fidelity and consistency of WAM generation is the critical future direction required to close the gap between pseudo-replay and real experience replay.

6. Significance

The paper posits that "A robot that can imagine its past can continue learning its future." By establishing WAMs as a foundation for continual learning without stored demonstrations, REGEN offers a pathway toward lifelong robot learning in scenarios where data privacy or storage constraints prevent the use of traditional replay buffers. The work highlights that the generative capabilities of WAMs can serve as an effective form of implicit memory, provided the limitations of current generative models are addressed.