🤖 machine learning

Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data

This paper proposes a method to enhance the sample efficiency of online reinforcement learning by guiding world models with abundant, non-curated offline data through experience rehearsal and execution guidance, thereby overcoming distributional shift issues and significantly outperforming baselines across diverse visuomotor tasks.

Original authors: Yi Zhao, Aidan Scannell, Wenshuai Zhao, Yuxin Hou, Tianyu Cui, Le Chen, Dieter Büchler, Arno Solin, Juho Kannala, Joni Pajarinen

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Zhao, Aidan Scannell, Wenshuai Zhao, Yuxin Hou, Tianyu Cui, Le Chen, Dieter Büchler, Arno Solin, Juho Kannala, Joni Pajarinen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to walk, run, or pick up a coffee cup. Traditionally, you'd have to sit there and watch the robot, telling it every single time it did something right or wrong. This is slow, expensive, and requires a lot of human effort to label the data.

This paper introduces a new method called NCRL (Non-curated offline data for efficient Reinforcement Learning) that teaches robots much faster by using a "trash-to-treasure" approach. Here is how it works, broken down into simple concepts:

1. The Problem: The "Garbage" Data Dilemma

Usually, to train a robot, you need a clean, organized dataset where every action is labeled with a score (e.g., "Good job!" or "Bad job!"). But in the real world, we have mountains of messy, unorganized data.

It has no scores (reward-free).
It's a mix of great moves and terrible mistakes (mixed quality).
It comes from different types of robots (multi-embodiment).

Think of this like a library filled with millions of books, but none of them have titles, summaries, or ratings. Most people would ignore this library because it's too hard to use. Previous methods tried to use this data, but they often failed because they assumed the data was neat and labeled.

2. The Solution: Building a "Mental Map" First

Instead of trying to learn the specific task immediately, the NCRL method first builds a World Model.

The Analogy: Imagine a student who wants to learn to drive a car. Instead of getting in a car and trying to drive to the grocery store immediately, they first spend months watching all kinds of driving videos—some from race cars, some from trucks, some from people driving badly, and some from experts. They don't know the destination yet; they just learn how the car moves, how the road looks, and how steering works.
The Paper's Claim: The authors trained a single "brain" (the World Model) on this messy, unlabelled data from 6 different types of robots. This brain learned the general physics of movement without needing to know what the final goal was.

3. The Trap: Why "Just Fine-Tuning" Fails

The researchers found that if you take this pre-trained brain and immediately try to teach it a specific new task (like "Walk to the door"), it often fails.

The Analogy: It's like taking a student who studied general driving videos and immediately putting them in a race car on a track they've never seen. The student panics because the specific track looks different from the videos they watched. The "mental map" they built doesn't match the reality of the new task.
The Paper's Finding: They discovered this happens because of a distributional shift. The messy data used for training looks very different from the specific data the robot encounters when it starts trying to solve the new task.

4. The Fix: Two Special Techniques

To fix this mismatch, the authors added two "training wheels" to help the robot bridge the gap between the messy past and the specific present.

Technique A: Experience Rehearsal (The "Flashcard" Method)

How it works: When the robot starts learning the new task, it doesn't just look at its new, tiny set of experiences. It goes back to its massive library of messy data and searches for the specific clips that look like the current situation.
The Analogy: Imagine the robot is stuck on a specific turn. Instead of just guessing, it pulls up a "flashcard" from its library of old videos that shows a similar turn, even if that old video was from a different robot or a different day. It "rehearses" these relevant memories to remind itself how things work, preventing it from forgetting what it learned earlier.

Technique B: Execution Guidance (The "Co-Pilot" Method)

How it works: The robot has a "Co-Pilot" (a policy trained on the messy data) that knows how to move safely, even if it's not perfect. During the early stages of learning, the robot alternates between trying its own new ideas and letting the Co-Pilot take the wheel for a few seconds.
The Analogy: Think of a new driver with a driving instructor in the passenger seat. The new driver tries to steer, but if they start to drift, the instructor gently takes over for a moment to keep the car on the road, then hands control back. This keeps the robot from wandering into "dead ends" where it can't learn anything, guiding it toward areas where it is more likely to succeed.

5. The Results: Twice as Fast

The paper tested this on 72 different tasks, ranging from a cheetah running to a robot arm stacking blocks.

The Claim: Using this method, the robots learned twice as fast as robots starting from scratch (learning without any prior data).
The Comparison: Even when compared to other methods that tried to use offline data, NCRL won by a significant margin, especially on hard tasks where the robot has to explore and figure things out on its own.

Summary

The paper argues that we don't need perfect, labeled data to teach robots. We can use the "messy" data we already have (videos of robots moving around without scores) if we:

Build a general understanding of the world first.
Use a "search" system to find relevant old memories when learning new tasks (Experience Rehearsal).
Use a "Co-Pilot" to guide the robot so it doesn't get lost in the beginning (Execution Guidance).

This allows robots to learn complex skills with far fewer attempts, making AI training much more efficient.

Technical Summary: Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data

Problem Statement
Leveraging offline data is a promising avenue for improving the sample efficiency of online Reinforcement Learning (RL). However, existing approaches predominantly rely on curated offline datasets that are labeled with rewards and often restricted to specific tasks or embodiments. Acquiring such data is expensive and laborious, particularly for robotic manipulation where retrospective reward annotation is required. While recent works have explored utilizing abundant "non-curated" data—characterized by being reward-free, of mixed quality, and collected across multiple embodiments—naive application of these datasets has proven insufficient. Specifically, pre-training a world model on such data and then naively fine-tuning it for online RL often fails to accelerate training. The authors identify the root cause of this failure as a distributional shift between the offline data used for pre-training and the online data collected during the fine-tuning phase. When the offline distribution does not sufficiently cover the downstream task's data distribution, the pre-trained world model struggles to benefit policy learning, leading to catastrophic forgetting or degraded model accuracy.

Methodology: NCRL
The paper proposes NCRL (Non-curated offline data for efficient RL), a two-stage pipeline designed to effectively utilize non-curated, reward-free, multi-embodiment data.

Multi-Embodiment World Model Pre-Training:
Instead of training separate models per task, NCRL pre-trains a single, task-agnostic world model on a large-scale non-curated dataset. The architecture is based on the Recurrent State Space Model (RSSM) with specific modifications to handle diverse inputs:
- Removal of task-specific losses.
- Zero-padding of actions to unify dimensions across different embodiments.
- Scaling to 280M parameters.
  The model is optimized using standard prediction, dynamics, and representation losses on the raw offline trajectories, without reward labels.
RL-Based Fine-Tuning with Rehearsal and Guidance:
To address the distributional shift and effectively transfer the pre-trained knowledge, NCRL introduces two key techniques during the online fine-tuning stage:
- Experience Rehearsal: Rather than discarding the offline data after pre-training, NCRL retrieves task-relevant trajectories from the non-curated dataset during online training. This is achieved by computing the neural feature distance between initial observations in the online buffer and offline trajectories using the pre-trained encoder. These retrieved trajectories are replayed to:
  - Prevent catastrophic forgetting by continuing to train the world model on relevant pre-training data.
  - Augment the initial state distribution ( $p_0(s)$ ) for model rollouts, enabling the generation of promising trajectories for policy updates.
  - Serve as a regularizer against distributional shift.
- Execution Guidance: To guide exploration toward regions where the world model has high confidence (and where offline data exists), NCRL trains a prior policy ( $\pi_{bc}$ ) via behavioral cloning on the retrieved offline data. During online data collection, the agent probabilistically alternates between the RL policy ( $\pi_\phi$ ) and the prior policy ( $\pi_{bc}$ ). This mixed-policy approach steers the agent toward high-quality regions of the state space early in training, mitigating the risks of hard exploration.

Key Contributions
The paper outlines four primary contributions:

C1: It proposes a more realistic setting for leveraging offline data, utilizing reward-free, mixed-quality, and multi-embodiment datasets, moving beyond the constraints of curated, reward-labeled data.
C2: It demonstrates that naive world model fine-tuning fails on many tasks due to the distributional shift between pre-training and fine-tuning data.
C3: It introduces experience rehearsal and execution guidance as specific techniques to mitigate this distributional gap and encourage exploration.
C4: It presents the NCRL framework, which integrates non-curated data in both pre-training and fine-tuning stages, demonstrating superior performance across a diverse set of tasks compared to existing approaches.

Experimental Results
The authors evaluate NCRL on 72 visuomotor control tasks spanning locomotion and robotic manipulation across 6 embodiments (using DMControl and Meta-World benchmarks). The offline dataset comprises 60k trajectories (10M state-action pairs) collected from unsupervised agents and noisy demonstrations.

Sample Efficiency: Under a limited sample budget of 150k online samples, NCRL achieves nearly twice the aggregate score of learning-from-scratch baselines (DrQ-v2 and DreamerV3). It matches the performance of these baselines trained with 3.3x to 6.7x more samples.
Comparison with Offline Methods: NCRL significantly outperforms state-of-the-art methods that leverage offline data, including R3M (visual pre-training), UDS-RLPD, ExPLORe, and JSRL-BC. Notably, while methods like JSRL-BC struggle with non-expert trajectories, NCRL effectively utilizes exploratory and mixed-quality data.
Comparison with Model-Based Methods: NCRL outperforms other world model pre-training approaches (e.g., iVideoGPT, pre-trained DreamerV3) that rely on novel architectures but lack mechanisms to reuse offline data during fine-tuning. Crucially, NCRL achieves this without relying on tricks like reward shaping or expert replay buffer initialization used by some baselines.
Task Adaptation: In a continual learning setting, NCRL enables agents to adapt to new tasks (e.g., Ant robot tasks) significantly faster than the PackNet baseline, achieving adaptation within 100 trials per task.

Significance and Claims
The paper claims that the primary significance of NCRL lies in its ability to unlock the potential of abundant, uncurated data for RL. By identifying distributional shift as the bottleneck for naive fine-tuning and addressing it through experience rehearsal and execution guidance, the authors demonstrate that world models pre-trained on non-curated data can substantially boost sample efficiency. The work suggests that the rich information contained in non-curated datasets (dynamics, informative states, action priors) is not fully leveraged by current methods that discard this data after pre-training or fail to align it with online objectives. NCRL provides a practical pipeline to bridge this gap, enabling efficient RL across a broader range of tasks and embodiments without the prohibitive cost of reward annotation.