🤖 machine learning

WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning

WOMBET is a framework that enhances sample-efficient and robust reinforcement learning in robotics by jointly learning a world model to generate high-quality, uncertainty-filtered offline data from a source task and adaptively fine-tuning it on a target task.

Original authors: Mintae Kim, Koushil Sreenath

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mintae Kim, Koushil Sreenath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to perform a complex task, like walking across a rocky terrain or picking up a fragile object. In the real world, you can't just let the robot try, fail, and crash a thousand times to learn. It's too expensive, too dangerous, and takes too long. This is the biggest problem in robotics today: data is expensive.

The paper introduces a new method called WOMBET (World Model-based Experience Transfer) to solve this. Think of WOMBET as a "Smart Simulator" that acts like a flight simulator for pilots, but with a twist: it doesn't just simulate; it curates the best possible practice sessions before the robot ever touches the real world.

Here is how WOMBET works, broken down into simple concepts and analogies:

1. The Problem: The "Blank Slate" vs. The "Bad Teacher"

Standard Online Learning: This is like throwing a robot into a jungle and telling it to figure out how to walk. It will fall down a lot, break things, and learn very slowly.
Standard Offline Learning: This is like giving the robot a video of someone else doing the task. If the video is perfect, the robot learns fast. But if the video is old, from a different environment, or has mistakes, the robot gets confused and fails.
The Gap: Most methods assume you already have a perfect video (dataset). But in reality, you often have to create that video first, and if you aren't careful, the video might be full of lies (hallucinations) because the simulator isn't perfect.

2. The WOMBET Solution: The "Smart Intern"

WOMBET acts like a brilliant, cautious intern who prepares a training manual for the robot. It does this in three main stages:

Stage A: The "Uncertainty-Aware" Simulator (The World Model)

Instead of just guessing how the world works, WOMBET builds a World Model. Imagine this as a crystal ball that predicts what happens if the robot takes a specific action.

The Catch: Crystal balls aren't perfect. Sometimes they are confident but wrong.
The Fix: WOMBET uses a "Penalty System." If the crystal ball is unsure about a prediction (high uncertainty), it adds a huge "penalty" to that path. It's like a teacher saying, "Don't go down that hallway; the floor might be rotten, and I'm not 100% sure." This forces the robot to only plan paths where the simulator is confident.

Stage B: The "Strict Editor" (Dual-Criterion Filtering)

The simulator generates thousands of practice runs (trajectories). But most are garbage. WOMBET acts as a strict editor with two rules for accepting a practice run:

Did it work well? (High Return): The robot must have successfully completed the task in the simulation.
Are we sure it's real? (Low Uncertainty): The simulator must be very confident that this outcome is possible.

The Result: It throws away the "lucky accidents" and the "wild guesses." It keeps only the high-quality, reliable practice sessions. This becomes the robot's "Offline Dataset."

Stage C: The "Adaptive Coach" (Online Fine-Tuning)

Now, the robot starts training in the real world using the practice manual.

The Strategy: At first, the robot relies heavily on the manual (Offline Data) because it's safe and reliable.
The Shift: As the robot starts interacting with the real world, it collects new data (Online Data). WOMBET acts as a smart coach who constantly adjusts the mix: "You're doing great, so let's trust your real-world experience more. But if you start making mistakes, let's look back at the manual."
The Goal: This prevents the robot from forgetting what it learned (stability) while allowing it to adapt to the real world's quirks (flexibility).

3. Why is this a Big Deal? (The Analogy of the Pilot)

Imagine training a pilot:

Old Way: Let the pilot fly a real plane and crash a few times to learn. (Too dangerous).
Bad Simulator Way: Put them in a simulator that glitches and tells them "You can fly through a mountain." They learn to fly into mountains. (Model bias).
WOMBET Way:
1. The simulator runs millions of flights but only records the ones where the physics were clear and the landing was perfect.
2. It throws away any flight where the simulator was "confused" or the landing was shaky.
3. The pilot trains on this curated, perfect dataset first.
4. Then, they get in a real plane. The instructor (WOMBET) watches closely, letting the pilot take over more as they prove they are safe, but stepping in if the pilot tries something the simulator said was risky.

Summary of Benefits

Sample Efficiency: The robot learns much faster because it doesn't waste time on bad guesses or dangerous crashes.
Safety: By filtering out "uncertain" data, the robot avoids dangerous behaviors before it even starts.
Robustness: It handles the difference between the "fake" simulation world and the "real" world much better than previous methods.

In short, WOMBET is a framework that says: "Don't just learn from whatever data you have. Build the best possible training data using a smart, cautious simulator, filter out the lies, and then use that high-quality foundation to teach the robot how to survive in the real world."

1. Problem Statement

Reinforcement Learning (RL) in robotics is often hindered by the high cost and safety risks associated with real-world data collection. While Offline-to-Online RL (pretraining on static data followed by online fine-tuning) offers a solution, existing methods suffer from two main limitations:

Static Data Assumption: Most approaches assume a pre-existing, high-quality offline dataset. They do not address how to generate reliable transfer data when such data is unavailable.
Model Bias and Distributional Shift: Model-Based RL (MBRL) can generate data, but synthetic rollouts often suffer from compounding prediction errors (model bias) and over-optimism. Conversely, standard offline-to-online pipelines struggle with the abrupt distributional shift between the source (pretraining) and target (fine-tuning) tasks, leading to unstable training or poor adaptation.

Core Challenge: How to jointly generate reliable, transferable prior data from a source task and utilize it to accelerate and stabilize learning in a target task without assuming a fixed dataset.

2. Methodology: WOMBET Framework

WOMBET (World Model-based Experience Transfer) is a unified framework that couples uncertainty-aware model-based data generation with adaptive offline-to-online fine-tuning. The process operates in an iterative loop involving three main stages:

A. Uncertainty-Penalized Data Generation (Source Task)

Instead of using a fixed dataset, WOMBET generates an offline dataset ( $D_S$ ) from a source task using a probabilistic ensemble world model ( $\hat{P}$ ).

Uncertainty-Penalized MPC: The agent performs Model Predictive Control (MPC) where the reward is penalized by epistemic uncertainty ( $u(s,a)$ ). The objective is to maximize:
$\tilde{J}_{\hat{P}}(\pi) = \mathbb{E}_{\pi, \hat{P}} \left[ \sum_{t=0}^{H-1} (r(s_t, a_t) - \lambda u(s_t, a_t)) \right]$
This ensures the planner avoids regions where the model is uncertain, maximizing a provable lower bound on the true return.
Dual-Criterion Filtering: Generated trajectories are filtered before being added to $D_S$ $D_{S}$ . A trajectory $\tau$ $τ$ is accepted only if:
1. High Return: $J(\tau) \geq J_{th}$ (ensures task relevance).
2. Low Uncertainty: $\bar{u}(\tau) \leq u_{th}$ (ensures reliability and minimizes model bias).
Relabeling: Accepted source trajectories are relabeled with the target reward function ( $r_T$ ) to align with the target task.

B. Adaptive Offline-to-Online Fine-Tuning (Target Task)

WOMBET performs off-policy policy optimization in the target task using a mixture of the generated offline data ( $D_S$ ) and newly collected online data ( $D_T$ ).

Adaptive Sampling: The training batch is sampled from a mixture distribution $D_{mix} = \alpha D_S + (1-\alpha) D_T$ $D_{mi x} = α D_{S} + (1 - α) D_{T}$ . The mixing coefficient $\alpha$ $α$ is dynamically adjusted based on the critic's Temporal Difference (TD) error:
- High TD error (underfitting/unstable) $\rightarrow$ Increase $\alpha$ (rely more on stable offline data).
- Low TD error (converging) $\rightarrow$ Decrease $\alpha$ (rely more on online data for adaptation).
Robust Critic Training: The critic is trained using an ensemble minimum target to induce pessimism and prevent value overestimation.
Implicit Regularization: To prevent value explosion on Out-of-Distribution (OOD) inputs, WOMBET employs LayerNorm in the critic network to bound activations and an ensemble minimum for Bellman targets. This provides architectural and algorithmic constraints without explicit penalty terms.

C. Iterative Refinement

The world model is continuously updated using the aggregated dataset ( $D = D_S \cup D_T$ ). As the model improves, the uncertainty estimates become more accurate, allowing the generation of a broader yet reliable set of trajectories for the next iteration.

3. Key Contributions

Coupled Data Generation and Transfer: WOMBET is the first framework to jointly optimize the generation of prior data and its utilization for transfer, rather than assuming a fixed dataset. It uses uncertainty-aware planning to construct the dataset iteratively.
Reliable Data Curation: It introduces a dual-criterion filter (high return + low epistemic uncertainty) to curate a compact, high-value, and bias-minimized offline dataset.
Adaptive Transfer Strategy: It proposes an adaptive sampling mechanism that balances bias and variance by dynamically adjusting the ratio of offline to online data based on critic reliability (TD error).
Theoretical Guarantees:
- Proves that uncertainty-penalized planning maximizes a lower bound on the true return.
- Derives a finite-sample error decomposition showing that WOMBET controls both distribution mismatch (via filtering) and approximation error (via adaptive sampling and regularization).

4. Experimental Results

Evaluated on diverse MuJoCo continuous control benchmarks, WOMBET demonstrates superior performance compared to strong baselines:

Sample Efficiency: WOMBET learns significantly faster than online-only baselines (SAC, PPO, TD3), achieving comparable or better final returns with less than half the interaction budget.
Necessity of Fine-tuning: Unlike offline-only methods (BC, IQL, CQL, MOPO) which degrade under distributional shifts, WOMBET's online fine-tuning allows it to adapt to new rewards and dynamics, outperforming zero-shot transfer methods.
Transfer Effectiveness: When compared to other offline-to-online methods initialized with the same dataset, WOMBET achieves higher asymptotic returns. This confirms that the improvement stems from its adaptive transfer mechanism (filtering + adaptive sampling) rather than just the data itself.
Ablation Studies: Removing either the dual-criterion filter or the adaptive sampling rule leads to significant performance drops, confirming that both components are essential for stable and efficient transfer.

5. Significance

WOMBET addresses a critical bottleneck in robotic RL: the lack of reliable, high-quality prior data. By shifting from a "static data" paradigm to a "dynamic data generation and refinement" paradigm, it enables:

Safe Exploration: Uncertainty penalization ensures that synthetic data used for pretraining does not exploit model errors.
Robust Adaptation: The adaptive mixing strategy prevents the instability often seen when transitioning from offline pretraining to online interaction.
Generalizability: The framework provides a principled way to transfer experience across tasks with different rewards or initial states but shared dynamics, making it highly applicable to real-world robotics where data collection is expensive.

In summary, WOMBET unifies the exploitation strength of Model-Based RL with the exploration capabilities of Model-Free RL, offering a robust, sample-efficient solution for experience transfer in complex control tasks.