💻 computer science

HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

The paper proposes HaWMPO, a hallucination-aware world model-based policy optimization framework that enhances generalist robot policies by estimating and suppressing prediction hallucinations during imagined rollouts, thereby significantly improving success rates in complex long-horizon manipulation tasks on both benchmarks and real-world robots.

Original authors: Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang

Published 2026-09-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that can learn to perform a wide variety of tasks, from stacking blocks to folding laundry, have long been a goal of artificial intelligence. To achieve this, researchers have developed "generalist" policies, which are essentially brain-like systems trained to understand language and vision to decide how a robot should move. However, teaching these systems in the real world is slow, expensive, and risky. Every time a robot tries a new action, it might break something or damage itself, making the process of trial and error dangerous. To solve this, scientists have turned to "world models," which are digital simulators that allow robots to practice inside a computer. These models predict what will happen next based on current actions, creating a safe space for learning. Yet, these digital simulators are not perfect; as they predict further into the future, they often start to invent details that never happened, a phenomenon known as "hallucination." If a robot learns from these fake scenarios, it might learn to perform actions that work in the computer but fail completely in the real world.

A team of researchers has developed a new method to help robots learn effectively from these imperfect digital simulations without being misled by their mistakes. They call their approach HaWMPO, a system designed to make robots aware of when their digital training ground is lying to them. Instead of treating every imagined scenario as equally valuable, the new system includes a special component that acts like a quality control inspector. This inspector looks at the sequence of images the simulator generates and assigns a score indicating how reliable that sequence is. If the simulator starts to drift into fantasy, creating images that don't match the laws of physics or the expected outcome, the inspector flags it. The system then uses this information to adjust the learning process, effectively telling the robot to ignore the parts of the simulation that seem too unreliable and focus on the parts that look realistic.

The researchers tested this method using a standard set of robotic tasks in a computer environment called LIBERO, which involves moving objects to specific locations. They compared their new system against other methods that use world models but lack this quality control feature. The results showed a clear advantage for the new approach. In the simulation, the robot using the hallucination-aware system achieved a success rate of 63.7 percent, which was significantly higher than the next best method. The improvement was particularly noticeable in tasks requiring spatial reasoning and object manipulation, where the robot's ability to distinguish between real and fake scenarios helped it learn more robust strategies. The researchers found that by penalizing the robot when it relied on highly hallucinated scenarios, the system prevented the robot from learning bad habits that would only work in a broken simulation.

To ensure this method worked outside the computer, the team also tested it on a physical robot in a real-world setting. They tasked the robot with two specific challenges: placing a tissue into a box and putting headphones onto a stand. Without the new system, the robot succeeded in these tasks about 60 percent of the time on the headphone task. After applying the hallucination-aware training, the success rate rose to an AUC of 0.8 on average, with the robot achieving 85% on the tissue task and 75% on the headphone task. This jump in performance demonstrates that the lessons learned in the digital world were actually transferable to the physical world, because the robot had learned to ignore the digital noise that usually confuses these systems. The researchers noted that the system works by constantly checking the consistency of the simulated future, ensuring that the robot only learns from trajectories that look physically plausible.

The study highlights that the key to better robot learning is not just having a simulator, but having a way to trust it. By building a system that can detect when a simulation is losing its grip on reality, the researchers have created a more stable path for robots to improve. While the current tests were conducted on relatively short tasks, the success suggests that this approach could be vital for more complex, long-term missions where a robot must plan many steps ahead. The work confirms that for robots to become truly general-purpose helpers, they must learn to distinguish between a useful prediction and a convincing lie, a skill that this new method provides by making the learning process itself aware of its own limitations.

Technical Summary: HaWMPO

Problem Statement

Generalist robot policies, particularly Vision-Language-Action (VLA) models, have demonstrated strong multi-task generalization but struggle with success rates in complex, long-horizon scenarios. While online reinforcement learning (RL) on physical robots can improve these policies, it is hindered by high sample inefficiency, costly physical interactions, and significant safety risks.

World models offer a promising alternative by enabling policy optimization through "imagined" rollouts in a virtual space. However, generative world models are prone to accumulated prediction errors and hallucinations during long-horizon rollouts. These hallucinations produce biased state transitions and false learning signals. If a VLA policy is trained directly on these hallucinated trajectories, it may exploit world-model biases rather than learning actions effective in real environments, leading to unstable policy updates and degraded performance.

Methodology: HaWMPO

The authors propose Hallucination-Aware World Model-based Policy Optimization (HaWMPO), a closed-loop RL pipeline designed to mitigate the negative effects of world model hallucinations during VLA post-training. The framework consists of three core components and a novel optimization mechanism:

1. Action-Conditioned Hallucination-Aware Model (HAM)

To address the issue of unreliable rollouts, HaWMPO introduces a dedicated model to estimate the reliability of generated image sequences.

Input: The HAM takes an eight-frame predicted video chunk ( $\hat{V}_c$ ), the conditioning actions ( $a_c$ ), the current observation ( $I_c$ ), and an anchor image ( $I_0$ ) as inputs.
Architecture: It utilizes a frozen DINOv3 encoder for visual features, Context and Action Projectors to map inputs to a shared space, and a Fusion Transformer (incorporating cross-attention and self-attention) to aggregate representations. A learnable query and MLP head output a continuous hallucination score ( $\hat{h}_c$ ).
Supervision: The model is trained using a composite target derived from four metrics: DINOv3 similarity, depth consistency, optical-flow-based trajectory consistency, and MUSIQ image quality. Higher scores indicate more severe hallucinations.

2. Reward-Soft Mechanism

Instead of treating all world-model-generated rollouts equally, HaWMPO weights their contributions based on the predicted hallucination level.

Mechanism: The raw reward ( $R$ ) provided by a reward model for a specific action chunk is modulated by the average hallucination score ( $H$ ) of that chunk.
Formula: $\hat{R} = (1 - \alpha \cdot H) \cdot R$ , where $\alpha$ is a tunable hyperparameter.
Effect: This mechanism imposes penalties on seemingly successful RL steps if they occur within highly hallucinated trajectories, preventing the policy from learning from biased signals.

3. Policy Optimization Pipeline

The system integrates the VLA policy, the action-conditioned world model, the HAM, and the reward model into a Group Relative Policy Optimization (GRPO) framework.

Process: The VLA agent executes rollouts in parallel imaginative environments constructed by the world model.
Training: For each action chunk, the HAM predicts a hallucination score, and the Reward-Soft mechanism modulates the reward. The GRPO algorithm then updates the VLA policy using these hallucination-aware rewards, calculating group-wise normalized advantages to stabilize learning.

Key Contributions

Closed-Loop RL Pipeline: The authors establish a pipeline integrating a world model, a hallucination-aware model, and a reward model, enabling online-style policy optimization without frequent real-world robot interactions.
Quantitative Reliability Estimation: They design an action-conditioned HAM that produces a continuous hallucination score, explicitly quantifying the reliability of world model predictions under specific action inputs.
Reward-Soft Mechanism: They propose a novel method to integrate hallucination scores into the GRPO optimization, penalizing severely hallucinated action segments to mitigate the adverse bias of world models on policy updates.

Experimental Results

Simulation (LIBERO Benchmark)

Evaluated on the LIBERO Object, Spatial, and Goal task suites:

Performance: HaWMPO achieved the highest average success rate of 63.7%.
Comparisons: This represents a 15.0% improvement over the base OpenVLA-OFT model, a 6.9% improvement over WMPO, and a 2.8% improvement over the strongest baseline, WoVR*.
Specific Gains: On the Spatial suite, HaWMPO reached 77.2% (surpassing WoVR* by 8.0 points), and on the Object suite, it reached 52.2% (surpassing WoVR* by 2.6 points).

Real-World Experiments (G1 Robot)

Experiments were conducted on two manipulation tasks: "Tissue-to-Box" and "Headphone-on-Stand."

Results: HaWMPO achieved an average success rate of 80.0% (85% on Tissue-to-Box, 75% on Headphone-on-Stand).
Improvement: This exceeds the base policy by 12.5 percentage points and the WoVR baseline by 7.5 percentage points.

Ablation and Analysis

Hallucination Detection: The HAM achieved an AUROC of 0.9375 in detecting hallucinated chunks against human annotations, outperforming single-metric baselines (e.g., DINO-only at 0.9062).
Reward-Soft Impact: An ablation study confirmed that removing the HAM and Reward-Soft mechanism led to significant performance drops, particularly on the Spatial suite (an 8.0 to 11.6 point drop at various training steps).
Hyperparameter Sensitivity: The optimal penalty strength ( $\alpha$ ) was found to be task-dependent, with $\alpha=0.3$ performing best for Object and Spatial, and $\alpha=1$ for Goal.

Significance and Claims

The paper claims that HaWMPO addresses a critical bottleneck in world model-based RL: the instability caused by prediction hallucinations. By explicitly modeling these hallucinations and down-weighting unreliable synthetic trajectories, the method enables more robust policy optimization. The authors assert that their approach allows VLA policies to leverage the efficiency of world models while avoiding the pitfalls of learning from biased, hallucinated data, resulting in superior performance in both simulation and real-world physical manipulation tasks.

Limitations

The authors acknowledge several limitations:

The HAM relies on manually designed supervision signals (feature discrepancy, depth, trajectory) which may not capture all types of hallucinations.
Evaluation is currently limited to relatively short-horizon tasks in the LIBERO benchmark; scalability to complex, long-horizon real-world scenarios is unexplored.
The hallucination-aware model and world model are trained separately, which may limit system adaptability during online RL. Future work is suggested for end-to-end joint optimization.