💻 computer science

MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation

MoWM proposes a mixture-of-world-model framework that enhances embodied action planning by fusing motion-aware latent representations with fine-grained pixel-space features to capture both compact dynamics and action-relevant visual details.

Original authors: Yangcheng Yu, Xin Jin, Yu Shang, Xin Zhang, Haisheng Su, Wei Wu, Yong Li

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yangcheng Yu, Xin Jin, Yu Shang, Xin Zhang, Haisheng Su, Wei Wu, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fold a T-shirt or pick up a specific block. To do this, the robot needs two things: super-sharp vision (to see exactly where the edges of the cloth are) and a sense of movement (to understand how things shift and flow when touched).

The researchers found that current robots usually struggle because they pick one "eye" style but ignore the other. This paper introduces MoWM, a way to give robots "hybrid vision."

Here is the breakdown using a simple analogy.

The Problem: The "Photographer" vs. The "Director"

Imagine you are trying to learn a complex dance routine by watching two different types of videos:

The Photographer (Pixel-based World Models): This video is ultra-high definition. You can see every single pore on the dancer's skin and every thread on their clothes. It’s beautiful, but it’s too much information. You spend so much time looking at the background wallpaper or the texture of the floor that you actually lose track of the dancer's actual movement. You get "lost in the details."
The Director (Latent-based World Models): This video is like a blurry, abstract sketch. You can’t see the dancer's face, but you can see the flow of the movement perfectly. You understand the rhythm and the direction, but you can't see exactly where the dancer's hand is touching the floor. You have the "vibe," but you lack the precision.

Current robots usually pick one: They either get distracted by the "wallpaper" (the background) or they are too "blurry" to perform delicate tasks.

The Solution: MoWM (The "Smart Overlay")

The researchers created MoWM (Mixture-of-World-Models). Instead of making the robot choose, they allow the robot to use both at the same time through a process called Feature Modulation.

Think of it like using Augmented Reality (AR) glasses:

The robot looks through the Photographer’s eyes to see the high-definition world.
But, it wears Director’s AR glasses that overlay "motion highlights" onto the view.

When the robot sees a piece of cloth, the "Director" part of its brain highlights the path the cloth will take when moved, while the "Photographer" part tells it exactly where the hem of the shirt is. By blending these two, the robot ignores the "noise" (the static background) and focuses only on the "signal" (the parts of the image that actually matter for the task).

Does it work?

The researchers tested this in two ways:

In a Digital Playground (CALVIN): They gave the robot long, complex sequences of tasks (like a scavenger hunt). MoWM beat almost all other methods, especially in long tasks where robots usually "lose their way."
In the Real World: They gave a real robot arm a very difficult job: folding a T-shirt into a neat rectangle. Because the robot could see the fine details of the fabric while understanding the "flow" of the folding motion, it succeeded.

The "Too Long; Didn't Read" Summary

Most robots are either too focused on tiny details (and miss the big picture) or too focused on the big picture (and miss the tiny details). MoWM acts like a smart filter that combines high-definition vision with a sense of motion, allowing robots to perform complex, delicate tasks with much higher success.

Technical Summary: MoWM (Mixture-of-World-Models)

MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation

1. Problem Statement

Embodied action planning requires robots to translate visual observations and language instructions into precise executable actions. Current research follows two main world-model paradigms, both of which have inherent flaws:

Pixel-based World Models (e.g., Video Diffusion): These models excel at preserving fine-grained visual details and appearance. However, they suffer from visual redundancy, encoding massive amounts of task-irrelevant information (like static backgrounds) that can obscure the motion cues necessary for accurate action decoding.
Latent-based World Models: These models operate in a compressed feature space, making them highly motion-aware and efficient at capturing temporal dynamics. However, the high level of compression often suppresses fine-grained spatial details, which are critical for precise manipulation and object interaction.

The core challenge is to create a representation that is simultaneously spatially precise (for detail) and motion-aware (for control).

2. Methodology

The authors propose MoWM, a hybrid framework that fuses representations from two complementary world models. The architecture is divided into two distinct stages:

Stage 1: Independent Pre-training of Hybrid World Models

Pixel-space World Model ( $G$ ): Based on Stable Video Diffusion (SVD), this model is fine-tuned on embodied data to predict future video sequences conditioned on text instructions. It captures high-resolution, low-level visual details.
Latent-space World Model ( $F$ ): Utilizing a transformer-based architecture with a V-JEPA encoder, this model learns to predict future state tokens in a compressed latent space. It focuses on global temporal dynamics and motion patterns.

Stage 2: End-to-End Action Planning via Feature Modulation

Feature Extraction: The pixel model provides multi-scale feature tensors ( $\Phi_{pixel}$ ) from various U-Net layers. The latent model provides a sequence of latent tokens ( $\Phi_{latent}$ ).
Feature Fusion: Both feature streams are projected to a shared embedding dimension. The model uses a concatenation-based fusion mechanism followed by a linear projection to create a fused representation ( $\Phi_{fused}$ ).
Residual Mechanism: To ensure spatial fidelity is not lost, a learnable gating mechanism combines the fused features with the original pixel features: $\Phi_{final} = W_{gate}\Phi_{fused} + \Phi'_{pixel}$ .
Action Decoding: The final fused feature ( $\Phi_{final}$ ) serves as the condition for a Diffusion Policy decoder, which iteratively denoises an action vector to produce the final robot commands.

3. Key Contributions

Hybrid Architecture: Introduced a novel way to combine the "what" (pixel-level detail) with the "how" (latent-level motion) in embodied AI.
Latent-to-Pixel Modulation: Developed a fusion strategy that uses motion-aware latent features to "guide" or modulate pixel features, effectively filtering out task-irrelevant visual noise.
Comprehensive Analysis: Provided an in-depth study of the inductive biases of pixel vs. latent spaces, proving that latent features help the model focus on action-centric information.

4. Results

SOTA Performance on CALVIN: MoWM achieved state-of-the-art results on the CALVIN benchmark, outperforming VLA-based (e.g., Pi0, OpenVLA) and world-model-based (e.g., VPP) baselines. Notably, it showed a 12.7% improvement in success rates for long-horizon (5th stage) tasks.
Real-World Validation: The model was successfully deployed on an AgileX robot platform to perform a complex, long-horizon cloth-folding task, demonstrating its ability to generalize from simulation to real-world physics.
Ablation Success: Experiments confirmed that concatenation-based fusion outperformed cross-attention and pixel-only approaches, proving that injecting latent dynamics into pixel representations is highly effective for action decoding.

5. Significance

This work shifts the focus of embodied planning from "generating perfect pixels" to "generating optimal features for control." By demonstrating that a mixture of different representation spaces can overcome the limitations of individual models, MoWM provides a blueprint for future large-scale embodied agents. It suggests that the path to robust, long-horizon robotic manipulation lies in the intelligent integration of high-fidelity visual perception and abstract physical reasoning.