🤖 machine learning

WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling

This paper introduces WorldBagel, a unified Vision-Language-Action-World modeling framework built on the BAGEL architecture, which demonstrates that unifying multimodal generation and world modeling leads to superior performance and more structured action representations across diverse robotic manipulation tasks compared to task-specific alternatives.

Original authors: Zelin Zhao, Min Shi, Bo Yuan, Haotian Xue, Jialuo Li, Lama Moukheiber, Humphrey Shi, Yongxin Chen

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zelin Zhao, Min Shi, Bo Yuan, Haotian Xue, Jialuo Li, Lama Moukheiber, Humphrey Shi, Yongxin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook a meal. Most current robots are like chefs who only look at the recipe (language) and the ingredients (vision) to decide what to do next. They are great at following instructions, but they don't really "understand" how the kitchen works. They don't intuitively know that if you push a pot too hard, it will slide off the counter, or that steam will rise from a boiling pan.

WorldBagel is a new kind of robot brain that changes this. Instead of just being a chef who follows orders, it acts like a simulator inside the robot's head. It doesn't just decide what to do; it constantly imagines what the world will look like after it does something.

Here is a simple breakdown of how it works, using the paper's own claims:

1. The "Two-Tower" Brain

Think of the robot's brain as having two specialized experts working in the same office:

The "Understanding" Expert (UND): This expert is great at looking at a picture and reading a sentence to figure out what the robot should do right now. It's the "doer."
The "Generation" Expert (GEN): This expert is a dreamer. It looks at the current scene and the robot's intended action, then asks, "If I do this, what will the kitchen look like in one second?" It predicts the future.

WorldBagel forces these two experts to share the same notebook. They don't just work side-by-side; they learn from each other. By trying to predict the future, the robot gets better at understanding the present.

2. The Secret Sauce: "Fourier" Actions

Robots move in smooth, continuous ways (like a hand gliding through the air), but computers usually think in steps or blocks. The paper introduces a clever trick called Fourier Feature Tokenization.

Imagine trying to describe a smooth wave. You could try to draw it with jagged Lego blocks (discretization), but it would look bumpy and inaccurate. Or, you could describe the wave using a specific set of musical notes (frequencies) that, when played together, recreate the smooth wave perfectly.

WorldBagel uses this "musical note" approach for robot movements. It translates the robot's smooth physical actions into a structured language of frequencies. This allows the robot to:

Predict its own movements with much higher precision.
Understand the "shape" of a task better than other methods.

3. The "Sequence Plan" (The Storyteller)

To teach this robot, the researchers didn't just feed it random data. They organized the learning like a storybook with a specific structure called a Sequence Plan.

Imagine a comic strip where the panels are mixed up. Sometimes the robot sees the picture, then the instruction, then the action, then the result. Other times, it sees the result first to learn what caused it. WorldBagel shuffles these "panels" (visuals, words, actions, and future predictions) in different orders during training. This helps the robot learn that:

Actions cause changes in the world.
Language guides those actions.
The future is a logical result of the past.

4. What Did They Find?

The researchers tested WorldBagel on three different "kitchens" (robotic environments):

LIBERO: A complex simulation with many different tasks.
Language Table: A real-world table where a robot pushes blocks based on spoken commands.
Franka: A physics-heavy simulation with a specific robot arm.

The Results:

Better at Everything: WorldBagel was better at completing tasks than robots that only focused on doing the action or only focused on predicting the future. It was the "jack of all trades" that mastered them all.
Sharper Predictions: When asked to guess what the next frame of video would look like, WorldBagel was much more accurate than its competitors.
Sturdier: When the researchers added "noise" (like shaking the robot's hand or changing the speed of its movements), WorldBagel didn't get confused. It kept predicting the future correctly because it understood the physics of the movement, not just the visual pattern.

The Bottom Line

The paper argues that by combining Vision (seeing), Language (reading), Action (doing), and World Modeling (imagining the future) into one single, unified system, robots learn faster and become more reliable.

WorldBagel proves that if you teach a robot to imagine the consequences of its actions, it becomes a much smarter and more capable worker. The code and the model are set to be released so others can build on this "unified" approach.

Technical Summary: WorldBagel

Problem Statement

World models aim to capture environment dynamics to support perception, reasoning, and decision-making, serving as a central objective in embodied intelligence and robotics. While recent Vision-Language-Action (VLA) models have demonstrated success in multi-task learning and generalization by incorporating language, they typically emphasize action prediction and task performance rather than the explicit generative modeling of environment dynamics. Conversely, traditional world modeling efforts often focus on visual dynamics and control without deeply integrating language or large-scale multimodal knowledge.

The paper argues that a natural convergence exists between multimodal VLA learning and world modeling. A unified model should not only understand visual observations and follow language instructions but also generate future visual states and predict actions within a shared representation space. However, it remains unclear how unified multimodal models should be adapted or evaluated as general Vision-Language-Action-World (VLAW) models, particularly regarding action tokenization, sequence planning, and training stability across heterogeneous datasets.

Methodology: WorldBagel

WorldBagel is a unified VLAW framework built upon BAGEL, a modern multimodal unified model featuring a two-tower architecture with dedicated Understanding (UND) and Generation (GEN) experts. WorldBagel extends this architecture to jointly support multimodal understanding, structured action modeling, and future world prediction.

1. Model Architecture

The framework operates over a shared multimodal token embedding space and consists of two primary branches:

VLA Branch (Understanding & Action Prediction): Visual states are encoded via a ViT encoder and processed through the UND expert, which specializes in cross-modal grounding. A specialized Fourier Feature Action Decoder (FFAD) is attached to predict continuous control signals.
World Modeling Branch (Action-Conditioned Generation): This branch conditions on past visual tokens, language instructions, and structured action tokens. It utilizes the GEN expert to predict future observations via flow matching, followed by a VAE decoder.

2. Key Technical Components

Fourier Feature Action Tokenizer (FFAT) & Decoder (FFAD): Instead of standard regression or discrete binning (which can be unstable or data-dependent), WorldBagel maps continuous action vectors into structured Fourier feature embeddings.
- Forward Mapping: Actions are expanded into frequency bands with exponentially increasing scales to capture both coarse and fine-grained variations.
- Prediction: The model predicts Fourier features in the latent space rather than raw action values, improving stability.
- Inverse Mapping: Actions are reconstructed via phase-consistent averaging across frequencies, ensuring continuity and local metric structure without iterative optimization.
Interleaved VLAW Modeling via Sequence Plans: The model organizes visual observations, language instructions, action tokens, and future state predictions into a unified autoregressive sequence. Flexible "sequence plans" define the ordering of these tokens, allowing the model to be trained on various objectives (e.g., predicting actions before future observations or vice versa) within a single forward pass.
LLM-Inspired Data Sampling: To handle heterogeneous datasets with varying scales and supervision signals, the authors employ:
- Mixture Dataset Sampling: Smoothed sampling probabilities based on dataset size to prevent large datasets from dominating.
- Priority Sequence-Plan Sampling: A strategy that assigns higher weights to "joint VLAW plans" (supervising both action and world prediction) compared to policy-only plans, ensuring the model frequently observes supervision for both control and environment dynamics.

3. Training Objective

The model is initialized from a pretrained BAGEL checkpoint and fine-tuned using Supervised Fine-Tuning (SFT). The total loss combines action prediction loss (regression in Fourier space) and vision generation loss (flow matching for future frames), weighted as:
$\mathcal{L} = \mathcal{L}_{action} + 0.1\mathcal{L}_{vision}$

Key Contributions

Unified VLAW Framework: The introduction of WorldBagel, which extends the BAGEL two-tower architecture to jointly model perception, language grounding, action prediction, and future observation generation.
Fourier Feature Action Modules: The proposal of FFAT and FFAD, which encode continuous control signals into structured tokens aligned with the multimodal space, improving action representation quality and stability.
Systematic Study of Unification: Through multi-task robotic manipulation and cross-domain experiments, the paper demonstrates that unification itself benefits learning effective VLAW models, yielding consistent empirical gains and deeper insights into multimodal world modeling.

Experimental Results

The authors evaluated WorldBagel on three benchmarks: LIBERO (simulated long-horizon manipulation), Language Table (real-world tabletop manipulation), and Franka (physics-based simulation).

Multi-task Performance: WorldBagel achieved an average success rate of 98.0% on the LIBERO benchmark, outperforming the strong baseline RynnVLA-002 (+0.6%) and various other VLA baselines (e.g., OpenVLA-OFT, $\pi0.5$ ).
World Modeling Quality: In action-conditioned world modeling tasks, WorldBagel consistently improved prediction fidelity (lower FVD, higher PSNR/SSIM, lower LPIPS) compared to baselines across all datasets. For instance, on LIBERO, it reduced FVD from 389.5 to 373.1.
Action Representation Quality: The Fourier-based approach (FFAD) reduced Action MSE to 0.028 and achieved a linear probe accuracy of 68.7% for task identity, significantly outperforming regression (49.8%) and other tokenization methods. This indicates the learned embeddings are more structured and semantically aligned.
Stability Under Distribution Shifts: Under perturbations (Gaussian noise, action scaling, temporal shifts), WorldBagel maintained higher prediction fidelity (PSNR) and lower LPIPS compared to RynnVLA-002. Eigenvalue spectrum analysis revealed that WorldBagel possesses a higher effective rank (33.7 vs. 16.3) and a lower dominant eigenvalue ratio, suggesting a richer and more stable representation of action dynamics.

Significance and Claims

The paper claims that unification itself is a key ingredient for scalable world models. By jointly modeling perception, language, action, and future dynamics within a single autoregressive framework, WorldBagel demonstrates that unified architectures can outperform task-specific alternatives.

The authors argue that their work provides:

Empirical Gains: Consistent improvements in both control performance (policy success) and predictive environment understanding (world modeling fidelity).
Methodological Insights: Evidence that Fourier-based action tokenization and interleaved sequence planning are critical for stabilizing heterogeneous multimodal training and learning high-quality, structured action representations.
Foundation for Future Work: The results suggest that architectural unification improves both policy learning and predictive environment modeling, offering a promising foundation for scalable multimodal world models in embodied agents.

The paper concludes that while recent unified vision-language models show unprecedented generation abilities, their role as world models has been largely uncovered; WorldBagel begins to systematically address this gap.