Grounded World Model for Semantically Generalizable Planning

1. Problem Statement

The paper addresses a critical limitation in current Vision-Language-Action (VLA) models and Model Predictive Control (MPC) systems for robotics: the lack of semantic generalization.

• The Limitation of Current VLAs: Most state-of-the-art VLAs are built by fine-tuning pre-trained Vision-Language Models (VLMs). While they perform well on training data, they often fail to generalize to novel tasks involving unseen visual signals (e.g., new object colors) or complex referring expressions. They tend to overfit to specific sentence structures or visual shortcuts rather than truly understanding the semantics of the task.
• The Limitation of Current World Models: Traditional world models used in MPC (e.g., DINO-WM, JEPA-WM) predict future states in pixel space or latent space but rely on goal images to score candidate trajectories.
  • Obtaining a goal image in advance is difficult for novel tasks.
  • Natural language is a more human-friendly and flexible interface than images, but integrating language directly into the scoring mechanism of latent world models has been unexplored.
• The Core Challenge: How to build a planning system that can execute tasks based on natural language instructions in unseen environments, provided the motions required to solve the task have been demonstrated previously, without requiring the model to relearn the physics or dynamics from scratch.

2. Methodology: Grounded World Model (GWM)

The authors propose Grounded World Model (GWM), a system that operates within a vision-language-aligned latent space to enable semantic generalization.

Core Architecture

1. Foundation Model Alignment: GWM utilizes the latent space of a pre-trained multi-modal retrieval model, Qwen3-VL-Embedding. This model encodes images, videos, and text into a shared embedding space where cosine similarity can be computed.
2. Latent Space Prediction: Instead of predicting future pixel images (which is computationally expensive and redundant), GWM learns a transition function $P_\theta$ that predicts the latent embedding of the future state ($z_t$) given the current observation and a sequence of actions.
   • The foundation model's weights remain frozen, preserving its pre-trained world knowledge and semantic understanding.
   • Only the transition function (a standard Transformer) is trained.
3. Rendering-based Action Tokenization (RAT): To feed actions into the vision encoder, the system renders the robot's future joint positions into images using the robot's URDF and camera parameters. This creates a unified representation for both observations and actions, making the system embodiment-agnostic (no learnable action encoder parameters).

Planning Workflow (GWM-MPC)

The system follows an MPC loop:

1. Trajectory Proposal: Candidate trajectories are retrieved from the training dataset using a K-Nearest Neighbors (KNN) approach based on current joint positions. This avoids the need for a separate action generation model.
2. Future Prediction: For each candidate trajectory, the GWM predicts the future state embedding ($z_t$) in the latent space.
3. Semantic Scoring: The system encodes the natural language instruction ($\ell$) into a goal embedding ($z_g$) using the same foundation model.
4. Selection: The optimal trajectory is selected by maximizing the cosine similarity between the predicted future embedding and the instruction embedding:
   $$a^* = \arg \max_{n} \frac{z_n \cdot z_g}{\|z_n\| \|z_g\|}$$
   This transforms the planning problem into a semantic retrieval task: "Which action sequence leads to a future that best matches the text instruction?"

3. Key Contributions

1. Grounded World Model (GWM): A novel architecture that grounds world model predictions directly to natural language semantics in a frozen latent space, bypassing the need for goal images and avoiding the catastrophic forgetting associated with fine-tuning VLMs.
2. Rendering-based Action Tokenization (RAT): A training-free, embodiment-agnostic method to encode robot actions as images, enabling zero-shot generalization across different robot morphologies (e.g., from Franka Panda to xArm6).
3. WISER Benchmark: The authors introduce the World-knowledge Integrated Semantic Embodied Reasoning (WISER) benchmark.
   • It consists of 288 tasks (24 categories $\times$ 12 tasks) involving pick-and-place operations.
   • Crucial Design: The test set uses unseen visual signals (cube colors) and unseen referring expressions (spatial descriptions) compared to the training set, while the required motions remain identical. This strictly tests semantic generalization rather than motion learning.
4. Empirical Evidence of VLA Limitations: The paper demonstrates that current SOTA VLAs (e.g., $\pi_0$, InstructVLA, GR00T) fail to generalize on WISER, achieving only ~22% success on test tasks despite 90% success on training tasks, proving they rely on memorization rather than semantic understanding.

4. Experimental Results

The experiments were conducted on the WISER benchmark using a Franka Panda robot (and cross-embodiment tests on xArm6).

• Semantic Generalization: GWM-MPC achieved an 87% success rate on the test set with novel visual signals and referring expressions. In contrast, the best-performing VLA (InstructVLA) dropped to 47%, and others like $\pi_0$ dropped to 8%.
• Cross-Embodiment: By using RAT, GWM-MPC trained on Panda data successfully controlled an xArm6 robot with a different action space and kinematics, achieving 83% success on test tasks without retraining.
• Efficiency: Training GWM required only 20 GPU hours (vs. hundreds for VLAs). While inference is slower due to multiple GWM forward passes (N=12), the system is highly data-efficient.
• Ablation Studies:
  • Removing the world model (MPC w/o GWM) reduced performance to random selection (~8% success).
  • Using a learnable action encoder (GWM-MPC-AC) instead of RAT caused a significant drop in generalization, confirming the benefit of RAT.
  • The system's performance is bounded by the foundation model (Qwen3-VL-Embedding), suggesting future improvements lie in better foundation models.

5. Significance and Conclusion

This work fundamentally shifts the paradigm of building robotic planners:

• Decoupling Semantics and Dynamics: Instead of forcing a VLM to learn both world dynamics and language understanding simultaneously (which leads to forgetting), GWM separates these. It leverages the frozen, robust semantic knowledge of a foundation model and only learns the specific transition dynamics required for the robot.
• Language as the Goal: It demonstrates that natural language can effectively replace goal images in world model planning, enabling more flexible and interactive robot control.
• Robustness to Novelty: The results prove that for tasks where the motion is known but the context (objects, descriptions) is new, a retrieval-based approach grounded in a strong foundation model outperforms end-to-end fine-tuned VLAs.

The paper concludes that GWM-MPC offers a promising, data-efficient, and semantically generalizable alternative to current VLA approaches, particularly for open-world robotic manipulation tasks.