💬 NLP

PAN: A World Model for General, Actionable, and Long-Horizon World Simulation

This paper introduces PAN, a world model built on the Generative Latent Prediction (GLP) architecture that overcomes existing limitations in general, open-domain, action-driven prediction and long-horizon consistency by combining stateful latent representations, closed-loop information flow, and a mixed LLM/diffusion backbone to enable advanced simulative reasoning and planning.

Original authors: PAN Team, Zihan Liu, Yi Gu, Mingkai Deng, Guangyi Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Yichi Yang, Kun Zhou, Jiannan Xiang, Zhiting Hu, Zhengzhong Liu, Eric P. Xing

Published 2026-09-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: PAN Team, Zihan Liu, Yi Gu, Mingkai Deng, Guangyi Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Yichi Yang, Kun Zhou, Jiannan Xiang, Zhiting Hu, Zhengzhong Liu, Eric P. Xing

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a mind that does not just react to the world, but anticipates it. For decades, scientists have sought to build artificial intelligence that can do more than recognize a cat in a photo or write a poem; they want machines that understand how the physical world changes when you push a cup, turn a steering wheel, or open a door. This ability to predict the future based on current actions is known as a "world model." It is the cognitive engine that allows living beings to plan, to imagine "what if," and to avoid danger before it happens. While modern computers have become brilliant at generating images and text, they often struggle to simulate the cause-and-effect logic of reality. They can create a beautiful video of a car driving, but they cannot reliably predict what happens if that car suddenly swerves to avoid a pothole. Without this predictive power, artificial intelligence remains a passive observer rather than an active participant capable of making decisions in a complex, changing environment.

A team of researchers at the Mohamed bin Zayed University of Artificial Intelligence has introduced a new system called PAN, designed to bridge this gap. Rather than simply generating videos from scratch, PAN acts as a simulator that learns how the world evolves in response to specific instructions. The researchers built this system on a foundation called Generative Latent Prediction, a method that forces the computer to maintain a consistent internal memory of the world while it predicts the future. Instead of trying to guess every single pixel of a future image at once, which often leads to confusion and errors over time, PAN breaks the task down. It first creates a compact, abstract summary of the current scene and the action being taken. It then uses a powerful language-based reasoning engine to predict how that summary will change. Finally, it translates that new summary back into a video. This three-step process allows the system to keep track of objects, people, and physics over long periods, ensuring that a tree remains a tree and a car remains a car even after many seconds of simulated time.

The researchers trained PAN on a massive dataset of eight million video clips paired with detailed descriptions of the actions happening within them. These videos covered everything from everyday human activities to complex interactions between people and their surroundings. By teaching the system to predict the next moment in a video based on a language command like "drive through a snowy forest" or "pick up a blue cup," the team enabled PAN to learn the rules of physical cause and effect. When tested, the system demonstrated a remarkable ability to maintain consistency over long sequences. In one set of experiments, the researchers asked the model to simulate a series of actions, such as a robot moving objects between trays. While other systems would quickly lose track of the objects or hallucinate impossible physics after a few steps, PAN continued to generate coherent, logical outcomes for over ten steps. It successfully predicted that if a robot grasps a yellow can and moves it to a new tray, the can would appear in that new tray in the next frame, and the old tray would be empty.

To verify that PAN was truly simulating reality and not just creating pretty pictures, the researchers evaluated it against a rigorous benchmark called the World Reasoning Arena. This test measured three specific skills: how accurately the system simulated the immediate result of an action, how well it maintained a consistent world over long periods, and whether its predictions could help a separate computer program make better decisions. The results showed that PAN outperformed other open-source models and competed closely with leading commercial systems. In tasks requiring a machine to plan a sequence of moves to achieve a goal, such as sorting items on a table, integrating PAN improved the success rate of the planning agent by more than twenty percent compared to using a standard model. This suggests that the system's predictions were not just visually plausible but causally accurate, providing a reliable map of the future that a decision-making agent could trust.

The study also highlighted what happens when these systems are missing key components. The researchers ran tests where they removed the language-based reasoning engine or the specialized method used to smooth out transitions between video segments. Without the reasoning engine, the system struggled to understand complex instructions involving agents, like a person or a robot, and the video transitions became jerky. Without the smoothing mechanism, the quality of the video degraded rapidly as the simulation extended, with objects warping or disappearing. These findings confirmed that the combination of a reasoning brain and a careful, step-by-step generation process is essential for long-term stability. The system does not rely on a single trick; it relies on a closed loop where the prediction is constantly checked against the requirement that it must look like a real, observable change in the world.

Despite its success, the researchers are clear about the system's current boundaries. PAN operates best when actions are described in natural language, such as "turn left" or "open the door." It is not yet designed to handle the precise, continuous muscle commands required for delicate robotic tasks, like controlling the exact force of a gripper. Furthermore, while the system is significantly more stable than previous attempts, it can still drift over very long periods, occasionally losing track of small details or inventing objects that were not there. The authors note that the system inherits some visual habits from the video generation models it was built upon, which prioritize looking realistic over being physically perfect. This means that while the simulation is generally reliable, it is not yet a flawless representation of physics.

The work represents a significant step forward in teaching machines to think about the future. By combining the ability to reason with language and the ability to generate visual sequences, PAN offers a new way for artificial intelligence to explore possibilities without risking real-world consequences. It allows an agent to run "thought experiments," testing different courses of action to see which one leads to the desired outcome. The researchers plan to release their code and data to the public, inviting others to build upon this foundation. As the field moves forward, the goal remains to create systems that can not only see the world but understand how it works, paving the way for intelligent machines that can navigate, plan, and act with the same foresight that humans use every day.

Technical Summary: PAN – A World Model for General, Actionable, and Long-Horizon World Simulation

1. Problem Statement

The development of Artificial Intelligence (AI) systems capable of physical and embodied intelligence requires moving beyond Large Language Models (LLMs) to world models: cognitive simulators that allow agents to reason about environmental evolution, whether spontaneous or action-driven. While recent advancements in video generation, 3D scene evolution, and game simulation have yielded impressive visual fidelity, significant limitations persist in general, open-domain, action-driven prediction, long-horizon consistency, and abstract reasoning.

Existing approaches face specific architectural trade-offs:

Reinforcement Learning (RL) World Models: (e.g., Dreamer, JEPA) often operate in narrow, task-specific domains with restricted action spaces. Purely latent-space objectives (like JEPA) risk representation collapse and information loss, discarding fine-grained details necessary for faithful simulation.
Large Language Models: While capable of reasoning, LLMs are not natively configured to learn from non-linguistic data, limiting their world knowledge to what is expressible through language.
Video Generation Models: These models excel at open-loop media generation from fixed prompts but lack persistent internal states, controllable action interfaces, and object-level causal dynamics. They are optimized for visual realism rather than simulative reasoning or counterfactual imagination.

The core challenge remains: how to combine open-domain visual coverage with action-conditioned temporal consistency while maintaining observation-grounded dynamic rollout.

2. Methodology: The PAN Architecture

The paper introduces PAN, a general world model built on the Generative Latent Prediction (GLP) architecture. GLP addresses the tension between latent abstraction (needed for scalable, long-horizon evolution) and observational grounding (needed to ensure predictions correspond to realizable sensory changes).

2.1 Core Architecture

PAN implements GLP through three coupled components:

Vision Encoder ( $h$ ): Maps observations (video chunks) into structured latent world states. It utilizes the vision tower of Qwen2.5-VL-7B-Instruct, partitioning frames into spatial patches and grouping consecutive frames for temporal coherence.
Predictive Backbone ( $f$ ): Evolves latent states forward under actions. PAN adopts the language model of Qwen2.5-VL-7B-Instruct as an autoregressive backbone. This leverages pre-trained multimodal knowledge and language-conditioned sequence modeling to support symbolic reasoning, compositional generalization, and long-horizon consistency. The backbone predicts the next latent state conditioned on the full history of latent states and language actions.
Video Diffusion Decoder ( $g$ ): Reconstructs observable outcomes from predicted latent states. Adapted from Wan2.1-T2V-14B, the decoder is not merely a visualization module but the primary training signal. It uses a flow-matching objective to supervise predictions in observation space, preventing the representation collapse associated with purely latent objectives.

2.2 Key Technical Innovations

Causal Swin-DPM (Shift-Window Denoising Process Model): To address the intractability of reconstructing all pixel-level details and to prevent error accumulation in long rollouts, PAN employs a chunk-wise causal denoising mechanism.
- It operates within a sliding temporal window holding two chunks at different noise levels (e.g., the earlier chunk at $K/2$ noise, the later at $K$ noise).
- By conditioning each chunk on partially noised (rather than fully denoised) representations of preceding chunks, the mechanism dampens incidental pixel-level details and focuses supervision on semantically persistent structure (scene layout, object presence, motion direction).
- A causal attention mask prevents information leakage from future actions, ensuring the model respects the temporal causality of the simulation.
Action Interface: Actions are specified as natural-language instructions at the video-chunk level (e.g., "drive through a snowy forest"), enabling language-conditioned visual rollouts.
Training Strategy: PAN is trained on 8 million video-text pairs (approx. 4,500 hours) from publicly accessible sources, with dense captions emphasizing temporal dynamics. Training occurs in two stages:
1. Module-wise Training: Adapting the video decoder to the Causal Swin-DPM architecture while freezing the encoder and backbone.
2. Joint Training: End-to-end training under the GLP generative supervision objective, coupling latent prediction with observation-space reconstruction.

3. Evaluation and Results

PAN was evaluated on the World Reasoning Arena (WR-Arena) benchmark, which measures competence across three dimensions: Action Simulation Fidelity, Long-Horizon Forecast, and Simulative Reasoning and Planning.

3.1 Performance Metrics

Action Simulation Fidelity: PAN achieved an overall score of 58.7, the highest among all open-source models and competitive with leading commercial systems (e.g., KLING, Gen-3). It demonstrated strong performance in both Agent Simulation (animating controllable entities, score: 70.3) and Environment Simulation (scene-level interventions, score: 47.1).
Long-Horizon Forecast: PAN achieved the highest scores in Transition Smoothness (53.6%) and Simulation Consistency (64.1%), outperforming commercial systems and open-source baselines. Unlike standalone video generators that degrade rapidly after 5–8 steps, PAN's consistency degrades gradually over 9+ steps.
Simulative Reasoning and Planning: When integrated with a VLM-based planning agent, PAN improved task success rates by 26.7% (Open-Ended Planning) and 23.4% (Structured Planning) compared to the agent acting without a world model. This confirms that PAN's simulations are causally informative, not just visually plausible.

3.2 Ablation Studies

Progressive ablation studies confirmed the necessity of each component:

Removing the VLM backbone (retaining the decoder) preserved short-horizon fidelity but degraded long-horizon consistency and agent dynamics.
Removing the Causal Swin-DPM mechanism resulted in a significant drop in transition smoothness and simulation consistency, highlighting its role in preventing error accumulation.
The full GLP system sustained coherent simulation where standalone generators (like Wan-2.1) deteriorated rapidly.

4. Significance and Claims

The paper argues that the primary purpose of a world model is to enable hypothetical thinking—allowing agents to simulate the consequences of alternative actions and select optimal plans. PAN's significance lies in its architectural design, which strictly separates the world model (predicting state transitions) from the agent model (deciding actions), a factorization that avoids the bias of training a single model on both prediction and reward signals.

Key claims regarding PAN's contribution include:

Generative Supervision as Grounding: By decoding latent transitions back to the sensory domain, GLP ensures that the representation remains anchored to reality, avoiding the degeneracies of purely latent-space objectives.
Scalable Long-Horizon Reasoning: The combination of an LLM backbone for symbolic dynamics and a diffusion decoder for perceptual realization enables coherent multi-step rollouts that maintain causal structure over extended horizons.
Practicality of Reconstruction: Contrary to the view that full reconstruction is intractable, PAN demonstrates that generative supervision is a stable and effective training objective when architectural choices (like Causal Swin-DPM) are used to modulate the trade-off between fidelity and semantic saliency.

The authors position PAN not merely as a media generator, but as a simulator for reasoning, providing a foundation for future work in open-domain, interactive, and causally grounded world modeling. The paper acknowledges limitations, including the current reliance on language-level action interfaces (which may not support fine-grained continuous control) and the persistence of long-horizon drift, suggesting these as directions for future research.