💻 computer science

ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games

This paper introduces ActSWM, an action-sensitive latent world model that addresses the "Context Collapse" failure mode by enforcing a transition-separation principle to ensure long-horizon rollouts remain distinguishable across different action sequences, thereby improving planning performance in open-world games and enabling action recovery from offline videos.

Original authors: Zhenfeng Gan, ZiTong Zeng, Jiajun Cheng, Yeke Song, Yongyi Tang, Xueqian Wang

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenfeng Gan, ZiTong Zeng, Jiajun Cheng, Yeke Song, Yongyi Tang, Xueqian Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play a video game, but you can't let it touch the controller for every single move. Instead, you want the robot to build a "dream" of the game in its head. It looks at the screen, imagines what would happen if it jumped, then what would happen if it ran, and then picks the best path forward without actually crashing into a wall. This is the world of World Models, a branch of artificial intelligence where machines learn to predict the future by simulating it inside a compressed, mathematical "dream space."

For this to work, the robot's dream needs to be sensitive to its choices. If the robot imagines jumping, the dream should show it flying up. If it imagines running, the dream should show it speeding forward. The big problem scientists have been facing is that these dreams often get "lazy." The robot might predict a future that looks perfect, but it doesn't actually change based on what the robot decides to do. It's like watching a movie where the plot is pre-written; no matter how much the character tries to turn left, the camera just pans right anyway. This paper tackles that specific glitch, asking: Can we teach a robot to dream in a way where the future actually listens to its actions?

The "Lazy Dream" Problem

Meet ActSWM, a new kind of AI brain designed to stop robots from having "lazy dreams." In the world of open-world games like Minecraft, agents (the robots) need to plan far into the future. They need to figure out, "If I mine this stone now, can I build a house later?" To do this, they use a Latent World Model. Think of this model as a super-fast simulator that runs in the background. Instead of processing every pixel of the game screen, it compresses the world into a simplified "latent" code—a mental shorthand.

The goal is for the agent to run thousands of these mental simulations in a split second to find the best move. But here's the catch: many existing simulators suffer from a bug the authors call Context Collapse.

Imagine you are telling a story to a friend. If you say, "I went to the park," your friend might imagine a sunny day. If you say, "I went to the park and it rained," they imagine a wet day. That's good. But Context Collapse is like a friend who, no matter what you say, always imagines the exact same sunny day. They are so focused on the general idea of "the park" that they stop listening to your specific details about the rain. In the AI world, this means the simulator predicts a plausible future (a sunny park) but fails to change that prediction based on the specific actions the robot takes (the rain). The robot thinks, "I can jump or I can run, and the future looks the same either way," which makes planning impossible.

The Solution: A Frozen "Action Detective"

The authors propose ActSWM (Action-Sensitive World Model) to fix this. Their secret sauce is a principle they call Transition Separation. They want to ensure that if the robot takes two different paths, the resulting dreams must look different.

To enforce this, they introduce a clever trick: a Fixed Action Readout. Imagine you have a detective whose job is to look at a scene and guess what action just happened. Usually, if the detective is learning alongside the scene, they might cheat. If the scene looks too similar for two different actions, the detective might just change their definition of "jumping" to match the scene, rather than forcing the scene to be different.

ActSWM stops this cheating by freezing the detective. They give the AI a detective with a fixed, unchangeable rulebook. The AI is then forced to make its "dreams" (the latent transitions) so distinct that this frozen detective can accurately tell them apart. If the AI tries to make the "jump" dream and the "run" dream look the same, the frozen detective will fail to guess the action, and the AI gets a penalty. This forces the AI to keep its future predictions sharp and responsive to every specific button press.

What They Found

The team tested this new brain in the chaotic, blocky world of Minecraft and across three other games (Counter-Strike 2, GTA V, and Apex Legends).

1. Stopping the Collapse:
When they compared ActSWM to older models, the difference was stark. In a test where they asked the AI to imagine a future with real actions versus a future where the robot did nothing (all-zero actions), the old models produced futures that were nearly identical. The new ActSWM, however, created a massive gap. The authors measured this "action gap" and found ActSWM produced a separation roughly 380 times larger than the strongest previous baseline. The robot's dreams were finally listening to the controller.

2. Better Planning:
Because the dreams were more accurate, the robot got better at playing. In Minecraft, when asked to perform tasks like placing a torch, mining a stone block, or building a pillar, ActSWM improved success rates significantly. For example, it improved the success rate of mining a stone block by 90.0% and building a pillar by 54.5% compared to the baseline. The robot could now reliably distinguish between a path that leads to a house and a path that leads to a cliff.

3. Reading the Mind from Video:
Finally, they tested if the AI could look at a video of a human playing and guess what buttons they were pressing. This is like watching a movie and guessing the actor's moves. Using a method called Cross-Entropy Minimization (CEM), ActSWM was able to recover the correct actions from offline videos much better than random guessing. In GTA V, it improved the "cost" of finding the right action by 252.38 compared to random search, and it correctly identified active key presses (like when a player actually pressed a button) with an accuracy boost of up to 0.711.

The Takeaway

The paper suggests that for AI agents to truly master complex, open-ended games, they need more than just accurate predictions of the future; they need predictions that are sensitive to action. If the future doesn't change when you change your mind, you can't plan. By freezing a simple "action detective" and forcing the AI to keep its future scenarios distinct, ActSWM proves that we can build world models that don't just daydream, but actually listen to the player. This isn't just a small tweak; it's a fundamental shift in how we teach machines to imagine the consequences of their own choices.

Technical Summary: ActSWM – Action-Sensitive World Models for Long-Horizon Planning

Problem Statement: Context Collapse in Latent World Models

Latent world models facilitate efficient model-predictive control (MPC) by optimizing control sequences within a compressed latent space. However, existing autoregressive latent predictors often suffer from Context Collapse, a specific failure mode distinct from standard prediction error accumulation.

In Context Collapse, a predictor may maintain high similarity to future states based on visual history alone, yet become insensitive to the specific action sequences conditioning the rollout. Consequently, distinct action sequences (e.g., a recorded trajectory vs. an all-zero action sequence) produce nearly indistinguishable latent futures. This phenomenon renders the model ineffective for planning: if alternative actions do not induce separable latent consequences, a planner cannot reliably compare downstream outcomes or optimize control sequences. While existing methods like the World-Action Model (WAM) incorporate inverse dynamics to retain action information in local transitions, they do not explicitly constrain predicted rollouts under alternative future action sequences to remain distinguishable.

Methodology: The ActSWM Framework

The authors propose ActSWM (Action-Sensitive World Model), a JEPA-based latent world model designed to enforce action sensitivity through a transition-separation principle. The core hypothesis is that a planning-useful latent dynamics model must keep alternative-action futures distinguishable and ensure the action associated with each local transition is recoverable.

Architecture and Components

ActSWM augments a standard latent prediction backbone with two specific mechanisms:

Rollout-Level Separation: The model is trained to ensure that future latent trajectories generated from the same context but conditioned on different action sequences (specifically, recorded actions vs. all-zero actions) remain separable.
Fixed Action Readout: To enforce transition-level sensitivity, ActSWM employs a parameter-frozen action readout ( $q_{\phi_0}$ ). Unlike adaptive decoders that can shift their decision boundaries to minimize loss, a fixed readout forces the latent transition representations themselves to become distinguishable to allow for accurate action recovery.

Training Objective

The training objective ( $\mathcal{L}$ ) combines four terms to balance prediction fidelity with action sensitivity:

Prediction Loss ( $\mathcal{L}_{pred}$ ): Standard multi-step prediction loss aligning the rollout conditioned on recorded actions with the encoded future trajectory.
Hinge Loss ( $\mathcal{L}_{hinge}$ ): A contrastive term that penalizes excessive similarity between the rollout conditioned on recorded actions and the rollout conditioned on all-zero actions. This explicitly encourages the model to preserve action-dependent differences over long horizons.
Readout Loss ( $\mathcal{L}_{readout}$ ): A loss term applied to both encoded transitions and rollout-predicted transitions using the frozen readout. This ensures that latent transitions contain sufficient information to recover the input action.
SigReg Loss ( $\mathcal{L}_{sig}$ ): Adopted from LeWorldModel (LeWM) to stabilize the scale and distribution of prediction-space features.

The total objective is:
$\mathcal{L} = \mathcal{L}_{pred} + \lambda_{sig}\mathcal{L}_{sig} + \lambda_{hinge}\mathcal{L}_{hinge} + \lambda_{readout}\mathcal{L}_{readout}$

Key Contributions

The paper outlines four primary contributions:

ActSWM Model: Introduction of a JEPA-based latent world model specifically designed to support long-horizon prediction while maintaining action-sensitive rollout dynamics.
Validation in Open-Ended Tasks: Demonstration of improved closed-loop planning in MineStudio tasks (stone mining, pillar building, torch placement), showing significant success rate improvements over the LeWM baseline.
Diagnosis of Context Collapse: A diagnostic protocol comparing rollouts conditioned on recorded actions versus all-zero actions. ActSWM achieves a roughly 380× larger separation between these futures compared to the strongest baseline, proving its rollouts remain responsive to action inputs.
Offline Action Recovery: Validation that the learned dynamics can recover control sequences from offline gameplay videos across three distinct domains (Counter-Strike 2, GTA V, Apex Legends) using Cross-Entropy Method (CEM) optimization, achieving up to a 16.5× improvement in cost over random search.

Experimental Results

Q1: Diagnosing Context Collapse

Using offline Minecraft trajectories, the authors evaluated whether models rely on action inputs or context alone.

Baseline Behavior: Models with longer contexts or multi-step training improved prediction quality under recorded actions but failed to differentiate between recorded and zero-action rollouts (small $\Delta_k$ ).
ActSWM Performance: ActSWM reduced similarity under zero-action conditions by over 80% while maintaining high fidelity for recorded actions, achieving the largest action gap ( $\Delta_k = 0.760$ ).

Q2: Closed-Loop Model-Based Planning

In MineStudio, ActSWM was used as the dynamics model for MPC.

Performance Gains: Compared to LeWM, ActSWM improved task success rates by:
- 5.3% for placing a torch.
- 90.0% for mining a stone block.
- 54.5% for building a pillar.
Implication: The results indicate that action-sensitive rollouts are critical for planning tasks requiring sustained control and sequential decision-making.

Q3: Discriminative Action Representations

The model was tested on its ability to infer actions from offline videos in CS2, GTA V, and Apex Legends using CEM.

Action Gap: ActSWM improved the CEM recovery cost over random search by up to 16.5× (e.g., 252.38 in GTA).
Active Accuracy: ActSWM significantly improved the accuracy of recovering actions when keys were actually pressed (up to 0.711 absolute improvement), whereas baselines often failed to distinguish active keys from the dominant inactive class.

Significance and Claims

The paper posits that action sensitivity is a core requirement for planning-oriented latent world models, distinct from and complementary to future-state prediction accuracy. The authors claim that:

Context Collapse is a distinct failure mode where accurate rollouts fail to respond to alternative actions, rendering them useless for planning.
Action sensitivity links counterfactual rollout comparison, reliable closed-loop planning, and scalable action labeling from gameplay video.
By enforcing a transition-separation principle via a fixed readout and contrastive rollout training, ActSWM enables world models to serve as effective planners in open-world environments, moving beyond simple state prediction to actionable dynamics.

The work does not claim to solve all open-world agent challenges (such as sample efficiency or deployment costs) but specifically addresses the limitation of existing latent models in distinguishing the consequences of different action sequences over long horizons.