🤖 AI

IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training

IMPACT is a scalable framework for training interaction-aware world models that addresses the under-supervision of dynamic objects in standard MSE training by using cross-attention to generate an internal interaction map for reweighting denoising supervision, thereby improving physical plausibility and visual quality without requiring external representations.

Original authors: Rongze Tang, Jianjie Fang, Zhaolu Wang, Ziyou Wang, Xvyuan Liu, Haisheng Su, Xin Zhang, Wei Wu, Chen Gao, Yong Li, Zhibo Chen

Published 2026-09-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rongze Tang, Jianjie Fang, Zhaolu Wang, Ziyou Wang, Xvyuan Liu, Haisheng Su, Xin Zhang, Wei Wu, Chen Gao, Yong Li, Zhibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that can watch a video of a robot arm picking up a cup and then predict exactly what happens next. It knows the cup will lift, the table will stay still, and the light will reflect off the ceramic. This kind of artificial intelligence, known as a world model, is becoming increasingly skilled at imagining the future. It learns from vast amounts of video data to understand how objects move and how the world changes over time. These systems are crucial for teaching robots to perform complex tasks, from assembling electronics to helping with household chores, because they allow machines to practice and plan in a virtual space before ever touching a real object. However, while these models are excellent at predicting the general flow of a scene, they often struggle when it comes to the specific moment of contact. When a robot hand grabs a tool or a human finger touches a button, the current technology frequently produces blurry, physically impossible, or inconsistent results. The object might melt into the hand, or the motion might look disconnected from the action that caused it.

Researchers have tried to fix this by feeding the computer extra information, such as detailed maps of depth or the precise paths objects should follow. While this helps, it requires expensive, time-consuming preparation and often limits how much data the system can learn from. A new study by a team of researchers from institutions including Tsinghua University and the University of Science and Technology of China offers a different solution. They discovered that the problem wasn't a lack of data, but rather how the computer was being taught to learn from it. By changing the way the training process focuses its attention, they created a method called IMPACT that significantly improves how these models handle physical interactions, without needing any extra external data or slowing down the system.

The core issue the researchers identified is a kind of imbalance in how the computer learns. When training these world models, the system is usually asked to predict the next frame of a video and is penalized for every mistake it makes. However, in a typical video, most of the picture is static background: a wall, a table, or a floor that doesn't change much. Because these static areas make up the vast majority of the pixels, the computer spends most of its effort trying to get the background right, simply because there is so much of it. The tiny, critical areas where the action happens—the gripper touching a block or a hand grasping a tool—are so small that they get lost in the noise. The computer effectively ignores them, treating them as unimportant because they occupy so little space. This leads to a situation where the video looks smooth and coherent overall, but the specific interactions are physically wrong or visually messy.

To solve this, the researchers developed a method that teaches the computer to pay more attention to the parts of the video where the action is actually taking place. They realized that the computer already has a built-in hint about where to look. When the system receives a command like "pick up the blue bowl," it uses a mechanism called cross-attention to link the words "blue bowl" to the visual parts of the video. This creates a mental map showing where the computer thinks the bowl is. The researchers used this existing map as a starting point. They then asked the computer to look at those specific areas and check how difficult it was to predict what would happen there. If the computer was struggling to predict the movement of the bowl, it meant that area was important and needed more focus.

The system, named IMPACT, takes this process a step further by sampling several possible regions around the object and weighing them based on how hard the computer found them to predict. It then creates a special guide, or map, that highlights these interaction zones. During training, the computer is instructed to prioritize correcting its mistakes in these highlighted areas, effectively telling it, "Ignore the wall for a moment; focus on the hand and the object." Crucially, this guide is generated entirely from the computer's own internal signals during the training process. It does not require any outside tools, manual labeling, or extra sensors to tell it where the action is. This makes the method highly scalable, allowing it to work with massive amounts of data without the heavy costs associated with previous approaches.

The researchers tested this new method on two very different types of tasks: a robot arm manipulating objects on a table and a human hand performing dexterous tasks from a first-person perspective. In both cases, they trained the models using standard video generation technology but applied the IMPACT method to the learning process. The results were consistent and significant. The models trained with IMPACT produced videos where the interactions were far more realistic. When a robot gripper closed on a block, the block stayed solid and moved correctly. When a human hand picked up a cup, the fingers wrapped around it naturally, and the cup responded with the right weight and motion. The visual quality of the interactions improved dramatically, with fewer distortions and more physically plausible movements compared to models trained with the standard, uniform approach.

In the robot arm tests, the new method improved the overall quality score by a noticeable margin, particularly in how well the robot followed instructions and how accurately it simulated the physics of the objects. For the human hand tasks, the improvement was even more striking in terms of visual fidelity. The models trained with IMPACT generated images that were sharper and more coherent, with the hand and the object interacting in a way that looked natural to the human eye. The researchers found that this approach worked well across different types of computer architectures and control signals, suggesting that the solution is robust and adaptable. By simply reweighting the learning process to focus on what matters most, they were able to unlock a higher level of physical realism without changing the underlying structure of the AI.

This work demonstrates that the key to better interaction in artificial intelligence may not be in adding more complex data or external constraints, but in refining how the system learns from the data it already has. The researchers showed that by identifying the mismatch in how attention is allocated during training and correcting it, they could guide the model to master the difficult, sparse details of physical interaction. The method requires no changes to the system when it is actually being used to generate new videos, meaning it is a pure improvement to the training phase. As world models continue to evolve to support more complex robotic tasks and simulations, techniques like IMPACT provide a scalable path forward, ensuring that the future these models imagine is not just visually smooth, but physically true.

Technical Summary: IMPACT – Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training

Problem Statement

World models have achieved significant progress in predicting future observations and simulating embodied environments. However, they struggle to model physically plausible interactions, often exhibiting object deformation, discontinuous motion, weak action coupling, and inconsistent contact.

Existing approaches attempt to resolve this by constraining generation with external dense representations (e.g., optical flow, depth maps, 3D geometry, or manual annotations). These methods face two primary limitations:

Scalability: Acquiring spatiotemporally dense representations requires costly auxiliary estimators or manual annotation, which does not scale with large training corpora.
Accuracy Ceiling: The supervision quality is bounded by the accuracy of these external signals.

Furthermore, the authors identify a fundamental supervision-allocation mismatch in standard world model training. The globally averaged Mean Squared Error (MSE) denoising objective uniformly weights all spatiotemporal positions. Consequently, prevalent static content dominates the optimization signal, while sparse dynamic-object regions—critical for interaction generation—are disproportionately under-supervised. This leads to models that generate visually coherent videos globally but fail to optimize the specific regions where interactions occur.

Methodology: IMPACT

The authors propose IMPACT (Interaction-aware Model training framework with Prior-guided Attention Calibration and Targeting). IMPACT addresses the supervision mismatch without requiring external representations or inference-time modifications. It leverages the internal object-conditioned cross-attention of large-scale pretrained video generation backbones as a spatiotemporal prior for interaction regions.

The framework operates within a single training step via two complementary components:

1. Forward Pass: Attention Distribution Sampling (ADS)

ADS converts semantic object grounding into a precise interaction map through three stages:

Object-Token Grounding: A frozen language model (Qwen2.5-0.5B) extracts the noun phrase denoting the manipulated object from the instruction (e.g., "blue bowl"). This phrase is aligned with specific text tokens in the transformer's input sequence.
Attention Aggregation: The cross-attention probabilities associated with these object tokens are aggregated across transformer layers and heads to form a coarse, object-conditioned proposal distribution (the attention map).
Sampling and Calibration:
- Candidate Sampling: ADS samples $K$ spatially coherent candidate regions from this prior distribution.
- Error-Based Weighting: Each candidate is evaluated using its detached local prediction error (the MSE within that specific region). Candidates covering regions where the model currently struggles (high error) receive higher weights.
- Interaction Map Construction: The weighted aggregation of these candidates forms a refined Interaction Map ( $G$ ). This map serves as a spatial prior that highlights regions likely to undergo action-conditioned changes, calibrated by the model's current prediction difficulty.

2. Backward Pass: Interaction-Weighted Supervision (IWS)

IWS utilizes the interaction map $G$ to reweight the denoising supervision:

Reweighting: The loss function is modified to assign higher weights to positions within the interaction map ( $w_p \in [1, \gamma]$ ), while retaining supervision over the entire spatiotemporal field. This ensures sparse interaction regions contribute more significantly to gradient optimization than their spatial extent would suggest.
Gradient-Decoupled Optimization: To prevent the model from collapsing the attention prior onto its own signal (i.e., optimizing the attention map solely to minimize the weighted loss), the authors decouple the optimization:
- Cross-Attention Parameters ( $\theta_{ca}$ ): Updated using the original global (uniform) MSE objective.
- Remaining Parameters ( $\theta_{rest}$ ): Updated using the interaction-weighted objective ( $L_{IWS}$ ).
- This separation ensures the interaction region estimation remains governed by the global objective, while the rest of the network learns from the spatially reallocated supervision.

Key Contributions

Identification of Supervision Mismatch: The paper identifies that standard MSE training under-supervises sparse interaction regions due to the dominance of static content, leading to poor physical plausibility.
Scalable Training Framework (IMPACT): A novel framework that uses internal cross-attention as a prior to reweight denoising supervision. It requires no external dense representations (like depth or flow) and incurs no inference-time overhead.
Dual-Component Design:
- ADS: Calibrates attention proposals using detached local prediction errors to form a precise interaction map.
- IWS: Targets denoising supervision to these regions while decoupling region estimation from regional optimization via gradient routing.
Empirical Validation: Demonstrated consistent improvements across robotic-arm and human-hand manipulation tasks using diverse control modalities and DiT backbones (Wan 2.2 and Cosmos-Predict 2.5).

Experimental Results

The authors evaluated IMPACT on two benchmarks: WorldArena (robotic-arm manipulation) and EgoDex (human-hand manipulation).

Robot-Arm Manipulation (WorldArena):
- IMPACT improved the overall EWMScore by +6.62 points (11.8%) on the Cosmos-Predict 2.5 backbone and +3.81 points (6.5%) on the Wan 2.2 backbone.
- It achieved state-of-the-art results in Controllability (71.19) and Physics Adherence (55.87), significantly outperforming general world models (e.g., Wan 2.6, CogVideoX) and representation-guided baselines.
- Specific interaction metrics (Interaction Quality and Action Following) saw dramatic improvements, e.g., Action Following rising from 0.0133 to 0.6260 on Cosmos-Predict 2.5.
Human-Hand Manipulation (EgoDex):
- IMPACT achieved the best results on visual metrics, reducing FVD from 366.12 to 110.94 and FID from 44.71 to 5.79 compared to the action-conditioned Wan 2.2 baseline.
- It also led in hand-interaction metrics, improving CLIP-Hand (0.952) and Hand IoU (0.772), indicating superior local interaction fidelity and hand localization.
Qualitative Analysis: Visual comparisons show that IMPACT produces sharper contact regions and more coherent object dynamics compared to baselines, which often blur or distort interaction zones.

Significance and Claims

The paper claims that IMPACT provides a scalable solution to the problem of physically plausible interaction generation in world models. By revisiting the training objective and leveraging internal attention mechanisms, IMPACT mitigates the supervision-allocation mismatch without the computational and data costs associated with external geometric or motion priors.

The authors emphasize that their approach:

Scales readily with training data size, as it does not rely on auxiliary estimators that grow in cost with corpus size.
Preserves inference efficiency, as the trained model retains its original architecture and requires no modifications during generation.
Improves physical consistency and visual quality simultaneously, demonstrating that better interaction modeling does not require sacrificing global video coherence.

The work suggests that for embodied simulation, the key to better interaction generation lies not in adding external constraints, but in better allocating the existing supervision signal to the regions where it matters most.