🤖 machine learning

MOSH-WM: Mask-Grounded Soft-Hamiltonian Dynamics for Object-Centric World Models

MOSH-WM introduces a mask-grounded soft-Hamiltonian world model that explicitly constrains object dynamics using spatial moments of slot-owned image masks, achieving significant improvements in long-term video forecasting accuracy and error accumulation over existing object-centric baselines on OBJ3D and CLEVRER datasets.

Original authors: Zhekai Wang, Haoxiang Huang, Xiang Liu, Zhikang Chen, Yueqing Sun, Qi Gu, Shiji Zhou, Miao Liu, Sen Cui

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhekai Wang, Haoxiang Huang, Xiang Liu, Zhikang Chen, Yueqing Sun, Qi Gu, Shiji Zhou, Miao Liu, Sen Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To understand how a computer might learn to see the world, we must first understand how it currently struggles to do so. When artificial intelligence watches a video, it often treats the scene as a single, flat image that changes over time. It learns to predict the next frame by spotting patterns in pixels, much like a child learning to finish a sentence by hearing the rhythm of the words. However, this approach often fails when the scene gets complex or when the computer has to guess what happens far into the future. The system might predict that a rolling ball suddenly turns into a different color or vanishes, because it has not truly learned the rules of how objects move and interact. It sees the surface but misses the structure. To build a more reliable vision, researchers are turning to "object-centric" learning, a method that teaches the computer to break a scene down into distinct, separate things—like a red ball, a blue box, and a green table—and track each one individually. The goal is to create a "world model," a mental simulation that allows the computer to imagine future events by evolving these separate objects according to the laws of physics, rather than just guessing the next picture.

A team of researchers has introduced a new system called MOSH-WM that attempts to solve a specific weakness in these object-based models. While previous systems could track objects, they often mixed up the physical location of an object with its visual appearance. Imagine trying to describe the movement of a car by only looking at its paint job; if the paint changes, the system might think the car itself has moved or changed shape. The new model separates these two ideas completely. It creates a distinct "physical state" for each object that tracks only where the object is and how fast it is moving, ignoring what the object looks like. This physical state is grounded in a "mask," which is a digital outline showing exactly which pixels in the video belong to that specific object. By using this outline to calculate position and speed, the system builds a stable, geometric understanding of the scene that is not confused by changes in color or texture.

The researchers tested this approach on videos of moving objects, asking the computer to watch a short sequence of six frames and then predict the next thirty frames. In these tests, the new system significantly outperformed the best existing methods. When measuring how closely the predicted images matched the real future frames, the new model reduced visual errors by 25 percent and errors in the physical layout of the objects by 33 percent. This improvement was not just a fleeting success; the system maintained its accuracy over the entire thirty-frame sequence, whereas older models began to drift, causing objects to lose their shape or change color as the prediction went on. The key to this stability was a technique inspired by physics, where the system uses a "soft" guide to keep the objects moving in a realistic way, blending mathematical rules with learned patterns. This guide acts like a gentle hand, nudging the prediction back toward the laws of motion without forcing it into a rigid formula that might break when the video contains imperfect details.

Crucially, the system keeps the visual details separate from the movement calculations. Once the computer has predicted where the objects will be, it uses a separate memory bank of past appearances to fill in the details, ensuring the red ball stays red and the blue box stays blue. This two-part process allows the model to be both physically accurate and visually rich. The researchers found that removing either the physical guide or the visual memory caused the system to fail, proving that both parts are necessary for the model to work. In tests involving collisions and complex interactions, the model continued to hold up, suggesting that this way of separating "where" an object is from "what" it looks like is a robust way to teach machines to understand the physical world. The work does not claim to have solved all problems of video prediction, but it demonstrates that by grounding the computer's understanding of motion in the actual shape of objects, rather than just their visual features, we can create simulations that are far more stable and reliable over time.

Technical Summary: MOSH-WM

Problem Statement

Object-centric world models aim to forecast future videos by evolving a set of entity slots. However, existing approaches often rely on unconstrained visual features for dynamics supervision. In these models, slots tend to entangle geometry, appearance, identity, and decoder-specific semantics. Consequently, while a slot predictor might match the immediate future frame, the object layout and appearance often drift significantly during long-horizon closed-loop rollouts. The core challenge is to maintain a physical state that is explicitly grounded in where objects are supported (geometry) while preserving separate visual information (appearance/identity) required for high-fidelity decoding.

Methodology

MOSH-WM (Mask-Grounded Soft-Hamiltonian World Model) addresses this by factorizing the representation into a geometric phase state and a causal visual context. The architecture operates through three primary stages:

1. Mask-Grounded Phase-State Factorization

Instead of using raw slot vectors for dynamics, MOSH-WM extracts a canonical state $(Q, P)$ grounded in the spatial support of the objects:

Input: A frozen SAVi interface provides visual slots and soft masks.
Position ( $Q$ ): For each slot $n$ , the model identifies the "owned" mask support ( $M^{own}$ ) via winner-take-all assignment. It pools fixed spatial basis functions over this support to compute a position-like coordinate $Q_{t,n}$ . This explicitly separates geometry from appearance.
Momentum ( $P$ ): The momentum-like variable is derived directly from the temporal finite difference of the position states ( $P_t = (Q_t - Q_{t-1})/\Delta t$ ).
Visual Context ( $C_t$ ): In parallel, a causal spatiotemporal encoder retains the history of slot features to preserve appearance and identity, which are excluded from the phase-state extractor.

2. Soft-Hamiltonian Closed-Loop Dynamics

The model propagates the phase state $(Q, P)$ using a "soft" physical bias rather than a strict Hamiltonian solver:

Energy Function: A learned energy head $H_\vartheta(Q_t, P_t)$ provides a structural prior.
Hybrid Update: The transition blends a learned residual field (predicting increments $DQ_t, DP_t$ ) with the gradient of the energy function. The update rules are:
$Q_{t+1} = Q_t + (1-\alpha)DQ_t + \alpha \cdot \text{clip}_\gamma(\nabla_{P_t} H_t)$
$P_{t+1} = P_t + (1-\alpha)DP_t - \alpha \cdot \text{clip}_\gamma(\nabla_{Q_t} H_t)$
Here, $\alpha$ controls the blend, and clipping prevents instability. This allows the model to capture residual dynamics while being guided by physical structure.

3. Physics–Appearance Reconciliation

After the physical update, the model must reconstruct decoder-compatible slots:

Gated Composer: A module combines the normalized phase displacement with the causal visual context $C_t$ to produce a physical slot proposal.
Bounded Residual: A causal reasoning module applies a small, bounded correction in the decoder space to refine the proposal.
Output: The final slots are decoded into the next frame and appended to the context history for the next step.

Key Contributions

The paper outlines three specific contributions:

Mask-Grounded Phase-State Extractor: A mechanism that computes geometric coordinates ( $Q$ ) from per-slot owned masks and fixed spatial bases, and momentum ( $P$ ) from raw temporal differences, explicitly excluding appearance-bearing slot features from the dynamics input.
Soft-Hamiltonian Dynamics Module: A closed-loop transition that blends energy-gradient directions with a bounded learned field to propagate the phase state, acting as a structural prior without claiming exact system identification.
Decoder Reconciliation Interface: A method using causal context, gated composition, and bounded residual compensation to recover decoder-compatible slots from the propagated phase state.

Experimental Results

The model was evaluated on OBJ3D, CLEVRER, and Physion benchmarks, comparing against baselines like SlotFormer, SAVi-dyn, and PredRNN.

OBJ3D (6 observed $\to$ 30 predicted):
- MOSH-WM reduced LPIPS by 25.0% and spatial MSE by 33.7% relative to the strongest object-centric baseline (SlotFormer).
- It improved PSNR by 2.43 dB and SSIM to 0.8916.
- Object-state metrics showed significant gains: Centroid ADE reduced by 15.4%, FDE by 9.5%, and pairwise-distance error by 14.9%.
- Long-Horizon Stability: The error accumulation over 30 frames was significantly slower than baselines, with the gap widening as the rollout horizon increased.
CLEVRER (6 observed $\to$ 10 predicted):
- Demonstrated transferability to scenes with collisions and occlusions, improving all five visual metrics (LPIPS, PSNR, SSIM, MSE, FG-MSE) over SlotFormer and SAVi-dyn.
Physion (Transfer Test):
- Evaluated using a different frozen observer (STEVE) with different slot dimensions and resolutions.
- The full model maintained the lowest Slot MSE throughout the 35-step rollout compared to "physics-only" and "residual-only" variants, confirming the robustness of the phase/context factorization across different observers.
Ablation Studies:
- Removing the physics component nearly doubled the LPIPS.
- Removing the causal context caused large visual degradation.
- The combination of physics and context outperformed either component alone.
- Capacity sweeps indicated that the specific architecture (2 attention blocks, MLP ratio 2) provided the best balance between expressivity and stability, rather than simply scaling up capacity.

Significance and Claims

The paper claims that MOSH-WM successfully decouples the geometric evolution of objects from their visual appearance. By grounding the phase state in mask-supported spatial moments, the model achieves more structurally interpretable and stable long-horizon predictions. The "soft-Hamiltonian" approach provides a directional bias that prevents the drift common in purely data-driven slot predictors, while the separate causal pathway ensures that the decoder receives the necessary appearance and identity information.

The authors emphasize that the model does not claim to identify exact ground-truth mechanical energy but rather uses the Hamiltonian field as a structural prior to organize phase evolution. The results suggest that this factorization makes closed-loop object-centric prediction more robust, particularly in scenarios requiring long-term consistency of object layout and identity. The work is presented as a step toward more interpretable world models that respect physical regularities without sacrificing visual fidelity.