🤖 AI

Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling

This paper proposes Hamiltonian World Models, a novel framework that encodes observations into structured latent phase spaces and evolves them using Hamiltonian-inspired dynamics to generate physically meaningful, action-controllable, and long-horizon stable predictions for embodied decision-making.

Original authors: Sen Cui, Jingheng Ma

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sen Cui, Jingheng Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: "Good Looking" vs. "Real"

Imagine you are teaching a robot to play catch. Currently, many AI systems act like a fantasy movie director. They are amazing at predicting what the next frame of a video will look like. If you throw a ball, the AI can generate a video where the ball arcs beautifully through the air.

But here's the catch: The AI doesn't actually understand physics. It just knows that "red blur usually follows a hand."

The Flaw: If you ask the AI to predict what happens if the robot pushes a heavy box, the AI might generate a video where the box slides smoothly forever (because that looks cool in a movie). In reality, friction would stop the box. If the robot tries to act on this "movie," it will crash or fail because the prediction wasn't physically true, even if it looked real.

The authors argue that for robots and self-driving cars, we don't just need a model that makes pretty videos; we need a model that understands how the world actually works.

The Current Approaches (The Three Wrong Paths)

The paper says current research is stuck in three separate lanes, none of which fully solve the problem:

The Video Makers: They focus on making the future look realistic (like a movie). Problem: The physics might be wrong.
The 3D Builders: They focus on building a perfect 3D map of a room. Problem: They are great at static pictures but bad at predicting how things move or crash.
The Abstract Thinkers: They try to compress the world into a simple "idea" or summary. Problem: These summaries often lose the specific details needed to know how much force is needed to move an object.

The Solution: The "Hamiltonian" Engine

The authors propose a new way to build these world models using a concept from physics called Hamiltonian Mechanics.

The Analogy: The Roller Coaster vs. The Magic Wand

Old Way (Magic Wand): The AI guesses the future by looking at patterns. It's like a magician guessing the next card in a deck. It works for a while, but eventually, it runs out of tricks and makes a mistake.
New Way (Roller Coaster): The authors want the AI to act like a roller coaster track.
- In physics, a roller coaster doesn't just "guess" where it goes next. It follows strict rules based on energy. It has potential energy (height) and kinetic energy (speed). The track (the math) forces the car to move in a way that conserves energy.
- The authors want to build a "latent phase space" (a hidden mental map) for the robot. In this map, the robot doesn't just store "what the image looks like." It stores Position (where things are) and Momentum (how fast they are moving).

How It Works (The Recipe)

The paper outlines a four-step process for this new "Hamiltonian World Model":

The Translator (Encoding): The robot looks at the real world (cameras) and translates the messy video into a clean, structured "energy map." It figures out: "That object is at position X and moving with momentum Y."
The Physics Engine (Hamiltonian Dynamics): Instead of guessing the next step, the model uses a mathematical "energy function." It asks: "If I push this object, how does the total energy change?" The model then calculates the future path based on these energy rules.
- Crucial Detail: The authors admit the real world isn't perfect. Things have friction and brakes. So, they add "dissipation" (friction) and "control" (the robot's push) to the math so it doesn't assume energy is perfectly conserved like in a vacuum.
The Projector (Decoding): Once the model has calculated the future path using physics, it translates that "energy map" back into a video so the robot can see what it looks like.
The Planner (Decision Making): The robot simulates different actions ("If I push left vs. right") using this physics engine. It picks the action that leads to the best outcome, knowing that the simulation is physically reliable.

Why This is Better

Stability: Because the model follows energy rules, it won't drift off into nonsense after a few seconds. A roller coaster can't suddenly fly off the track unless the track breaks; similarly, this model won't predict impossible physics easily.
Data Efficiency: The robot doesn't need to watch a million videos to learn that "heavy things fall." The physics rules are built-in (like a pre-installed operating system), so it learns faster.
Interpretability: If the robot makes a mistake, we can look at the "energy map" and see exactly where the physics calculation went wrong. We aren't dealing with a "black box" that just guesses.

The Challenges (What the Paper Admits)

The authors are honest that this isn't a magic bullet yet:

Real Life is Messy: Real robots deal with sticky tape, squishy pillows, and sudden crashes (collisions). Pure physics math is hard to apply to these "messy" things.
Hard to Learn: It is very difficult to teach an AI to look at a video and correctly guess the hidden "momentum" and "position" numbers just by looking at pixels.
Not Perfect: The model treats the world as a "structural backbone" (a skeleton) rather than a perfect simulation. It's a guide, not a law.

Summary

The paper argues that to make robots truly smart, we need to stop just teaching them to predict videos and start teaching them to simulate physics. By using a "Hamiltonian" approach (focusing on energy, position, and momentum), we can build world models that are more stable, require less data to learn, and actually understand how the physical world moves, rather than just guessing what the next frame will look like.

Based on the paper "Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling," here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

Current world models, which are central to embodied intelligence, robotics, and autonomous driving, face a critical bottleneck: the gap between visual plausibility and physical validity.

The Limitation of Current Routes: Existing approaches generally fall into three categories:
1. 2D Video-Generative Models: Focus on synthesizing realistic future frames (e.g., GAIA-1, Genie) but often violate physical laws (momentum conservation, contact consistency) over long horizons.
2. 3D Scene-Centric Models: Focus on geometric reconstruction (e.g., NeRF, 3D Gaussian Splatting) but struggle with action-conditioned dynamic evolution.
3. JEPA-like Latent Models: Focus on abstract predictive representations but lack alignment with explicit physical quantities (position, velocity, energy, momentum).
The Core Issue: These models rely on highly expressive but weakly constrained neural sequence models ( $z_{t+1} = f_\theta(z_t, a_t)$ $z_{t + 1} = f_{θ} (z_{t}, a_{t})$ ). While flexible, they learn short-term statistical correlations rather than stable physical dynamics. This leads to:
- Long-horizon drift: Small prediction errors accumulate rapidly.
- Poor data efficiency: Models must re-learn basic physical laws from scratch.
- Lack of interpretability: It is unclear if failures are due to perception or dynamics.
- Action-Effect Mismatch: Generated futures may look realistic but are dynamically infeasible for planning.

2. Methodology: Hamiltonian World Models (HWM)

The authors propose Hamiltonian World Models, a framework that treats world modeling as a physically grounded generative dynamical system. Instead of predicting pixels or abstract vectors directly, the model evolves a structured latent phase space.

A. Core Architectural Components

The HWM architecture separates perception, dynamics, generation, and planning:

Structured Latent Encoding ( $E_\theta$ ):
- Observations ( $o_{\le t}$ ) are encoded into a structured latent phase state $z_t$ .
- Unlike a flat vector, $z_t$ is composed of entity-level variables: $z_t = \{(q_t^i, p_t^i)\}_{i=1}^N$ , where $q$ represents generalized coordinates (position) and $p$ represents generalized momenta.
Hamiltonian Dynamics Evolution ( $T_H$ ):
- The state evolution is governed by a learned Hamiltonian function $H_\phi(z)$ , representing the system's energy landscape.
- The dynamics follow Hamilton's equations: $\dot{z} = J \nabla_z H_\phi(z)$ , where $J$ is the symplectic matrix.
- Decomposition: The Hamiltonian is parameterized to separate kinetic energy ( $K$ ), potential energy ( $U$ ), and interaction terms ( $V$ ).
Controlled-Dissipative Extension:
- Recognizing that real-world robotics involves friction, damping, and non-conservative forces, the pure Hamiltonian flow is extended:
  $\dot{z}_t = (J - R_\psi(z_t)) \nabla_z H_\phi(z_t) + G_\psi(z_t)a_t + \epsilon_\psi(z_t, a_t)$
  - $R_\psi$ : Dissipation matrix (friction/damping).
  - $G_\psi a_t$ : Action-conditioned control input.
  - $\epsilon_\psi$ : Residual term for unmodeled phenomena (e.g., contact discontinuities), constrained to prevent the model from degenerating into a generic black-box network.
Generative Decoding ( $D_\theta$ ):
- The predicted latent trajectory $\hat{z}_{t+1:t+H}$ is decoded into future visual observations (video frames).
Planning Interface ( $U$ ):
- Candidate action sequences are rolled out through the HWM. The utility function $U$ evaluates these rollouts to select the optimal action $a^*_t$ .

3. Key Contributions

Paradigm Shift: The paper argues that the next stage of world modeling must move from "visual generation" to "physical structure modeling." It redefines the world model as a system where the core is latent dynamics constrained by physical laws, and video generation is merely the rendering of that state evolution.
Hamiltonian Inductive Bias: By embedding Hamiltonian mechanics into the latent space, the model enforces energy conservation (or controlled dissipation) as a structural prior. This replaces the learning of arbitrary vector fields with the learning of an energy landscape.
Structured Phase-Space Representation: The proposal to encode states as $(q, p)$ pairs provides a natural, interpretable latent space that aligns with the needs of embodied agents (reasoning about position and velocity/momentum).
Architectural Separation: The framework explicitly decouples the physical evolution engine (Hamiltonian module) from the perceptual rendering engine (video generator). This ensures that planning relies on physically consistent dynamics rather than just visual coherence.

4. Results and Expected Outcomes

Note: As this is a preprint proposing a new framework, the paper primarily presents theoretical arguments and architectural designs rather than extensive empirical benchmarks against SOTA. The "results" are the demonstrated theoretical advantages:

Long-Horizon Stability: By using symplectic integration and structure-preserving dynamics, HWMs are theoretically expected to suffer less from compounding errors compared to standard RNN/Transformer-based latent models.
Data Efficiency: The incorporation of physical priors (energy functions) reduces the hypothesis space, allowing the model to learn complex dynamics with fewer data samples compared to purely data-driven approaches.
Interpretability: The latent variables have clear physical meanings (coordinates, momenta, energy). This allows for better diagnosis of model failures (e.g., distinguishing between a perception error and a dynamics violation).
Action Controllability: The explicit inclusion of control terms ( $G_\psi a_t$ ) ensures that the generated futures are causally linked to agent actions, making them reliable for Model Predictive Control (MPC).

5. Significance and Future Directions

Bridging the Gap: HWMs offer a principled path to unify the strengths of video generation (perceptual richness) with the reliability of physics-based simulation.
Foundation for Embodied AI: For robotics and autonomous driving, where safety and physical feasibility are paramount, a model that guarantees dynamic feasibility is superior to one that merely looks realistic.
Scalability: The approach suggests a path toward "Physical World Foundation Models" that can generalize across unseen objects and tasks by learning reusable physical mechanisms (energy landscapes) rather than memorizing specific visual sequences.
Challenges: The authors acknowledge significant hurdles, including the difficulty of inferring meaningful $(q, p)$ from high-dimensional pixels, handling non-rigid/deformable objects, and modeling complex contact dynamics that violate smooth Hamiltonian flow.

In summary, this paper proposes a fundamental rethinking of world models, moving away from "generative video" toward "physically native simulation" by leveraging Hamiltonian dynamics as the core inductive bias for learning, planning, and decision-making in embodied agents.