🤖 machine learning

Unlocking Temporal Generalization in Hamiltonian Video Dynamics Models

This paper identifies and resolves specific failure modes in Hamiltonian Generative Networks that hinder temporal generalization in non-conservative video dynamics, enabling stable predictions at variable time steps well beyond the training distribution through targeted architectural fixes.

Original authors: Eli Laird, Corey Clark

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eli Laird, Corey Clark

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot that watches a video of a bouncing ball and learns exactly how physics works. You want this robot to predict what happens next, whether you ask it to guess the next second, the next millisecond, or even the next hour.

Most video-predicting robots today are like a student who memorized a single math problem: they can solve it perfectly if you ask at the exact speed they practiced, but if you change the speed, they get confused. They are stuck in "discrete time," meaning they only know how to take steps of a fixed size. If you ask them to move faster or slower, they just freeze or repeat their old steps, failing to understand the smooth flow of time.

The researchers in this paper tried a different approach using something called Hamiltonian Generative Networks (HGN). Think of these as robots that don't just memorize steps; they learn the underlying "energy rules" of the universe, like a continuous river of time. In theory, this should let them predict the future at any speed, whether you zoom in or zoom out.

However, when the team tested these robots in the real world (or rather, in a simulated world of a bouncing ball with friction and pushes), they found the river of time had some dangerous rapids. When they asked the robot to predict at speeds it had never seen before (specifically, step sizes larger than the 0.4 time units it was trained on), the robot started to crash in two very specific ways.

The First Crash: The Energy Explosion
The first problem was like a video game character getting hit by a glitchy force field. The robot had a "force map" that decided how hard to push the ball. When the time steps got too big, this map went wild, injecting way too much energy into the system. The result? The ball didn't just bounce; it exploded into high-frequency, blurry artifacts, smearing across the screen. The robot was essentially hallucinating a massive explosion because it wasn't told to keep its energy in check.

The Second Crash: The Drifting Clock
Even after the researchers fixed the explosion (by putting a "speed limit" on the force map using a technique called spectral normalization), the robot still failed. This time, it didn't explode; it just got out of sync. Imagine a runner who is running at the right speed but starts taking slightly too long of a stride. After a few laps, they are no longer running in rhythm with the music. The robot's prediction stayed within the right bounds, but it drifted out of phase with reality. The ball was in the right place, but at the wrong time. The researchers found this was caused by "global truncation error," which is a fancy way of saying the robot's internal calculator was taking steps that were too big to be precise, causing it to lose track of the exact timing.

The Fix: Sub-stepping
To fix the drifting clock, the researchers tried a clever trick called sub-stepping. Instead of asking the robot to take one giant leap of 1.5 time units, they told it to take four smaller, more careful steps of 0.375 units each (since 1.5 / 4 = 0.375).

Here is the magic: The robot's internal "physics brain" didn't change at all. It was the exact same model trained on the same data. But by breaking the big jump into smaller, manageable chunks, the robot's predictions snapped back into perfect alignment. The error vanished, and the robot could predict the future at speeds 1.5 times faster than its training, as long as it was allowed to "think" in smaller steps.

What This Means
The paper suggests that for these continuous-time video models to work at different speeds, you need two things:

Tighten the reins: Make sure the force map doesn't inject infinite energy (spectral normalization).
Slow down to speed up: When you need to predict a long time into the future, don't take one giant leap. Break it down into smaller steps (sub-stepping) so the math stays accurate.

The authors show that with these two tweaks, a single model trained at one speed can successfully predict dynamics at both finer and coarser resolutions. They didn't just guess this; they measured it using metrics like MAE, PSNR, SSIM, and LPIPS on a simulated 64×64 pixel oscillator. The results showed that while the model failed badly without these fixes, adding sub-stepping (specifically N=4 substeps) made the error curves flat and stable, even when the evaluation step size went up to 1.5.

So, the next time you want a robot to understand time, don't just teach it to memorize steps. Teach it the rules of energy, put a speed limit on its pushes, and remind it that sometimes, to go fast, you have to take small steps.

Technical Summary: Unlocking Temporal Generalization in Hamiltonian Video Dynamics Models

Problem Statement

Current world models predominantly operate on discrete-time grids with a fixed step size baked into their architecture. While effective at their training resolution, these models lack mechanisms to generalize across variable temporal resolutions. This limitation hinders applications requiring hierarchical planning (coarse and fine time scales), sim-to-real transfer (where simulation and control rates differ), and scientific engines that must query dynamics at multiple timescales.

While Neural ODEs offer a continuous-time alternative by parameterizing dynamics as a vector field, they often lack physical inductive biases, leading to instability in long-horizon rollouts due to violations of conservation laws. Hamiltonian Generative Networks (HGN) address this by grounding predictions in a continuous-time energy function, theoretically enabling frame-rate independence. However, in practice, HGNs fail to maintain temporal generalization in non-conservative (externally forced, dissipative) environments when evaluated at step sizes larger than those seen during training. The specific failure modes and their remedies in these settings have not been systematically characterized.

Methodology

The authors extend the Hamiltonian Generative Network (HGN) architecture to handle externally forced, dissipative environments using a Port-Hamiltonian structure. The model consists of:

Encoder/Decoder: A convolutional encoder maps pixel frames to a latent phase-space state $(q, p)$ (position and momentum). A decoder reconstructs images from the position coordinates $q$ .
Port-Hamiltonian Predictor: The dynamics are governed by the equations:
$\dot{q} = \frac{\partial H}{\partial p}, \quad \dot{p} = -\frac{\partial H}{\partial q} - \gamma \frac{\partial H}{\partial p} + G(a)$
Where $H(q, p)$ is a learned separable Hamiltonian, $\gamma$ is a learned damping coefficient, and $G(a)$ is a learned action-force map.
Integrator: A modified symplectic leapfrog integrator is used, which folds damping and forcing symmetrically into momentum half-steps.

To diagnose why generalization fails at larger step sizes (extrapolation), the authors introduce an inference-time substepping protocol. This involves splitting a single evaluation step $\Delta t$ into $N$ smaller integration substeps ( $\Delta t/N$ ) while preserving the total action impulse. This allows the authors to distinguish between representational failures (learning a poor continuous vector field) and numerical failures (truncation error from under-resolved integration).

Key Contributions and Failure Analysis

The paper identifies two distinct failure modes that occur when HGNs are rolled out at step sizes exceeding the training regime, along with targeted fixes for each:

1. Latent Magnitude Growth (Energy Explosion)

Cause: In unconstrained settings, the action-force map $G(a)$ is not bounded. At large step sizes, the numerical solver overshoots, and the unconstrained force map injects energy along unstable phase-space directions. This leads to exponential growth in latent magnitude and high-frequency artifacts in the decoded video.
Fix: The authors apply spectral normalization to the action-force map $G(a)$ , bounding its norm to 1. This caps per-step energy injection, preventing amplitude blow-up across the tested range of step sizes.

2. Phase Drift Accumulation (Temporal Misalignment)

Cause: Even with energy explosion constrained, predictions at large step sizes drift out of phase with the ground truth. This manifests as global truncation error accumulation from an under-resolved integrator, rather than a failure of the learned Hamiltonian representation itself.
Fix: The authors employ inference-time substepping. By increasing the number of integration steps $N$ within a single observation interval, the effective integration step size is reduced. When the effective step size approaches the training scale, the phase drift is eliminated, and temporal alignment is recovered.

Results

The authors evaluate their "bounded port-HGN" on a forced, damped mass-spring oscillator rendered as 64x64 RGB video. The model is trained at a step size of $\Delta t = 0.4$ and evaluated at various step sizes ranging from 0.05 to 1.5.

Interpolation: The model successfully generalizes to step sizes smaller than training ( $\Delta t < 0.4$ ) due to the inherent properties of the symplectic integrator, which refines any target step without retraining.
Extrapolation:
- Without fixes, the baseline model fails at $\Delta t > 0.4$ due to both amplitude explosion and phase drift.
- Applying spectral normalization eliminates amplitude explosion but leaves phase drift.
- Applying substepping (e.g., $N=4$ ) to the spectrally-normalized model recovers high-quality predictions (measured by MAE, PSNR, SSIM, and LPIPS) even at step sizes significantly larger than training (e.g., $\Delta t = 1.5$ ).
- The results demonstrate that the learned vector field is valid; the failure was purely numerical, arising from the mismatch between the integration step size and the solver's stability requirements.

Significance and Claims

The paper claims that the inability of Hamiltonian video models to generalize to variable temporal resolutions is not a fundamental limitation of the learned continuous-time representation, but rather a consequence of specific numerical instabilities in non-conservative settings.

By combining spectral normalization of the forcing term with inference-time substepping, the authors demonstrate that a single model trained at one frame rate can stably predict dynamics at both finer and coarser temporal resolutions. This provides a practical recipe for enabling temporal generalization in continuous-time video generation, allowing these models to be queried at arbitrary frame rates without retraining. The work highlights that for successful deployment in hierarchical planning or sim-to-real transfer, one must address both the energy injection constraints and the numerical resolution of the integrator.