💻 computer science

Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems

This paper introduces Morpheus, a physics-informed evaluation framework using 130 real-world videos and conservation laws to demonstrate that contemporary video generation models fail to accurately encode Newtonian dynamics despite producing visually appealing results.

Original authors: Antonios Tragoudaras, Chenyu Zhang, Daniil Cherniavskii, Antonios Vozikis, Thijmen Nijdam, Derck W. E. Prinzhorn, Mark Bodracska, Nicu Sebe, Andrii Zadaianchuk, Efstratios Gavves

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonios Tragoudaras, Chenyu Zhang, Daniil Cherniavskii, Antonios Vozikis, Thijmen Nijdam, Derck W. E. Prinzhorn, Mark Bodracska, Nicu Sebe, Andrii Zadaianchuk, Efstratios Gavves

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of incredibly talented artists who can paint moving pictures (videos) that look so real, you'd swear you could reach out and touch them. These artists are the new "Video Generative Models" (like SORA, Veo, or Kling). Everyone is excited because people hope these artists don't just paint pretty pictures, but actually understand how the world works—like a "world simulator" that knows how gravity pulls things down or how a ball bounces.

But here's the catch: Do they actually know the rules of physics, or are they just really good at faking it?

This paper introduces a new test called Morpheus to find out. Think of Morpheus not as an art critic, but as a physics detective.

The Problem: The "Looks Good" Trap

Previously, to check if a video was "real," people just asked humans or AI chatbots: "Does this look like a real ball falling?"

The Flaw: This is like judging a magic trick by asking, "Did it look like a rabbit appeared?" The rabbit might have appeared, but maybe it was a fake rabbit made of paper.
The Issue: A video can look smooth and beautiful (aesthetic) but still break the laws of physics (e.g., a ball rolling uphill without being pushed, or a ball losing energy when it shouldn't). Current tests often miss these subtle errors because they only look at the "picture," not the "math."

The Solution: The Morpheus Test

The authors built a laboratory of 130 real-world experiments (like dropping balls, rolling cans, swinging pendulums, and bouncing objects). They filmed these carefully in a controlled room.

Then, they asked the video AI models to recreate these videos based on the first frame or a text description.

Instead of just asking, "Does it look real?", Morpheus asks: "Does the math work?"

Here is how the test works, using a simple analogy:

1. The "Trajectory" (The Path)

Imagine a ball falling. In the real world, gravity pulls it down in a very specific curve.

Old Way: Compare the AI's ball path to the real ball path pixel-by-pixel.
Morpheus Way: This is too strict. If the AI's ball is slightly heavier or the wind is slightly different, the path changes, but it's still a valid physics path. Morpheus doesn't care if the path is identical; it cares if the path follows the rules.

2. The "PINN" (The Physics Teacher)

The paper uses a special AI tool called a Physics-Informed Neural Network (PINN). Think of this as a strict physics teacher who knows the laws of motion by heart.

The teacher looks at the AI's video and tries to fit the ball's movement to the laws of Newton (like $F=ma$).
If the ball moves in a way that violates gravity or conservation of energy, the teacher gives it a bad grade.
If the movement follows the laws perfectly, the teacher gives it an A+.

3. The "Conservation" (The Energy Bank)

In the real world, energy is like money in a bank account. You can't create it or destroy it; you can only move it around (e.g., from height to speed).

Morpheus checks the "bank account" of the AI's video.
Real Video: The total energy stays constant (or decreases slightly due to friction, which is normal).
AI Video: Often, the "bank account" goes wild. The ball might suddenly gain energy out of nowhere or lose it too fast. Morpheus catches this instantly.

What Did They Find?

The results were a bit of a reality check for the AI community:

The "Pretty Lie": Even the most advanced AI models (like Veo3 and Kling) can make videos that look stunningly beautiful and smooth.
The Physics Failure: However, when Morpheus checked the math, almost all of them failed.
- They struggled to keep energy constant.
- They often made objects disappear, duplicate themselves, or just stop moving when they should be rolling.
- Even when given a video of a real ball falling to start with, the AI couldn't predict the next few seconds correctly according to physics.

The "Prompt" Power-Up

The researchers tried giving the AI better instructions (prompts) and showing it more frames (not just the start, but the end too).

Result: It helped a little bit. If you show the AI the start and the end of a video, it does a better job guessing the middle. But even with the best help, the models still couldn't match the physical accuracy of the real world.

The Bottom Line

The paper concludes that while these AI models are amazing artists, they are currently terrible physicists. They can paint a picture of a falling apple that looks perfect, but they don't actually understand why it falls or how fast it should go.

Until they pass the Morpheus test, we can't trust them to be "world models" for things like self-driving cars or robots, where understanding real physics is a matter of safety, not just art.

In short: The AI is great at faking the look of reality, but it hasn't yet learned the rules of reality.

Technical Summary: Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems

Problem Statement

Recent video generative models (VGMs), such as SORA, COSMOS, and Veo3, have achieved unprecedented visual fidelity, sparking interest in their potential as "world models" capable of understanding and predicting physical dynamics. However, current evaluation paradigms rely heavily on subjective human judgments, Vision-Language Model (VLM) assessments, or trajectory matching against ground truth. These methods suffer from significant limitations:

Subjectivity: Human and VLM evaluations are qualitative and prone to hallucinations, often failing to detect subtle violations of physical laws (e.g., incorrect mass distribution or force coupling) while rewarding visually plausible but physically incorrect motion.
Ambiguity of Trajectory Matching: Comparing generated trajectories to a single ground-truth trajectory is physically ambiguous. Multiple distinct initial conditions (e.g., varying mass or friction) can yield different but equally valid physical trajectories. Trajectory matching metrics often penalize valid physical realizations that simply diverge from a specific reference path.
Lack of Quantitative Physical Reasoning: Existing benchmarks focus on perceptual metrics (visual coherence, temporal consistency) rather than adherence to governing physical equations and conservation laws.

Consequently, there is a critical gap in rigorously quantifying whether VGMs truly encode Newtonian mechanics or merely mimic the appearance of physical motion.

Methodology: The Morpheus Framework

To address these gaps, the authors introduce Morpheus, a physics-informed evaluation framework designed to measure the ability of VGMs to comprehend Newtonian dynamics in controlled rigid-body settings. The framework consists of three core components:

1. Dataset and Conditioning

Real-World Data: The authors curated a dataset of approximately 130 real-world videos capturing nine distinct physical phenomena: falling, projectile motion, rolling, sliding, holonomic pendulum, double pendulum, bouncing, collision, and spring oscillation.
Controlled Conditions: Experiments are conducted under controlled conditions to isolate specific physical laws (e.g., conservation of energy, momentum, periodic motion).
Augmentation: To ensure robustness against specific visual setups, the dataset is augmented using style transfer (Cosmos-Transfer1) guided by object masks. This creates diverse visual variants while preserving the underlying physical dynamics, expanding the conditioning scenarios ten-fold.
Prompting Strategies: The framework evaluates models under three conditioning levels: textual prompts, single-image prompts, and multi-frame (video) prompts, to assess how input precision affects physical fidelity.

2. Trajectory Extraction and Preprocessing

Segmentation: Object masks are extracted using SAM-2 (Segment Anything 2) and Depth Anything for depth verification.
State Estimation: Centroids are tracked to generate 2D trajectories. Velocities and accelerations are estimated using central difference methods, followed by Savitzky-Golay smoothing and linear regression within sliding windows to reduce noise from tracking imperfections.
Filtering: A discard rate metric is computed to filter out videos with object disappearance, duplication, or insufficient motion, ensuring only valid trajectories are scored.

3. Physics-Informed Evaluation Metrics

Morpheus moves beyond pixel-level comparison to evaluate adherence to physical invariants and governing equations:

Dynamical Score: Utilizes Physics-Informed Neural Networks (PINNs) to fit the extracted trajectories to the known Ordinary Differential Equations (ODEs) of the specific physical system (e.g., $\ddot{y} + g = 0$ for free fall). The score is derived from the Normalized Mean Squared Error (NMSE) between the PINN's learned trajectory and the data, normalized to be scale-invariant.
Physical Invariance Score: Quantifies the stability of conserved quantities (e.g., total energy, angular momentum, oscillation period) over time. The score is calculated based on the standard deviation of these invariants; a perfect score (1.0) indicates constant conservation, while high variance indicates a violation of physical laws.

Key Contributions

Morpheus Framework: The introduction of a systematic, interpretable, and quantitative evaluation framework that leverages physics-informed neural networks and conservation laws to assess VGMs, moving beyond subjective or purely geometric metrics.
Comprehensive Benchmarking: A large-scale evaluation of state-of-the-art models (including closed-source models like Veo3, Kling, and COSMOS, and open-source models like Wan2.1, CogVideoX, and LTX-Video) across diverse Newtonian scenarios.
Empirical Findings on Physical Fidelity: The study reveals a substantial disconnect between visual realism and physical correctness. Even the best-performing models struggle to encode Newtonian mechanics accurately, despite generating aesthetically pleasing videos.
Analysis of Conditioning and Failure Modes: The paper demonstrates that multi-frame and keyframe conditioning significantly improves physical consistency compared to single-image prompts. It also identifies specific failure modes, such as "stillness" (objects failing to move) and object hallucination (duplication/disappearance), which are prevalent in current models.

Experimental Results

Performance Gap: Real-world videos achieve near-perfect scores (Dynamical Score: 0.96–0.99; Invariance Score: ≥0.90). In stark contrast, the best-performing VGMs achieve a combined score of only ~0.66, highlighting a significant gap in physical reasoning.
Impact of Conditioning:
- Keyframe Interpolation: Models conditioned on start and end frames (e.g., Wan2.1 Keyframe) perform best, achieving the highest aggregate scores and lowest discard rates. Defining boundary conditions constrains the generation space effectively.
- Multi-Frame vs. Single-Frame: Multi-frame conditioning outperforms single-frame extrapolation, as temporal context helps infer velocity and acceleration that are ambiguous in static images.
- Single-Frame Limitations: Even top-tier closed-source models struggle with single-image extrapolation, often resulting in high discard rates due to static or erratic motion.
Model Comparison: Open-source models like Wan2.1 (with keyframe conditioning) and Cosmos-Predict2 are competitive with, and sometimes outperform, proprietary models like Veo3 and Kling-2.5, suggesting that physical reasoning capabilities depend on architecture and conditioning strategies rather than just model scale or proprietary data.
VLM Evaluation Failure: The paper demonstrates that strong VLM evaluators (e.g., VideoPHY2-AUTOEVAL) fail to reliably distinguish real physics from generated physics. They often assign low scores to clean real-world recordings (hallucinating violations) and reward generated videos that violate specific physical laws, confirming the necessity of physics-informed metrics over VLM-based judgment.

Significance and Claims

The paper claims that Morpheus provides a necessary step toward aligning video generation with "world models" required for embodied AI and robotics. The authors argue that while visual fidelity has improved, current models are far from encoding Newtonian mechanics accurately enough to be trusted as simulators for physical interaction.

The significance of this work lies in its shift from perceptual evaluation to quantitative physical verification. By establishing that current models fail to respect fundamental conservation laws (energy, momentum) even in simple rigid-body scenarios, the paper underscores the limitations of treating VGMs as world models without further architectural or training innovations. The framework offers a reproducible stress test for core physical laws, providing a solid foundation for future research into models capable of reliable physical reasoning.