Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning

This paper introduces Physion-Eval, a large-scale benchmark featuring over 10,000 expert human reasoning traces that diagnose physical realism failures in state-of-the-art video generation models, revealing that the vast majority of generated videos violate real-world physical constraints.

The Gist

Imagine you have a magical movie studio that can create any video you can dream of. You ask it to "show me a glass of water falling off a table," and it spits out a beautiful, high-definition clip. It looks amazing! The lighting is perfect, the glass shimmers, and the motion is smooth.

But then, you watch closely. As the glass hits the floor, instead of shattering, it bounces like a rubber ball. Or maybe the water splashes upward before hitting the ground. Or perhaps the glass disappears for a split second and reappears on the other side of the room.

To a casual viewer, it might just look like a "glitch." But to a physicist, it's a violation of the universe's rulebook.

This is the problem Physion-Eval is trying to solve. Here is the paper explained in simple terms, using some everyday analogies.

1. The Problem: The "Magic Trick" vs. Reality

Current AI video generators (like Sora, Veo, or Kling) are incredible artists. They are like master painters who can mimic the look of the world perfectly. They know how light hits a surface or how a shadow falls.

However, they are terrible at understanding how the world actually works. They don't know that if you push a heavy box, it won't slide through a wall. They don't know that if you drop an egg, it won't turn into a bird. They are "hallucinating" physics.

The paper asks: "Do these AI movies follow the laws of physics, or are they just pretty lies?"

2. The Solution: A New "Physics Test"

The authors created a massive test called Physion-Eval. Think of this as a giant "Spot the Fake" game, but instead of looking for deepfakes (fake people), they are looking for fake physics.

They took real-world videos of things happening (like pouring coffee, cutting a cake, or a car crashing) and asked five of the smartest AI video models to recreate them. Then, they brought in 90 human experts (people with degrees in physics and engineering) to watch the AI videos and act as "Physics Detectives."

3. The Findings: The AI is Failing Hard

The results were shocking. The AI models are failing the physics test more often than you'd expect.

• The "Third-Person" View (Watching from a distance): Even when watching from a safe distance, 83% of the AI videos had at least one physical mistake.
• The "First-Person" View (Like wearing a GoPro): This was even worse. 93.5% of the videos had mistakes.

Why is the first-person view worse?
Imagine trying to walk through a crowded room while blindfolded. It's hard. Now imagine an AI trying to simulate walking through a room while holding a cup of coffee. The AI gets confused by the camera moving, the objects bumping into it, and the liquid sloshing. It's like a novice driver trying to parallel park while the car is spinning; the AI loses track of where objects are and how they should move.

4. The "Robot Judge" vs. The Human Eye

The researchers also asked AI "Critics" (super-smart Large Language Models like Gemini or GPT) to watch the videos and tell them if the physics was real.

The Result: The AI Critics were terrible at this job.

• The Analogy: Imagine a robot librarian trying to spot a typo in a book written in a language it doesn't fully understand. It might see the words look "pretty" and say, "This looks correct!" even if the sentence makes no sense.
• The human experts could spot the errors instantly. The AI critics, however, often missed the obvious mistakes or made up fake reasons for them (hallucinations). They couldn't tell the difference between a "real" physics error and a "fake" one.

5. What Did They Find? (The "Glitch Menu")

The experts categorized the mistakes into 22 different types. Here are a few examples of what the AI got wrong:

• The "Ghost" Object: A cup disappears for a second and then reappears.
• The "Anti-Gravity" Splash: Water splashes up into the air before hitting the ground.
• The "Magic" Cut: A knife cuts through a tomato, but the tomato doesn't separate; it just gets a weird scar.
• The "Time Travel" Effect: An object moves backward in time without a cause.

6. Why Does This Matter?

You might think, "So what? It's just a fun video."

But the paper argues that we want to use these AI models for serious things in the future, like:

• Training Robots: If you train a robot on a video where a cup floats through a table, the robot will try to walk through tables and break.
• Movie Making: If a director uses AI to generate a scene where a car crashes, but the physics are wrong, the audience will feel "something is off," even if they can't say why.
• Scientific Simulation: We need accurate simulations to understand real-world events.

The Big Takeaway

Physion-Eval is a wake-up call. It tells us that while AI video generators are getting better at looking pretty, they are still very bad at understanding reality.

They are like actors who can memorize their lines perfectly but don't understand the plot of the movie. They know how to stand and look dramatic, but they don't know that if they jump off a cliff, they should fall down, not float up.

The paper provides a new tool (the dataset and the benchmark) to help developers fix this, so that in the future, the AI movies we watch will not just look real—they will be real.

Technical Summary

1. Problem Statement

Video generation models (e.g., Sora, Veo, Kling) are rapidly evolving into "world simulators" for applications in storytelling, embodied AI, and simulation. However, a critical gap remains: do these generated videos obey the laws of physics?

Existing evaluation methods rely heavily on:

• Automated metrics: (e.g., FVD, LPIPS) which measure perceptual quality or distributional similarity but fail to detect subtle violations of dynamics, contact, or causality.
• Coarse human judgments: (e.g., preference ranking or rubric-based checks) which lack the granularity to diagnose when and why a physical failure occurs.
• Limited scope: Prior benchmarks often focus on exocentric (third-person) views, neglecting egocentric (first-person) scenarios crucial for embodied AI, and rely on automated critics that correlate poorly with human perception.

The core problem is the lack of a large-scale, fine-grained benchmark that uses expert human reasoning to diagnose physical realism failures in both exocentric and egocentric videos.

2. Methodology

The authors introduce Physion-Eval, a comprehensive benchmark and evaluation framework built on three pillars:

A. Dataset Curation

• Source Data: The benchmark is derived from real-world videos in WISA-80K (exocentric) and EPIC-KITCHENS (egocentric).
• Physical Categories: It covers 22 fine-grained physical phenomena, including dynamics, thermodynamics, optics, rigid-body interactions, deformation, fluid flow, and thermal effects.
• Generation: The authors generated 12,718 videos using five state-of-the-art Text-and-Image-to-Video (TI2V) models: Sora 2, Veo 3.1, Kling 2.5, Hailuo 2.3, and Wan 2.2.
• Standardization: All videos are standardized to 16:9 aspect ratio, 720×1280 resolution, and 4–8 seconds duration, with audio removed to focus on visual physics.

B. Human Evaluation Studies

The paper conducts two complementary human studies:

1. Perceptual Detection (Untrained Viewers): 16 untrained viewers evaluated 1,500 pairs of real vs. generated clips to measure the "perceptual gap." They used Youden's J statistic to quantify the drop in perceived physical realism compared to real videos.
2. Expert Reasoning Benchmark (Physion-Eval): 90 STEM-expert annotators (with physics/engineering backgrounds) performed deep diagnostic analysis on the generated videos.
   • Workflow: A three-expert adjudication process (two independent annotators + one senior reviewer).
   • Annotations: Each trace includes:
     • Temporal Grounding: Precise glitch localization (0.1s precision).
     • Structured Taxonomy: Classification into 22 categories (e.g., Contact Failure, Object Permanence, Causal Violation).
     • Natural Language Reasoning: Explanations of the violated physical principles.
   • Scale: The final dataset contains 10,990 expert reasoning traces.

C. Automated Critic Evaluation

The authors benchmarked 10 state-of-the-art Multimodal Large Language Models (MLLMs) (including Gemini 3.0 Pro, GPT-5.2, Qwen3-VL, and Cosmos-Reason) against the human ground truth to test if automated critics can detect and explain these physical glitches.

3. Key Contributions

1. Physion-Eval Benchmark: The first large-scale dataset of temporally grounded human reasoning annotations for diagnosing physical realism failures. It spans 12,718 videos and 10,990 expert traces across 22 physical categories.
2. Discovery of a "Physical Realism Gap": The study reveals that current models fail to model physics reliably, with failure rates of 83.3% (exocentric) and 93.5% (egocentric).
3. Human vs. MLLM Discrepancy: A systematic demonstration that current MLLM critics fail to detect physical glitches that are obvious to untrained humans, particularly in egocentric views.
4. Taxonomy of Failures: A structured classification of physical failures (e.g., temporal coherence breakdown, material inconsistency, causal violations) that enables precise diagnosis of model weaknesses.

4. Key Results

A. Prevalence of Physical Glitches

• Failure Rates: Across five leading models, 83.3% of exocentric and 93.5% of egocentric generated videos contain at least one human-identifiable physical glitch.
• Glitch Density: On average, videos contain 1.15 to 1.92 glitches per clip, indicating that multiple violations often occur simultaneously.
• Severity: Average severity scores range from 2.69 to 3.86 (on a 1–5 scale), indicating substantial physical inconsistencies.

B. Human vs. MLLM Performance

• Detection Capability: Untrained human viewers achieved a Youden's J statistic of 24.9%–37.1% (exocentric) and 48.4%–61.8% (egocentric).
• MLLM Failure: The best MLLM critics (Gemini 3.0 Pro) achieved only 19.1% (exocentric) and 9.8% (egocentric).
• Reasoning Quality: MLLMs frequently hallucinate explanations (e.g., attributing errors to non-existent shadows) and provide incorrect temporal localization. They struggle to identify when a violation occurs, often missing subtle, time-dependent dynamics.

C. Model-Specific Insights

• Exocentric: Kling 2.5 showed the lowest glitch density and severity.
• Egocentric: Wan 2.2 (open-source) surprisingly performed best in the egocentric setting, while commercial models (often optimized for cinematic aesthetics) struggled more with the stable object dynamics required in first-person views.
• Common Failure Modes:
  • Exocentric: Temporal coherence breakdown and material/state inconsistencies.
  • Egocentric: Temporal coherence breakdown and object permanence violations.

D. Impact of Dynamics

• Models struggle significantly more with high-dynamics scenarios (rapid motion, collisions, irreversible state changes) compared to static or slow-moving scenes.
• MLLM critics show almost no ability to detect glitches in low-to-medium intensity/dynamics regimes.

5. Significance and Future Directions

• New Standard for Evaluation: Physion-Eval moves beyond "does it look real?" to "does it act real?" by providing a diagnostic tool for physical realism.
• Guidance for Model Development: The structured failure traces can be used to train physics-grounded video critics and enable closed-loop self-improvement for video generation models.
• Embodied AI & Simulation: The findings highlight that current models are not yet reliable for training embodied agents or simulating complex physical interactions, as they fail to respect conservation laws, causality, and object permanence.
• Limitations: The study focuses on visually observable cues; latent quantities (force, energy) are inferred. It also primarily covers single dominant interactions rather than complex multiphysics environments.

In conclusion, Physion-Eval demonstrates that while video generation models have achieved impressive visual fidelity, they fundamentally lack a compositional understanding of physical laws. The benchmark provides the necessary data and reasoning traces to bridge this gap, guiding the next generation of physically grounded AI.