🤖 AI

PhyGround: Benchmarking Physical Reasoning in Generative World Models

The paper introduces PhyGround, a comprehensive benchmark featuring 250 curated prompts, a 13-law taxonomy, and a specialized VLM judge (PhyJudge-9B) to rigorously evaluate and diagnose the physical reasoning capabilities of generative video models through large-scale, quality-controlled human annotation.

Original authors: Juyi Lin, Arash Akbari, Yumei He, Lin Zhao, Haichao Zhang, Arman Akbari, Xingchen Xu, Zoe Y. Lu, Enfu Nan, Hokin Deng, Edmund Yeh, Sarah Ostadabbas, Yun Fu, Jennifer Dy, Pu Zhao, Yanzhi Wang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juyi Lin, Arash Akbari, Yumei He, Lin Zhao, Haichao Zhang, Arman Akbari, Xingchen Xu, Zoe Y. Lu, Enfu Nan, Hokin Deng, Edmund Yeh, Sarah Ostadabbas, Yun Fu, Jennifer Dy, Pu Zhao, Yanzhi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot that can dream up videos. It can make a cat chase a laser pointer or a car drive through a city. But here's the problem: sometimes, in its dreams, the cat floats like a balloon, the car drives through a brick wall, or the water in a cup disappears into thin air. The video looks cool, but it breaks the basic rules of how our real world works.

The paper PhyGround is like a strict, very detailed "physics report card" for these video-making robots. The researchers wanted to stop just asking, "Does this look cool?" and start asking, "Does this actually obey the laws of physics?"

Here is how they built this report card, explained simply:

1. The Problem with Old Report Cards

Before this, checking if a video followed physics was like a teacher giving a student a single grade for a whole science test. If the student got the math right but failed the chemistry, the teacher just gave a "B" for the whole test. You didn't know what they got wrong.

Also, the old tests relied on a few tired people watching hours of videos. Sometimes, the grader gets sleepy, gets confused, or just guesses. And the computer programs used to grade the videos were like general knowledge bots—they were good at spotting if a picture looked "pretty," but they couldn't tell if a shadow was pointing the wrong way.

2. The New Solution: A Physics Detective Kit

The researchers created PhyGround, which is like a detective kit with three special tools:

Tool #1: The "Specific Law" Checklist
Instead of one big grade, they broke physics down into 13 specific laws (like Gravity, Shadows, Fluids, and Collisions).
- The Analogy: Imagine grading a soccer game. Instead of just saying "Good Game," the referee checks specific things: "Did the ball stay on the field?" "Did the goalie touch the ball with their hands?" "Did the players run the right distance?"
- They wrote 250 specific prompts (instructions) for the robots, like "Throw a ball so it hits a wall and bounces back." This forces the robot to try a specific physical action, so the grader knows exactly what to look for.
Tool #2: The "Super-Grader" Human Team
They didn't just ask a few friends to watch the videos. They recruited 459 people (a mix of physics experts and regular people) to act as a massive jury.
- The Analogy: Instead of one judge deciding the winner of a talent show, they had a stadium full of people voting. To make sure no one was just clicking buttons randomly, they used strict rules: everyone had to sit at a desk (no phones), watch the videos carefully, and they were only asked to grade a small number of videos so they wouldn't get tired.
- They found that when you have this many people, the results are incredibly stable and reliable.
Tool #3: The "Physics-Specialist" Robot Judge (PhyJudge-9B)
Humans are great, but they are slow. So, the researchers trained a new AI robot, called PhyJudge-9B, using the answers from the 459 humans.
- The Analogy: Think of this like a student who studied the answer key from the 459 human judges. Now, this student robot can grade thousands of videos instantly.
- The paper shows this robot is much better than other famous AI judges (like Gemini). While other robots might get confused and say, "Oh, that shadow looks weird, I'll give it a zero!" even if it's fine, PhyJudge-9B learned to look at the specific rules (like "Does the shadow move with the object?") and grade much more accurately.

3. What They Found

When they tested 8 different video-making robots with this new system, they found some surprising things:

No Robot is Perfect: None of the robots got a perfect score. They all still break physics rules sometimes.
Specialists vs. Generalists: Some robots were great at making water flow (Fluids) but terrible at making solid objects bounce (Solid-Body). Others were the opposite.
The "Hidden" Flaws: If you just looked at the "Overall Score," two robots might look tied. But when you looked at the specific laws, one was actually failing at gravity while the other was failing at shadows. This detailed view helps developers know exactly what to fix.

The Bottom Line

PhyGround is a new, open-source way to test video AI. It moves away from vague "looks good" scores and uses a massive team of humans and a specialized robot judge to check if the videos follow the 13 specific laws of physics. It's like giving video AI a final exam where you can see exactly which questions they got wrong, helping them learn to dream up worlds that feel truly real.

Technical Summary: PhyGround – Benchmarking Physical Reasoning in Generative World Models

1. Problem Statement

Generative world models are increasingly utilized for video generation, with the expectation that they capture the physical rules governing real-world dynamics. However, current models frequently violate basic physical principles (e.g., objects floating without support, fluids disappearing, or shadows conflicting with light sources), undermining their authenticity and applicability in domains requiring reliable simulation.

Existing physics-focused video benchmarks face three critical limitations:

Coarse Evaluation Frameworks: They often operate at the domain level (e.g., a single "fluid dynamics" score), masking specific law-level failures and failing to distinguish between minor deviations and severe violations.
Annotation Biases and Fatigue: Human evaluation is vulnerable to response biases, fatigue, and ordering effects, particularly when small annotator pools are used without rigorous experimental design.
Insufficient Automated Evaluation: Standard metrics (FVD, SSIM) measure distributional similarity rather than physical adherence. Existing VLM-as-judge approaches often rely on closed-source models with evolving architectures, making results difficult to reproduce or audit, and lack specialization for fine-grained physical diagnosis.

2. Methodology

PhyGround addresses these challenges through a criteria-grounded benchmark design, a large-scale quality-controlled human study, and an open-source specialized judge.

2.1 Benchmark Design

Prompt Suite: The benchmark consists of 250 curated prompts derived from four sources (VideoPhy-2, Physics-IQ, OpenVid, WorldModelBench). Unlike previous works using implicit prompts, PhyGround augments each prompt with an explicit expected physical outcome (e.g., specifying that a ball must fall off a table) to ensure the model attempts a scorable physical event.
Physics Taxonomy: The evaluation decomposes physical reasoning into 13 distinct laws across three domains:
- Solid-Body Mechanics (6 laws): Gravity, Inertia, Momentum, Impenetrability, Collision, Material properties.
- Fluid Dynamics (5 laws): Buoyancy, Displacement, Flow dynamics, Boundary interaction, Continuity.
- Optics (2 laws): Reflection, Shadow.
Scoring Mechanism: Each law is operationalized via 2–3 observable sub-questions (e.g., "Do unsupported objects float upward?"). Annotators and judges score each law on a 1–5 Likert scale, capturing the severity of violations rather than binary pass/fail judgments.

2.2 Large-Scale Human Evaluation

Scale and Protocol: The study recruited 459 annotators (including 35 physics experts and non-experts) to evaluate 2,000 videos generated by 8 models. This resulted in 5,796 complete annotations and over 37,400 fine-grained labels.
Quality Control: Drawing on social science experimental design, the study employed randomization of video order and annotator assignment, capped workloads (~12.6 videos per annotator), and hid model identities to reduce demand effects. A two-round quality-control pipeline filtered low-quality annotations based on score consistency, dimension discriminability, peer agreement, and behavioral engagement, retaining 352 annotators.
Reliability: The retained dataset exhibited high split-half model-ranking correlations (Spearman's $\rho > 0.90$ ), ensuring stable aggregate rankings.

2.3 Automated Judge: PhyJudge-9B

Training: PhyJudge-9B is a 9B-parameter Vision-Language Model (VLM) fine-tuned via Supervised Fine-Tuning (LoRA) on the structured human annotations from PhyGround.
Architecture: The model is trained to output scores for specific laws based on video frames, augmented prompts, and a criterion-specific rubric containing the sub-questions.
Reproducibility: Unlike closed-source judges, PhyJudge-9B is fully open-source with fixed weights, allowing for independent auditing and reproducibility.

3. Key Results

The benchmark evaluated eight modern video generation models (one closed-source: Veo-3.1; seven open-source).

Overall Performance: No model achieved strong physical adherence. The highest "Overall" score (combining general and physics dimensions) was 3.28/5 (approx. 66%), shared by Wan2.2-27B-A14B and Veo-3.1.
Domain Decomposition: Holistic scores obscured significant domain-level differences:
- Wan2.2-27B-A14B excelled in Solid-Body (3.23) and temporal coherence but lagged in Fluids (3.18).
- Veo-3.1 led decisively in Fluid (3.65) and Optical (3.69) domains but scored lower in Solid-Body (3.12).
- Cosmos-Predict2.5-2B showed a "spiky" profile, ranking 7th overall but achieving a high Optical score (3.35) while struggling with Fluids (2.77).
Difficulty Profile: Aggregated across models, Solid-Body mechanics (2.79) proved the most challenging domain, followed by Fluid (3.08) and Optical (3.24). Within Solid-Body, Momentum and Collision were the hardest laws, suggesting current models struggle with contact resolution and rebound dynamics.
Judge Performance: PhyJudge-9B achieved a significantly lower aggregate relative bias compared to the closed-source Gemini-3.1-Pro (3.3% vs. 16.6%). It also outperformed other baselines (Claude, Qwen) and demonstrated that adding sub-questions to prompts significantly improved the accuracy of VLM judges, whereas Chain-of-Thought (CoT) reasoning did not consistently yield further benefits.

4. Contributions

Criteria-Grounded Per-Law Taxonomy: A novel evaluation framework that decomposes physical reasoning into 13 specific laws with observable sub-questions, enabling diagnostic insights that holistic scores cannot provide.
Large-Scale, Quality-Controlled Human Dataset: The largest reported human-annotator pool (459 annotators) for a physics-focused video benchmark, designed with social science protocols to ensure reliability and stability ( $\rho > 0.90$ ).
PhyJudge-9B: An open, physics-specialized VLM judge that achieves substantially lower bias than state-of-the-art closed-source models and supports reproducible, auditable automated evaluation.
Comprehensive Benchmark Release: Full release of 250 prompts, human annotations, model checkpoints, and evaluation code.

5. Significance

PhyGround advances the evaluation of generative world models by shifting from coarse, holistic assessments to fine-grained, law-specific diagnostics. By revealing that models often possess distinct strengths and weaknesses across different physical constraints (e.g., a model may be strong in fluid dynamics but weak in collision physics), the benchmark provides a solid foundation for targeted model improvement. Furthermore, the release of PhyJudge-9B addresses the critical need for reproducible and auditable automated evaluation, moving the field away from reliance on opaque, closed-source APIs. The work underscores that while current video generation models show promise, substantial gaps remain in their ability to simulate complex physical interactions, particularly in contact dynamics and fluid continuity.