🤖 AI

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

This paper introduces Physics Question Scene Graph (PQSG), a hierarchical, graph-based evaluation pipeline that uses vision-language models to assess the physical plausibility of text-to-video generations with fine-grained precision, validated through the new FinePhyEval dataset which demonstrates PQSG's superior correlation with human judgments compared to prior methods.

Original authors: Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Atin Pothiraj, Jaemin Cho, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a robot chef to cook a meal based on a recipe you wrote. The robot is amazing at chopping vegetables and plating the food; the final dish looks beautiful and realistic. However, when it tries to boil water, the water turns into ice cubes that float upward, or the soup disappears into the pot instead of bubbling. The robot failed the laws of physics, even though the picture looks good.

This is exactly the problem the paper "Physics Question Scene Graph (PQSG)" tackles. While AI video generators are getting better at making videos that look real, they often fail at making videos that act real. They break basic rules like gravity, how liquids flow, or how solid objects bounce.

Here is a simple breakdown of how the authors fixed the way we test these AI robots.

1. The Problem: The "One-Size-Fits-All" Grade

Previously, if you wanted to grade an AI video, you might ask a human (or a computer) for a single score, like "7 out of 10."

The Flaw: If a video gets a "7," you don't know why it failed. Did the robot forget to put the spoon in the soup? Did the spoon float? Did the soup turn into jelly?
The Paper's Insight: You can't just give a single grade. You need to check the video step-by-step, like a teacher grading a math test by checking every single step of the calculation, not just the final answer.

2. The Solution: The "Detective's Checklist" (PQSG)

The authors created a system called Physics Question Scene Graph (PQSG). Think of this as a hierarchical detective checklist that an AI uses to inspect a video.

Instead of asking, "Is this video good?", the system breaks the video down into a tree of specific questions. Crucially, these questions are connected like a flowchart:

Level 1: The Objects (The Cast)
- Question: "Is there a paper towel?"
- Logic: If the answer is No, the detective stops. There is no point in asking if the paper towel absorbs liquid if the towel doesn't exist.
Level 2: The Actions (The Plot)
- Question: "Did the grabber tool let go of the paper towel?"
- Logic: If the towel exists but wasn't dropped, the physics test hasn't even started yet.
Level 3: The Physics (The Laws of Nature)
- Question: "Did the paper towel absorb the liquid, or did it dissolve?"
- Logic: Only now do we check if the laws of physics were followed.

The "Graph" Part:
The "Scene Graph" is just a fancy way of saying these questions are linked. If the first question fails, the system automatically knows the later questions are invalid. This prevents the AI from getting confused or "hallucinating" (making things up) about physics that never happened because the object wasn't there.

3. The New Dataset: "FinePhyEval"

To test this new detective checklist, the authors built a new dataset called FinePhyEval.

They took 65 tricky prompts (like "a ball drops on a pillow") and generated videos using top-tier AI models (like Sora 2, Veo 3, and Wan 2.1).
They then had humans watch these videos and answer the exact same checklist questions.
This created a "Gold Standard" to see if their new AI detective was as good as a human.

4. What They Found

When they ran their new system (PQSG) against the old ways of grading videos, they found:

Better Grades: PQSG correlated much better with human opinions. It knew exactly why a video was bad, not just that it was "okay."
The "Closed-Source" Advantage: The private, expensive models (Sora 2, Veo 3) were better at following physics than the open-source ones (Wan 2.1).
The Weak Spot: Even the best AI models are great at making objects (a ball) and actions (dropping it), but they still struggle with the physics (does the ball bounce realistically?).
The AI Detective's Limit: The system uses a smart AI (a Vision-Language Model) to ask and answer the questions. While this AI is great at spotting objects, it still makes mistakes on complex physics, often guessing "Yes" when the answer is "No" (a "yes-bias"). It's like a detective who is very confident but sometimes wrong about the science.

5. The Bonus: Fixing the Video

The paper also showed that this checklist isn't just for grading; it's for fixing.

They used the checklist to tell the video generator what went wrong (e.g., "The smoke went sideways instead of up").
They fed this feedback back into the AI to generate a new video.
Result: The video got significantly better after just one round of this "correction."

Summary

The paper introduces a new way to grade AI videos that acts like a structured, step-by-step inspector. Instead of giving a vague score, it asks specific questions about objects, actions, and physics in a logical order. This helps us understand exactly where AI video generators are breaking the laws of physics and gives us a tool to help them fix those mistakes.

Technical Summary: Physics Question Scene Graph (PQSG)

Problem Statement

While text-to-video generation models have achieved significant progress in photorealism, they frequently fail to adhere to fundamental physical laws (e.g., solid mechanics, fluid dynamics, optics). A critical bottleneck in advancing these models is the lack of reliable, fine-grained, and automated evaluation methods capable of localizing specific physical violations. Existing metrics often provide coarse, aggregate scores that fail to capture the logical dependencies inherent in physical reasoning (e.g., an object must exist and perform an action before its physical interaction can be judged). Furthermore, current methods struggle to distinguish between generation failures (e.g., an object is missing) and physical plausibility failures (e.g., an object falls unrealistically), often being "fooled" by videos that are visually consistent but physically impossible.

Methodology: Physics Question Scene Graph (PQSG)

To address these limitations, the authors introduce Physics Question Scene Graph (PQSG), a hierarchical, question-based evaluation pipeline. PQSG decomposes the evaluation of a generated video into a Directed Acyclic Graph (DAG) of atomic verification questions generated from the text prompt.

Core Components

Hierarchical Structure: The graph organizes questions into three distinct categories:
- Object: Verifies the presence and attributes of key objects mentioned in the prompt.
- Action: Verifies that objects are performing the correct actions specified in the prompt.
- Physics: Evaluates the plausibility of interactions and dynamics (e.g., deformation, absorption, gravity) which are often implicit physical commonsense rather than explicit prompt instructions.
Logical Dependencies (Edges): The graph enforces strict logical dependencies. Questions are queried only if their prerequisite parent nodes are answered affirmatively. For instance, if an "Object" node is answered "No" (the object is missing), all subsequent "Action" and "Physics" nodes related to that object are automatically marked "No" without being queried. This prevents hallucination and ensures contextually valid queries.
Two-Stage Pipeline:
- Question Generation (QG): A Vision-Language Model (VLM) is prompted with in-context examples to generate the hierarchical graph of questions from a text prompt.
- Question Answering (QA): The generated video and the PQSG graph are passed to a VLM. The model answers questions sequentially, providing binary judgments ("Yes"/"No") supported by reasoning. The final score is the ratio of "Yes" responses, with separate scores calculated for Object, Action, and Physics categories.

Dataset: FinePhyEval

To validate PQSG and benchmark subtasks, the authors constructed FinePhyEval, a dataset comprising:

195 human-annotated prompt-video pairs.
Prompts: Sourced from Physics-IQ, designed to test physical understanding.
Videos: Generated by diverse state-of-the-art models: Sora 2, Veo 3, Wan 2.1, and Cosmos-Predict2.5-14B.
Annotations: Includes human Likert-scale judgments (1-5) across four categories (Object, Action, Physics, Overall) and ground-truth annotations for the QG and QA subtasks (444 QA pairs).

Key Results

1. Correlation with Human Judgment

PQSG demonstrates superior correlation with human overall Likert scores compared to prior metrics (VideoScore, VideoPhy-2, PhyGenEval, DSG).

PQSG (w/ GPT-5.5): Achieved a Pearson's $r$ of 0.478, outperforming the next best baseline (Direct VQA at 0.382).
Granularity: Unlike aggregate metrics, PQSG provides fine-grained feedback, pinpointing exactly which category (Object, Action, or Physics) caused a failure.
Human Alignment: The study found that human "Overall" scores correlate most strongly with the Physics category ( $r=0.85$ ), underscoring that physical realism is the primary driver of human quality assessment.

2. Model Benchmarking

Using PQSG, the authors ranked video generation models:

Proprietary vs. Open Source: Proprietary models (Sora 2, Veo 3) significantly outperformed open-source models (Wan 2.1, Cosmos-14B) in physical realism.
Category Performance: All models struggled more with Action and Physics generation than with Object generation. For example, Wan 2.1 scored 0.53 on Action and 0.46 on Physics, compared to 0.86 on Object.

3. Subtask Evaluation (QG and QA)

Question Generation: VLMs (Gemini-2.5-Pro, GPT-5.5) achieved high precision (>92%) and recall (>95%) in generating verification questions compared to human annotations.
Question Answering: While VLMs performed well on Object and Action categories (accuracy ~88%), they struggled significantly with Physics questions (accuracy ~64%).
Failure Modes: QA models exhibited a "yes-bias," often hallucinating physical interactions (e.g., smoke rising correctly) when the video showed otherwise. This suggests current VLMs lack robust spatio-temporal physical reasoning.

4. Iterative Refinement

The authors demonstrated that PQSG can be integrated into an iterative loop to improve video generation. By using PQSG feedback to refine prompts (via GPT-5.5) and re-generating videos, the average PQSG score increased by nearly 15% in a single iteration, plateauing at ~82% after two iterations. This was achieved without modifying model architecture or retraining.

Significance and Claims

The paper claims that PQSG addresses a critical gap in video generation evaluation by providing a structured, fine-grained, and automated metric for physical plausibility.

Diagnostic Capability: It moves beyond "is the video good?" to "why is the video physically implausible?", isolating failures in object presence, action execution, or physical laws.
Dependency Modeling: By enforcing logical dependencies in the question graph, PQSG ensures evaluations are contextually valid and reduces hallucination, a feature missing in prior unstructured approaches.
Benchmarking Utility: The introduction of FinePhyEval provides a standardized benchmark for evaluating the physical reasoning capabilities of both video generation models and the VLMs used to evaluate them.
Practical Application: The framework is shown to be a practical tool for improving video generation quality through prompt refinement, offering a pathway to higher-fidelity simulations for robotics and embodied AI without requiring model retraining.

The authors remain modest regarding the current limitations, noting that the system's performance is bounded by the physical reasoning capabilities of the underlying VLMs, particularly in the QA stage where accuracy on physics questions remains below human performance.