💻 computer science

Physics-IQ Verified

This paper introduces Physics-IQ Verified, an improved benchmark that systematically audits and refines the original Physics-IQ dataset to provide a more reliable evaluation of physical understanding in video generative models by enhancing prompt quality, ground-truth accuracy, and implementing a balanced sample-level scoring system.

Original authors: Tim Rädsch, Yuki M Asano, Hilde Kuehne, Stefan Bauer, Priyank Jaini, Robert Geirhos, Carsten T. Lüth

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tim Rädsch, Yuki M Asano, Hilde Kuehne, Stefan Bauer, Priyank Jaini, Robert Geirhos, Carsten T. Lüth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a student's science project. The student has to predict what happens next in a physics experiment, like a ball falling or a magnet pulling a paperclip.

For a long time, the "Physics-IQ" benchmark was the standard test for video-generating AI. It worked like this: You showed the AI a starting picture and a sentence (a prompt), asked it to generate the next few seconds of video, and then compared its video to a real video of the experiment. If the AI's video looked like the real one, it got a good grade.

However, the authors of this paper, Physics-IQ Verified, realized the grading system itself was flawed. They found that the test was sometimes grading the AI on things that had nothing to do with understanding physics. They decided to audit the test, fix the errors, and create a "Verified" version.

Here is a simple breakdown of the three main fixes they made, using everyday analogies:

1. Fixing the "Exam Questions" (Prompt Quality)

The Problem: The original test questions (prompts) were sometimes vague or confusing.

Analogy: Imagine asking a student, "What happens with the ball?" without telling them if the ball is red or blue, or if it's being dropped or thrown. If the student guesses the wrong color, they might get the physics right but fail the test because their description didn't match the specific setup.
The Fix: The authors rewrote the questions to be crystal clear. They added specific details about the scene, the camera angle, and exactly what actions should happen, while removing confusing or contradictory instructions. They also made sure the questions were written in a way that the specific AI models could understand best.
Result: The AI isn't being penalized for guessing the wrong color or camera angle; it's being graded strictly on whether it understands the physics of the movement.

2. Cleaning the "Visual Noise" (Artifact Removal)

The Problem: The real-world videos used for comparison (the "answer key") sometimes had accidental glitches.

Analogy: Imagine grading a student's drawing of a falling apple. But, in the teacher's reference photo, a fly buzzed across the lens, or a shadow flickered because the lightbulb was loose. If the student's drawing didn't include the fly or the flicker, the computer grading system would mark it wrong, even though the apple fell perfectly. The "noise" was confusing the grade.
The Fix: The authors went through the reference videos and digitally "erased" these accidental glitches (like a rotating stand that wasn't part of the experiment or a camera shake). They ensured the "answer key" only showed the actual physical event.
Result: The AI is no longer punished for failing to predict random, accidental events that no one could have predicted.

3. Changing the "Report Card" (Scoring System)

The Problem: The original way of calculating the final score was a bit like averaging a whole semester's grades into one big number, which hid specific failures.

Analogy: Imagine a student gets a 100% on a hard math test but a 0% on an easy spelling test. If you just average the two, you get a 50%. You can't tell if they are bad at spelling or just had a bad day. The original test did this by averaging scores across the whole dataset, which meant some easy experiments weighed the same as hard ones, and some failed experiments were hidden in the average.
The Fix: They created a new scoring system that looks at every single experiment individually and gives them equal weight. It's like giving a report card for every single question on the test, then averaging those.
Result: This makes it much easier to see exactly where an AI is failing. Is it bad at fluid dynamics? Is it bad at magnets? The new score tells you exactly that.

What Happened When They Re-Tested?

The authors took six different AI video generators and ran them through both the old test and the new "Verified" test.

The Rankings Changed: Just like when a teacher re-grades a test with clearer instructions and a better answer key, the "class rankings" shifted. Some AI models that looked good on the old test dropped in rank because they were actually relying on the old test's flaws. Other models that were previously underrated moved up because they actually understood the physics better.
The Conclusion: The new "Physics-IQ Verified" benchmark gives a more honest and accurate picture of which AI models are truly learning how the physical world works, rather than just memorizing patterns or getting lucky with vague questions.

In short, the paper didn't invent a new AI; it invented a better ruler to measure the ones we already have, ensuring we are measuring what we think we are measuring.

Technical Summary: Physics-IQ Verified

Problem Statement

Video Generative Models (VGMs) are increasingly positioned as "world models" capable of simulating physical reality for robotics and general visual reasoning. However, assessing whether these models have genuinely learned physical reasoning or merely learned to produce plausible-looking motion remains a critical challenge. The existing Physics-IQ benchmark [1] was designed to address this by comparing model-generated video continuations against ground-truth recordings of real-world physical experiments.

Despite its adoption as a standard evaluation protocol, the authors identify that the original Physics-IQ benchmark contains systematic measurement errors that confound the assessment of physical understanding. These errors arise from three primary sources:

Ambiguous Prompts: Original text prompts often lack clarity, contain factual errors, omit key information, or fail to adhere to model-specific prompting best practices. This introduces degrees of freedom in the output that are orthogonal to physical understanding, inflating metric variance.
Spurious Metric Activations (Artifacts): Ground-truth videos contain visual events unrelated to the target physical phenomena (e.g., rotating apparatus parts, camera noise, or recording errors). These "artifacts" trigger activation maps used by the benchmark's metrics, causing the evaluation to measure unrelated events rather than the physical effect itself.
Flawed Score Aggregation: The original score aggregates metrics at the dataset level. This approach obscures sample-level failure modes and assigns unequal weight to samples based on their physical variation, potentially down-weighting experiments with low natural variability and up-weighting those with high variability.

Methodology

The authors propose Physics-IQ Verified, a refined benchmark that systematically audits and corrects the original evaluation pipeline. The methodology involves three distinct improvements:

1. Improving Prompt Quality

The authors re-engineered the text prompts to function as precise "exam questions" that clearly specify the experimental setup and the catalyst for the physical phenomenon without revealing the outcome.

Correction of Unclear Prompts: Prompts were screened and corrected for four severity levels: factual incorrectness, temporal impreciseness, omitted key information, and vague language.
Templated Structure: To ensure consistent conditioning across different VGMs, prompts were decomposed into six structured fields: SETUP, SCENE, ACTION, CAM, STYLE, and SCOPE.
- CAM, STYLE, and SCOPE are fixed fields ensuring a realistic scientific demonstration style, static camera guidance, and a constraint against hallucinating new actors.
- Instructions were rewritten in positive terms to avoid negation, which many models handle poorly.

2. Cleaning of Artifacts

The authors introduced a targeted removal strategy for "spurious metric activations" in the ground-truth videos.

Artifact Definition: Artifacts are defined as metric activations caused by visual events not part of the observed physical effect. They are categorized as deterministic (predictable, e.g., rotating machinery) or non-deterministic (random recording errors).
Removal Strategy: Using manual annotations, artifacts are removed via frame freezing:
- Post-effect removal: Frames after the physical phenomenon ends are frozen to remove tail-end artifacts.
- Mid-effect removal: Spatial regions containing artifacts during the phenomenon are frozen to isolate the physical effect.
- This process ensures metrics assess the physical phenomenon rather than unrelated visual noise.

3. Improving Metric Aggregation

The authors redefined the scoring mechanism to operate at the sample level rather than the dataset level.

Sample-Level Scoring: A score is calculated for each individual video sample ( $n$ ) by averaging the normalized metrics.
Equal Weighting: This approach ensures that every sample and every metric contributes equally to the final score, preventing experiments with high physical variation from dominating the aggregate.
MSE Interpretation: The Mean Squared Error (MSE) metric is inverted (ratio of physical variation to generated error) so that higher values indicate better performance, aligning it with the Intersection over Union (IoU) metrics.
Clipping: Sub-scores are clipped to the range $[0, 1]$ before aggregation to prevent a single exceptional metric from disproportionately inflating the total score.

Key Results

The authors evaluated six image-to-video (I2V) models (three open-source: Wan 2.2, HunyuanV-1.5, Cosmos3-N; three closed-source: Sora 2, P-Video, Grok Imagine Video) using both the original and verified protocols.

Score Changes: The verified evaluation generally increased scores for most models (due to improved prompts) but decreased scores for others (due to artifact removal).
- Sora 2 showed the largest score increase, benefiting significantly from the improved prompts.
- Wan 2.2 was the only model to show a performance decrease, suggesting its original high scores were partly driven by confounding factors (ambiguous prompts or artifacts).
Ranking Shifts: The refinements led to moderate but meaningful changes in model rankings.
- Wan 2.2 dropped from 1st to 3rd place.
- Sora 2 moved from 6th to 5th.
- Cosmos3-N improved from 5th to 4th.
- P-Video fell from 4th to last place.
Statistical Significance: The correlation between the original and verified rankings was moderate ( $\text{Kendall's } \tau = 0.46$ , $\text{Spearman's } \rho = 0.65$ ), indicating that the evaluation protocol significantly influences model assessment. Bootstrap analysis confirmed that these ranking differences are statistically meaningful and not due to random variance.

Significance and Claims

The paper claims that Physics-IQ Verified provides a more reliable signal for measuring the physical understanding of VGMs. By removing confounding factors such as prompt ambiguity and ground-truth artifacts, and by ensuring equal weighting of samples, the benchmark offers a finer-grained analysis of model capabilities.

The authors emphasize that the original benchmark's fidelity to actual physical understanding was compromised by measurement errors. The proposed refinements do not merely adjust scores but fundamentally alter the evaluation landscape, revealing that some models previously ranked highly may have been benefiting from evaluation artifacts. The work aims to advance the community by providing a robust, traceable, and physically accurate evaluation standard, thereby facilitating the development of the next generation of physically grounded video generative models.

The code and data for the benchmark are made available at the Physics-IQ Verified GitHub repository.