🤖 AI

Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

This paper introduces Sci-VBench, a comprehensive benchmark comprising 1,253 expert-annotated examples across four scientific disciplines to evaluate the knowledge- and reasoning-intensive video generation capabilities of 16 frontier models, revealing that while visual realism has improved, significant gaps remain in scientific and causal correctness, particularly between proprietary and open-source systems.

Original authors: Diandian Zhang, Tingyu Song, Lin Fu, Zheyuan Yang, Yilun Zhao

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Diandian Zhang, Tingyu Song, Lin Fu, Zheyuan Yang, Yilun Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a movie where a wizard casts a spell, a rocket launches, or a heart beats. For years, the biggest question for computer scientists was simple: "Does this look real?" If the pixels were sharp, the colors bright, and the motion smooth, we cheered. But there's a new, much trickier question now: "Does this actually make sense?"

This is the world of video generation, where artificial intelligence (AI) tries to create moving pictures from just a sentence of text. Think of it like a super-creative robot that reads a recipe and tries to cook the dish. Until recently, we were mostly checking if the food looked tasty. But what if the robot accidentally put salt in the cake instead of sugar? It might look like a perfect cake, but it would taste terrible and fail the test. In science, this is dangerous. If an AI simulates a chemical reaction or a medical procedure, it can't just look good; it has to follow the strict, invisible laws of physics and biology. If the AI breaks those laws, the video is a lie, no matter how pretty it is.

This is exactly the problem a new paper tackles. The researchers built a massive "exam" called Sci-VBench to test if AI video makers can actually understand science, not just mimic it. They didn't just ask, "Is the video pretty?" They asked, "Did the robot actually understand how the world works?"

The Great Science Video Exam

The team created a giant test bank with 1,253 different challenges, covering 60 different subjects like physics, medicine, engineering, and even history. Imagine a test where you have to generate a video of a specific experiment, like hitting a knee with a hammer to see the leg kick, or watching two balls roll down different tracks to see which wins. The catch? The AI has to figure out the hidden rules of how those things move on its own. It can't just guess; it has to get the science right.

To grade these videos, the researchers didn't just rely on a computer's opinion. They brought in 61 human experts—real scientists and students from universities—to act as the judges. These experts wrote down exactly what a "perfect" video should look like, step-by-step, creating a strict scoring guide (a rubric) for every single test. They checked four things:

Does it look good? (Is the video clear and smooth?)
Did it listen? (Did it include all the specific things the prompt asked for?)
Is the science right? (Did the objects move according to real laws of nature?)
Is the story consistent? (Did the video make sense from start to finish without glitching?)

The Shocking Results: Pretty but Clueless

When they ran 16 of the world's most advanced AI video models through this exam, the results were a bit of a wake-up call.

First, the good news: The AI models are getting incredibly good at making things look real. When the judges checked for basic visual quality (like smoothness and color), almost all the models scored very high, clustering tightly together. They are all masters of the "painting" part of the job.

But then came the bad news. When the judges checked if the AI actually understood the science, the scores dropped dramatically and varied wildly.

The proprietary models (the big, expensive ones from companies like Google, OpenAI, and Alibaba) generally did better. The top performer, Gemini-Omni-Flash, managed to get the science right most of the time, but even it wasn't perfect.
The open-source models (the free ones anyone can download) struggled significantly more. While they looked just as pretty as the expensive ones, they often failed the science tests. For example, in a test about a knee-jerk reflex, some models made the wrong leg kick, or made the leg move in a way that defies human anatomy.

The paper found a huge gap between "looking real" and "being real." The AI can create a video of a ball rolling down a hill that looks stunning, but if the ball suddenly floats up or changes color in the middle of the roll, the AI has failed the science test. The researchers noted that this gap is especially wide in Scientific and Causal Correctness—the ability to understand cause and effect.

Why the AI Gets It Wrong

The researchers dug deeper to see why the AI fails. They found three main types of mistakes:

Ignoring Instructions: The AI might miss tiny details, like forgetting to show a specific tool mentioned in the prompt.
Fake Science: This is the big one. The AI prioritizes making the video look "cool" or "smooth" over making it physically accurate. It might make a chemical reaction happen instantly because it looks dramatic, even though real chemistry takes time.
Time Travel Glitches: The video might start with two balls, but halfway through, they merge into one, or the lighting changes randomly. The AI loses track of the story as time passes.

Interestingly, the researchers tried a simple fix: they rewrote the prompts to be more explicit, telling the AI exactly what to do. This helped a little bit, improving the scores for some models, but it didn't fix the core problem. The AI still couldn't generate the deep, mechanical understanding of how the world works. It's like giving a robot a more detailed recipe; it might follow the steps better, but if it doesn't understand why you mix the ingredients, it still won't bake a good cake.

The Bottom Line

The paper concludes that while AI video generators have become amazing artists, they are still clumsy scientists. They can paint a beautiful picture of a rocket launch, but they often don't understand the physics of the launch itself. The gap between "looking real" and "being real" is still wide, and until AI can truly grasp the rules of the universe, its scientific videos will remain just that—illusions. The researchers suggest that to move forward, we need to stop just asking "Is it pretty?" and start demanding, "Does it make sense?"

Technical Summary: Sci-VBench

Problem Statement

While video generative models have achieved significant advances in visual fidelity, motion coherence, and controllability, a critical gap remains in their ability to faithfully realize events and constraints specified by prompts within scientific domains. Existing benchmarks primarily focus on perceptual quality, prompt alignment, and generic physical commonsense. However, in science-domain settings, a video may appear visually plausible while violating conservation laws, reversing causal relations, or depicting impossible state transitions. Current evaluation methods lack the capacity to assess whether generated videos preserve mechanistic fidelity and domain-specific knowledge, particularly across diverse scientific disciplines beyond basic physics. There is a need for a benchmark that moves beyond surface-level realism to test knowledge-grounded, temporally consistent synthesis where success hinges on scientific reasoning.

Methodology

Benchmark Construction (Sci-VBench)

The authors introduce Sci-VBench, a comprehensive benchmark comprising 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science (45.3%), Healthcare (23.9%), Engineering (22.6%), and Humanities & Social Sciences (8.2%).

The construction process adheres to four desiderata:

Breadth: Coverage of 60 subjects from astrophysics to public policy.
Depth: Every prompt is authored by a domain expert, requiring grounded scientific understanding and multi-step causal reasoning.
Completeness: Targets phenomena that unfold over time, ensuring no single frame satisfies the prompt.
Reproducibility: A rubric-based protocol with per-example scoring anchors.

Data Pipeline:

Subject Selection: A user study with 133 students informed the selection of 60 subjects where core concepts are expert-level and verifiable via video evidence.
Prompt Annotation: Experts select target concepts from canonical textbooks and craft minimal prompts specifying only the initial setup and intervention, omitting the expected mechanistic trajectory. This forces models to infer the mechanism rather than reproduce a described outcome.
Evaluation Specification: For each prompt, experts generate a Reference Guide (defining target concepts and expected phase-based phenomena) and a detailed Evaluation Rubric with 1–5 scoring anchors. This externalizes expert knowledge, allowing non-experts and MLLMs to score consistently.
Quality Control: Every example undergoes independent review by a second domain expert. An audit of 200 randomly sampled examples confirmed high inter-annotator agreement on independently authored specifications (Quadratic Weighted $\kappa \approx 0.73\text{--}0.79$ ).

Evaluation Protocol

The paper establishes a dual protocol for evaluation:

Low-level Perceptual Fidelity (LPF): Assessed using VBench-based Vision Tools (VT), utilizing six established automatic metrics (Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality) normalized to a 1–5 scale.
Reasoning-Centric Dimensions: Evaluated via a Rubric-Conditioned MLLM-as-Judge pipeline (using Qwen3.5-397B-A17B). The judge is provided with the prompt, the video, the high-level reference guide, and the specific 1–5 anchored rubric for the dimension being scored. The dimensions are:
- Prompt Grounding (PG): Faithfulness to explicit conditions (entities, states, spatial arrangement).
- Scientific and Causal Correctness (SCC): Consistency with domain knowledge and causal mechanisms.
- Spatiotemporal Consistency (SC): Coherent temporal evolution and stable spatial relationships.

Human expert ratings serve as the reference labels. A controlled study validated that providing evaluation specifications significantly improves non-expert agreement with experts and that the rubric-conditioned MLLM-as-Judge aligns more closely with expert judgments than prior automatic evaluation methods (e.g., VideoScore, VideoReward).

Key Results

The authors benchmarked 16 frontier models (8 proprietary, 8 open-source) on the testmini split (150 examples).

Perceptual Saturation vs. Reasoning Gaps: Automatic perceptual-quality scores (VT) are nearly flat across all systems (ranging 3.79–4.12), indicating that visual realism has plateaued for current benchmarks. In contrast, performance on reasoning-centric dimensions varies substantially.
Proprietary–Open-Source Gap: A pronounced performance gap exists between proprietary and open-source models, concentrated entirely on Prompt Grounding and Scientific and Causal Correctness.
- Top Performer: Gemini-Omni-Flash achieved the highest averages in both automatic (3.38) and human (3.18) evaluations.
- Open-Source Limitations: The best open-source model (MiniMax-H3) scored 1.63 on SCC, roughly half the score of the best proprietary system (3.34).
- Consistency: Open-source models performed comparably to proprietary ones on Spatiotemporal Consistency, suggesting the gap is not in temporal coherence but in mechanistic reasoning.
Disciplinary Variance: No single model dominates all disciplines. For instance, Gemini-Omni-Flash led in Engineering, Natural Science, and Humanities, while Sora-2 performed best in Healthcare.
Prompt Rewriting: Rewriting prompts to be more explicit improved scores for Prompt Grounding and Scientific Correctness (e.g., +51.7% SCC for HunyuanVideo-1.5), but the gap with proprietary models remained. This suggests that while underspecified instructions contribute to errors, the fundamental limitation lies in the generator's inability to model mechanisms, not just the prompt's clarity.

Error Analysis

The paper identifies three primary failure modes:

Poor Adherence to Instructions: Models miss fine-grained details specified in prompts (e.g., failing to render a specific switch mechanism).
Inaccurate Simulation of Scientific Principles: Models prioritize aesthetics over physical realism, producing factually incorrect dynamics (e.g., incorrect knee-jerk reflex direction, failure to blink during corneal reflex simulation).
Deficiencies in Temporal Coherence: Issues include object permanence failures (objects merging or disappearing), inconsistent visual styles, and weak causal links between events.

Significance and Claims

The paper claims that Sci-VBench fills a critical gap in evaluating text-to-video models as "world simulators" for expert domains. Its primary contributions are:

A Multidisciplinary Benchmark: Providing the first large-scale dataset (1,253 examples) covering 60 subjects across four major disciplines with expert-verified ground truth.
Portable Evaluation Specifications: Demonstrating that detailed, per-example rubrics allow non-experts and MLLMs to achieve high agreement with domain experts, enabling scalable and reproducible evaluation without requiring constant expert recruitment.
Diagnostic Insights: Revealing that current advances in visual realism have not translated into reliable modeling of scientific and causal dynamics. The persistent gap between proprietary and open-source models on reasoning dimensions highlights that the next frontier in video generation is not just visual fidelity, but mechanistic fidelity.

The authors conclude that while leading models produce visually convincing outputs, they frequently fail to capture the underlying scientific mechanisms, indicating a need for improved instruction adherence and mechanistic consistency in future model development.