🤖 AI

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

This paper introduces the Visual Dependency Gap (VDG) to demonstrate that high benchmark accuracy in video large language models often stems from language priors rather than genuine visual understanding, revealing that temporal order contributes minimally to performance while frame diversity drives most gains.

Original authors: Jae Joong Lee

Published 2026-07-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jae Joong Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. You give it a camera and a brain, and you ask it questions about what it sees. For a long time, scientists have measured how "smart" these robots are by giving them a test: show them a video, ask a question, and see if they get the right answer. If the robot gets a high score, we assume it is truly "seeing" and understanding the video. But here is the catch: just because a robot gets the right answer doesn't mean it looked at the picture. It might have just guessed based on the words in the question, like a student who knows the teacher always asks about "cats" on Tuesdays and just answers "cat" without looking at the blackboard. This paper dives into the messy, confusing world of "Video Large Language Models"—super-smart computers that can read text and watch videos—to ask a scary question: Are these robots actually watching the movies, or are they just really good at guessing the answers using only the text?

The researchers decided to play a little trick on twenty different video-watching robots, ranging from tiny ones to massive ones with huge brains. They took a standard test called MVBench, which has hundreds of questions about videos, and they ran the test twice for every single question. The first time, they showed the robot the real video. The second time, they showed the robot the exact same question, but the video was replaced with a solid black screen. If the robot was truly "seeing," its score should drop to near zero when the video disappears. If the robot kept getting the answers right even with a black screen, it meant it wasn't watching at all; it was just using language tricks to guess.

The results were a bit of a shock. The team found that for many questions, the robots performed almost exactly the same whether they were watching a video or staring at a black void. They introduced a new way to measure this called the "Visual Dependency Gap" (VDG), which is basically the difference between how well a robot does with a video versus without one. They discovered that for some types of questions, like "What color is the car?" (Attribute Perception), the robots really did need the video to get it right. But for other types, like "What happens next?" (Temporal Reasoning), the robots got the answers right even when the screen was black. This means the test questions weren't actually testing if the robots understood time or sequence; they were just testing if the robots could guess the answer based on the wording of the question.

The researchers also tried to figure out why the robots were getting these answers. They broke the video down into different parts: just one single frame, a bunch of frames shuffled out of order, and the full video with time moving forward. They found that the robots didn't care about the order of events at all. Whether the video played forward, backward, or was just a jumbled mess of pictures, the robots got the same score. The only thing that seemed to help was seeing different pictures (frame diversity), not seeing them in the right order. In fact, for some of the newest, most advanced models, the video input actually seemed to confuse them, making them perform slightly worse than if they had just ignored the video entirely and guessed based on the text.

One of the most interesting findings was about how these robots handle compression. Usually, if you squish a video file to make it smaller, the quality gets worse. The researchers expected that if a robot was truly "seeing," a blurry, compressed video would make it fail. But the robots' scores stayed flat and steady even when the video was heavily compressed. The team realized this wasn't because the robots were super-robust; it was because the questions were so easy to guess from the text that the robots didn't need the visual details at all. The "robustness" was an illusion created by bad test questions.

Finally, they looked at how these robots changed as they got bigger and smarter. They found that while bigger models generally got better at using the video, some of the very newest models actually got worse at relying on visual information. One model, Qwen3-VL, seemed to have forgotten how to use the video frames effectively, relying almost entirely on guessing from the text, even though its overall test scores looked good. The paper concludes that we can't just trust the leaderboard scores anymore. A high score doesn't mean a robot is a great observer; it might just mean it's a great guesser. To know if a robot is truly seeing, we need to check if it fails when the screen goes black. If it doesn't fail, it wasn't watching in the first place.

Technical Summary: Accuracy Without Grounding

Problem Statement

The prevailing assumption in video large language model (LLM) evaluation is that higher benchmark accuracy equates to stronger visual understanding. This paper challenges that assumption, arguing that benchmark accuracy often conflates genuine visual grounding with the exploitation of language priors (text-only solvability). While prior work has identified that some video questions can be answered without vision, these studies have not quantified this phenomenon per-question or statistically proven that accuracy and visual dependency are separable axes. Consequently, leaderboard improvements may signal better language modeling rather than genuine progress in spatio-temporal reasoning.

Methodology

The authors introduce the Visual Dependency Gap (VDG), a per-question diagnostic metric defined as the difference in correctness between an original video condition and a black-screen condition (where all frames are replaced by solid black images, preserving text and options).
$VDG(q) = \mathbb{1}[\text{correct}_{\text{original}}] - \mathbb{1}[\text{correct}_{\text{black}}]$
The study evaluates 20 video LLMs spanning 2–78B parameters and 10 architecture families (including open-weight models like Qwen, InternVL, LLaVA, and proprietary API models like GPT-4o and Gemini) across three benchmarks: Video-MME, MVBench, and EgoSchema.

Key experimental protocols include:

McNemar Dissociation Testing: Paired McNemar tests are applied to binary correctness outcomes to determine if models differ in visual dependency independently of overall accuracy.
Four-Condition Diagnostic Ladder: To decompose visual dependency, the authors test models under four conditions: (1) Black screen, (2) Single frame (repeated), (3) Shuffled frames (diverse content, no temporal order), and (4) Original video. This isolates contributions from spatial recognition, frame diversity, and temporal ordering.
Compression Robustness Analysis: Videos are re-encoded with H.264 at varying CRF levels (18–38) to test if aggregate accuracy stability masks bidirectional answer flips.
FPS Ablation: Frame rates are varied from 0.5 to 24 FPS to rule out frame sparsity as a cause for low temporal reasoning scores.

Key Contributions

Statistical Dissociation of Accuracy and Grounding: Using paired McNemar tests, the paper proves that accuracy and visual dependency are statistically separable. For instance, on MVBench, InternVL2-8B significantly outperforms Qwen2-VL-7B on original video ( $p=0.0003$ ) but is statistically indistinguishable on black screens ( $p=0.53$ ).
Task-Type VDG Spectrum: The authors establish a stable hierarchy of visual dependency across task types. Attribute Perception is the most visually grounded category (VDG $\approx$ 0.41), while Temporal Reasoning approaches the language-only baseline (VDG $\approx$ 0.06). This ranking holds consistently across 20 models and three distinct benchmarks.
Decomposition of Visual Signals: The diagnostic ladder reveals that frame diversity (observing multiple distinct frames) supplies the vast majority of visual benefit, whereas temporal ordering contributes near-zero accuracy across 16 open-weight models.
Compression Robustness Illusion: The study demonstrates that flat accuracy curves under H.264 compression are artifacts of bidirectional cancellation (questions flipping from correct to incorrect canceling out those flipping from incorrect to correct), rather than true model robustness. Questions that are sensitive to compression are disproportionately visually grounded.

Key Results

Dissociation: Models that perform best on original video do not necessarily perform better on black screens, indicating that leaderboard rankings often reflect language-prior exploitation rather than visual capability.
Temporal Reasoning Failure: Across all models, temporal ordering provides negligible accuracy gains. The VDG for temporal reasoning tasks is near zero, suggesting current benchmarks do not effectively test temporal understanding.
Frame Diversity Dominance: The "diversity delta" (accuracy gain from shuffled frames vs. single frame) is the dominant visual signal (ranging from +0.075 to +0.185), while the "temporal delta" (original vs. shuffled) is near zero.
Generational Regression: While scaling generally improves VDG (e.g., Qwen2.5-VL scaling from 3B to 7B increases VDG from 0.210 to 0.297), the newer Qwen3-VL series shows a regression in VDG (e.g., Qwen3-VL-8B VDG of 0.107 vs. Qwen2.5-VL-7B's 0.297). This is attributed to a collapse in frame diversity extraction ( $\Delta_{diversity} \approx 0.034$ ), despite higher black-screen accuracy (stronger language priors).
Quantization Artifacts: Large models (e.g., InternVL2.5-78B) under 4-bit quantization can exhibit negative or zero VDG, indicating that their benchmark accuracy is entirely driven by language priors.
Cross-Benchmark Generalization: The VDG tier taxonomy (Perceptual > Action > Temporal/Linguistic) successfully predicts task-type performance on EgoSchema, a benchmark with zero video source overlap with the training data, confirming the metric's generalizability.

Significance and Claims

The paper claims that benchmark accuracy is a poor proxy for visual grounding. The authors argue that the field risks optimizing for language-prior exploitation rather than genuine visual understanding. By introducing VDG as a standard audit instrument, the paper aims to:

Distinguish between models that truly rely on visual input and those that solve tasks via text.
Reveal that current "Temporal Reasoning" benchmarks largely fail to test temporal understanding, as temporal ordering contributes near-zero accuracy.
Highlight that "robustness" to compression is often an illusion caused by benchmark composition rather than model capability.

The authors position VDG not as a measure of model capability, but as a diagnostic for benchmark validity. They conclude that for video benchmarks to measure visually grounded capability, they must move beyond aggregate accuracy and incorporate diagnostics like VDG to ensure questions require visual input. The code and annotations are made publicly available to facilitate this audit.