💻 computer science

VidHal: Benchmarking Temporal Hallucinations in Vision LLMs

This paper introduces VidHal, a novel benchmark designed to evaluate temporal hallucinations in Vision Large Language Models by generating captions with varying levels of hallucination and proposing a caption ordering task to enable fine-grained assessment of model limitations.

Original authors: Wey Yeh Choong, Yangyang Guo, Mohan Kankanhalli

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wey Yeh Choong, Yangyang Guo, Mohan Kankanhalli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot friend who can watch videos and talk about them. You ask it, "What's happening in this video?" and it gives you a detailed answer. Usually, it's great. But sometimes, the robot starts making things up. It might say a car is driving left when it's actually driving right, or claim a person is jumping when they are just walking. In the AI world, we call these confident lies hallucinations.

For a long time, researchers have been testing how often these robots lie about photos. But videos are different. A photo is a frozen moment; a video is a moving story with time, direction, and sequence. Until now, nobody had a good way to test if these robots were lying about the story of the video, not just the picture.

This paper introduces VidHal, a new "exam" designed specifically to catch these video-based lies.

The Problem: The Robot's "Daydreaming"

Think of a Vision Large Language Model (VLLM) like a student who is really good at memorizing facts but sometimes gets confused when things move.

The Photo Test: If you show a picture of a cat, the robot might hallucinate and say, "The cat is wearing a hat," even if it isn't. We've known how to test this for photos.
The Video Test: If you show a video of a cat running left, the robot might say, "The cat is running right," or "The cat stopped and then started again." Because videos have time and direction, these lies are harder to spot and harder to test.

The Solution: VidHal (The Video Hallucination Benchmark)

The authors created VidHal, which is like a giant library of video clips, each paired with three different descriptions:

The Truth: A perfect description of what actually happened.
The "Maybe" Lie: A description that is almost right but has a small error (e.g., "The car turned left" when it actually turned slightly left).
The Big Lie: A description that is completely wrong (e.g., "The car flew into the sky").

The Creative Analogy: The "Lie Detector" Game
Imagine you are playing a game with your robot friend. You show it a video of a person juggling three balls. Then, you give it three cards with sentences written on them:

Card A: "The person is juggling three balls." (Truth)
Card B: "The person is juggling four balls." (Small Lie)
Card C: "The person is juggling a watermelon." (Big Lie)

Instead of just asking, "Which one is true?" (which is too easy), VidHal asks the robot to rank them.

"Put these cards in order from the most accurate to the most ridiculous."

This is a much harder test. It forces the robot to not just spot the big lie, but to understand the nuance between a small mistake and a big one. It's like asking a student to grade a test paper not just on "Pass/Fail," but to explain exactly how many points were lost on each question.

What They Found

The researchers tested 23 different AI models (both free, open-source ones and expensive, private ones like GPT-4 and Gemini) using this new exam. Here's what they discovered:

The Robots Are Still Daydreaming: Even the smartest models struggle with the "moving story" parts of videos. They are great at saying "That's a dog," but terrible at saying "The dog ran left then right."
Bigger Isn't Always Better: Surprisingly, some smaller models performed just as well as the massive ones. It seems that just adding more "brain power" (parameters) doesn't automatically fix the problem of lying about time and direction.
The "Snapshot" Bias: The researchers found that many robots are actually cheating. Instead of watching the whole video, they are looking at just one single frame (a snapshot) and guessing the rest.
- Analogy: It's like trying to guess the plot of a movie by looking at just one still photo from the middle. You might get the characters right, but you'll get the story wrong. The paper shows that when you force the robots to look at the whole video, they get better, but they still rely too much on that single snapshot.

Why This Matters

This paper is a wake-up call. It tells us that while AI is getting better at seeing, it's still getting lost in the story of time.

For Developers: They need to build robots that pay attention to the flow of time, not just the static images.
For Us: It reminds us to be careful when trusting AI with video analysis (like security footage or medical videos). If the AI can't tell you the correct order of events, it might miss something critical.

In a nutshell: VidHal is a new, tricky test that catches AI robots when they make up stories about videos. It shows that even our smartest robots are still prone to "daydreaming" about time and direction, and we need to teach them to watch the whole movie, not just the still photos.

1. Problem Statement

Vision Large Language Models (VLLMs) have demonstrated impressive capabilities in understanding images and videos. However, they remain prone to hallucinations—generating responses that appear plausible but contradict visual evidence. While existing research has extensively addressed hallucinations in image-based VLLMs (focusing on objects and attributes), video-based hallucinations remain underexplored.

Video hallucinations present unique challenges due to:

Spatiotemporal Dynamics: Errors often involve the misinterpretation of movement, direction, and the chronological order of events, which are absent in static images.
Nuance: Current evaluation methods (often binary Yes/No or simple object detection) fail to capture subtle, fine-grained errors where a model might get the general action right but fail on the specific direction or sequence.
Lack of Granular Metrics: Existing benchmarks lack the ability to quantify the degree of hallucination, treating all errors as equally severe.

2. Methodology: The VidHal Benchmark

The authors introduce VidHal, a comprehensive benchmark designed to evaluate video-based hallucinations with fine-grained granularity.

A. Dataset Construction

Source: The dataset comprises 1,000 video instances sourced from four public datasets: TempCompass, Perception Test, MVBench, and AutoEval-Video.
Temporal Aspects: VidHal covers five distinct temporal dimensions, categorized into two semantic levels:
1. Lexical Semantics (L-Sem): Misinterpretations of words related to temporal features.
  - Attribute: Size, Shape, Color, Count, State Change.
  - Object: Recognition and Interaction Classification.
  - Action: Movements and behaviors.
  - Direction: Orientation and movement trajectory.
2. Clause Semantics (C-Sem): Errors in event descriptions and sequences.
  - Event Order: The chronological sequence of events.
Caption Generation Pipeline:
- Anchor Caption: An accurate description is generated for each video using GPT-4o based on metadata.
- Hallucinatory Captions: For each video, $M-1$ (where $M=3$ ) additional captions are generated by GPT-4o. These captions are engineered to exhibit varying levels of hallucination severity (Moderate vs. High) relative to the anchor.
- Validation: A rigorous human validation process was conducted on 100 samples with ~15 annotators per sample, achieving an 87% agreement rate on the ordering of hallucination severity.

B. Evaluation Protocol

To move beyond binary accuracy, the authors propose two novel evaluation tasks:

Multiple-Choice Question Answering (MCQA): The model selects the most accurate caption from a set of options (Anchor + Hallucinatory). This measures coarse-grained understanding.
Caption Ordering (Novel Task): The model must rank a set of captions from most accurate to least hallucinated.
- Naive Ordering: Ranking all captions at once.
- Relative Ordering: Decomposing the task into pairwise comparisons to mitigate instruction-following issues and cyclic ranking errors.
- Metric: The authors adapt Normalized Discounted Cumulative Gain (NDCG) from information retrieval. Unlike binary metrics, NDCG awards partial credit for correctly ordered pairs, allowing for the detection of subtle hallucinations (e.g., distinguishing between a "moderate" and "high" hallucination).

3. Key Contributions

VidHal Dataset: The first benchmark specifically dedicated to video-based hallucinations, covering a diverse range of temporal concepts (Action, Direction, Order, Object, Attribute) with multi-level hallucination annotations.
Novel Evaluation Task & Metrics: Introduction of the Caption Ordering task and an NDCG-based metric that quantifies fine-grained hallucination severity, addressing the limitations of binary evaluation.
Comprehensive Empirical Analysis: Evaluation of 23 VLLMs (10 open-source, 3 proprietary, and baselines), revealing critical limitations in current models' temporal reasoning.

4. Experimental Results

The authors evaluated models including LLaVA-NeXT-Video, VideoLLaMA2, Qwen2.5-VL, InternVL2.5, GPT-4.1, and Gemini-2.5.

Overall Performance: Even state-of-the-art models (e.g., Gemini-2.5 Pro, GPT-4.1) struggle with fine-grained temporal reasoning. While proprietary models generally lead, some open-source models (e.g., LongVU, Qwen2.5-VL) perform competitively.
Aspect-Specific Findings:
- Models perform relatively well on Action and Object recognition (likely due to heavy training on image-centric priors).
- Models perform significantly worse on Direction and Event Order, indicating a specific weakness in spatiotemporal dynamics.
Parameter Scale vs. Performance: Surprisingly, smaller models (e.g., 7B) sometimes outperform larger variants within the same family, suggesting that simply scaling parameters does not automatically solve video hallucinations.
Hallucination Differentiation:
- Models can generally distinguish between a perfect caption and a severely hallucinated one.
- Critical Failure: Models struggle to differentiate between a moderately hallucinated caption and the ground truth, or between captions with different degrees of hallucination.
Image Prior Reliance: Ablation studies show that models heavily rely on single-frame spatial information (image priors). When tested on single frames vs. full videos, performance overlap is high, especially for incorrect predictions, indicating models often ignore temporal context.
Instruction Following: The "Relative Caption Ordering" task (pairwise) proved more robust and effective than "Naive Ordering" (ranking all at once), particularly for weaker models.

5. Significance and Future Directions

Holistic Understanding: VidHal shifts the focus from "does the model see the object?" to "does the model understand the story and dynamics of the video?"
Benchmarking Standard: The proposed NDCG-based ranking metric provides a more nuanced tool for evaluating hallucination severity, which is crucial for developing safer and more reliable VLLMs.
Research Implications: The results suggest that future VLLM development must:
- Incorporate broader temporal features during pre-training.
- Mitigate the over-reliance on single-frame spatial priors.
- Develop architectures specifically designed for spatiotemporal reasoning rather than just extending image-based models.

In conclusion, VidHal exposes a significant gap in current VLLM capabilities regarding temporal reasoning and provides the necessary tools (dataset, tasks, metrics) to drive research toward alleviating video-based hallucinations.