🤖 AI

Distorted or Fabricated? A Survey on Hallucination in Video LLMs

This survey provides a comprehensive analysis of hallucinations in Video Large Language Models by introducing a systematic taxonomy distinguishing between dynamic distortion and content fabrication, reviewing current evaluation and mitigation strategies, and outlining future research directions to build more robust video-language systems.

Original authors: Yiyang Huang, Yitian Zhang, Yizhou Wang, Mingyuan Zhang, Liang Shi, Huimin Zeng, Yun Fu

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiyang Huang, Yitian Zhang, Yizhou Wang, Mingyuan Zhang, Liang Shi, Huimin Zeng, Yun Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read friend who loves watching videos. You show them a clip of a cat sleeping on a red rug, and they describe it perfectly. But then, you show them a video of a man cooking in a kitchen, and your friend confidently says, "He's putting on a blue shirt and singing 'Happy Birthday'!"

You look at the video: the man is wearing a white apron, and the kitchen is silent. Your friend is hallucinating. They are making up details that sound plausible but simply aren't there.

This paper is a "user manual" for understanding why these Video Large Language Models (Vid-LLMs) get so creative with the truth, and how we can fix them. Here is the breakdown in simple terms:

1. The Two Big Types of "Lying"

The authors realized that video models don't just make random mistakes; they make two specific kinds of errors. Think of it like a detective solving a crime scene:

Type A: The "Distorted Memory" (Dynamic Distortion)

Imagine you watched a movie yesterday, but your memory of the order of events is jumbled. You remember the hero saving the day before the villain appeared, or you think the chase scene lasted 5 minutes when it was only 30 seconds.

What it is: The model sees the right things (a man, a kitchen, a pot), but it gets the timeline or identity wrong.
The Glitch:
- Time Travel: It says the man cooked before he chopped the onions (backwards causality).
- Time Warp: It thinks an action took 10 seconds when it took 2.
- Identity Crisis: It sees a man in a red shirt in the first half of the video and a man in a blue shirt in the second half, then claims it's the same person changing clothes instantly.
Why it happens: The model is bad at tracking motion and remembering who is who over a long period. It's like watching a movie with a broken remote that skips frames.

Type B: The "Overactive Imagination" (Content Fabrication)

Now imagine you walk into a kitchen and see a cake. Your friend immediately says, "Someone just baked this!" even though the kitchen is empty and silent. They are guessing based on what usually happens, not what is happening.

What it is: The model makes up things that have no visual proof. It relies on its "general knowledge" or gets tricked by sound.
The Glitch:
- The "Script" Error: It sees a dog and a ball, so it assumes the dog is playing fetch, even if the dog is just sleeping. It's following a script rather than watching the scene.
- The "Sound Trap": If there is loud music in the background, the model might say, "The person is dancing!" even if the person is sitting perfectly still. The sound overpowered the eyes.
Why it happens: The model trusts its training data (what usually happens) more than the actual video pixels.

2. How Do We Catch Them? (The Benchmarks)

The paper reviews a bunch of "tests" (benchmarks) researchers have built to catch these liars.

The "Order" Test: "Did the man put on his shoes before or after he tied his laces?"
The "Count" Test: "How many times did the bird flap its wings?"
The "Identity" Test: "Is the person in the beginning of the video the same as the person at the end?"
The "Sound vs. Sight" Test: "Is the person actually crying, or is that just sad music playing?"

The Bad News: Current models are great at simple tasks but terrible at long videos. They get confused easily when the video is long, or when the audio is loud.

3. How Do We Fix Them? (The Cures)

The paper suggests two main ways to stop the models from hallucinating:

Fixing the "Eyes" (Better Encoders):
Right now, many models look at video like a stack of still photos. They need to learn to see motion. Imagine teaching a model to watch a video like a human does—tracking the flow of movement, not just the static frames. We need "motion-aware" cameras inside the AI.
Fixing the "Brain" (Counterfactual Learning):
We need to train the models to say, "Wait, I know that dogs usually play with balls, but in this specific video, the dog is sleeping." We do this by showing them "trick" videos where the usual rules don't apply, forcing them to rely on the visual evidence rather than their guesses.

4. The Future: Building a Trustworthy AI

The authors conclude that if we want AI to be useful in real life (like helping a self-driving car or a robot nurse), we can't let it make up facts.

The Goal: Build an AI that acts like a faithful witness, not a creative writer.
The Plan: Give the AI better memory (so it doesn't forget who the characters are), better motion sensors (so it understands time), and a strict rule: "If you can't see it, don't say it."

Summary Analogy

Think of a Vid-LLM as a tour guide for a video.

Dynamic Distortion is when the guide gets the itinerary wrong (saying you visited the museum before the park, when it was the other way around).
Content Fabrication is when the guide starts inventing stories about the tour ("Look, that's a dragon!") because they think that's what tourists want to hear.

This paper is the guidebook for teaching the tour guide to stick to the facts, respect the timeline, and stop making things up.

1. Problem Definition

Despite significant advancements in Video Large Language Models (Vid-LLMs), these systems suffer from hallucinations: outputs that are linguistically coherent and contextually plausible but contradict the actual spatiotemporal evidence in the input video.

While hallucinations in static image-based Vision-Language Models (VLMs) are well-studied (focusing on objects, attributes, and relations), video introduces unique complexities:

Temporal Dynamics: Reasoning over causality, duration, and motion trajectories.
Multimodal Integration: Handling conflicts between audio and visual cues.
Long-Range Consistency: Maintaining entity identity and scene context across extended sequences.

Existing surveys on Multimodal LLMs (MLLMs) treat video hallucinations superficially, lacking a structural or causal analysis specific to the temporal nature of video. This paper aims to fill that gap by providing a mechanism-driven taxonomy and a comprehensive review of evaluation and mitigation strategies.

2. Methodology: Mechanism-Driven Taxonomy

The core contribution of the paper is a new mechanism-driven taxonomy that classifies video hallucinations based on visually observable failure modes rather than input attributes (e.g., video length or genre). The taxonomy divides hallucinations into two primary categories, each with two subtypes:

A. Dynamic Distortion

Definition: The model detects entities correctly but misrepresents their spatiotemporal evolution or referential consistency. Visual evidence exists, but the relationships are mis-modeled.

Spatiotemporal Dynamics: Errors in temporal relationships.
- Event Misordering: Reversing action causality or motion direction.
- Duration Distortion: Over- or underestimating action length.
- Frequency Confusion: Miscounting repeated actions.
Referential Inconsistency: Semantic-level failures where distinct entities or scenes are conflated across time.
- Character Conflation: Mistaking different individuals for the same person.
- Scene Conflation: Merging actions/settings from distinct contexts into a single narrative.

B. Content Fabrication

Definition: The model produces outputs lacking visual evidence, driven instead by statistical priors or dominant non-visual cues.

Context-Driven Fabrication: Reliance on statistical associations from training data rather than visual grounding.
- Object–Action: Inferring an action based on an object's presence without motion cues (e.g., "cooking" because a stove is visible).
- Scene–Event: Predicting typical events based solely on background settings.
Audio-Visual Conflict: Auditory cues override visual evidence.
- Action Attribution: Generating actions based on background sounds (e.g., "dancing" due to music) when no visual motion exists.
- Emotion Inference: Inferring emotions from vocal tone rather than facial expressions.

Decision Framework: The authors propose a decision checklist (Figure 3) to classify hallucinations: If visual evidence exists but relations are wrong $\rightarrow$ Dynamic Distortion. If no visual evidence exists and the claim is prior-driven $\rightarrow$ Content Fabrication.

3. Key Contributions

First Mechanism-Driven Taxonomy for Vid-LLMs: A structured framework distinguishing between distortion (misinterpretation of existing evidence) and fabrication (generation without evidence), addressing the unique temporal and multimodal challenges of video.
Comprehensive Benchmark Review: The paper analyzes 19 existing benchmarks (Table 1), categorizing them by hallucination type. It highlights that while Spatiotemporal Dynamics are well-covered, Referential Inconsistency and Audio-Visual Conflict remain significantly underexplored, particularly in long-form videos.
Mitigation Strategy Analysis: A review of 15+ mitigation strategies (Table 2), categorized by the hallucination type they address. The paper distinguishes between:
- Training-based methods: (e.g., DPO, RL Fine-tuning) which offer high gains but high cost.
- Training-free methods: (e.g., Contrastive Decoding, Post-hoc correction) which are model-agnostic but offer limited gains.
Root Cause Analysis: Identifies that Dynamic Distortion stems from limited temporal encoding capacity and weak long-range memory, while Content Fabrication arises from insufficient visual grounding and the dominance of priors/audio.

4. Results and Findings

Benchmark Performance: State-of-the-art models show a clear divide. They perform well on high-level tasks (e.g., >80% accuracy on VidHalluc or EventHallusion) but struggle with fine-grained temporal reasoning (e.g., 37.8% on VideoHallucer) and long-context consistency (e.g., 53.1% on ELV-Halluc).
Mitigation Efficacy:
- Training-based approaches (e.g., VistaDPO, AVHModel-Align-FT) yield the highest improvements (up to 30–36% gain) by reshaping model priors.
- Training-free approaches (e.g., SEASON, SmartSight) provide modest gains (+2–7%) but are easier to deploy.
Gap Identification: The survey reveals a critical lack of benchmarks and methods for Audio-Visual Conflict and Referential Inconsistency in long-form streaming scenarios.

5. Significance and Future Directions

This survey consolidates scattered research to provide a systematic understanding of Vid-LLM hallucinations, laying the groundwork for building robust, safety-critical video systems (e.g., autonomous driving, embodied AI).

Proposed Future Directions:

Video-Native Architectures: Moving away from static image encoders toward motion-aware visual encoders (e.g., using optical flow, VideoMAE) to preserve temporal dynamics.
Explicit Memory Mechanisms: Integrating structured memory (e.g., State Space Models like Mamba, episodic memory) to maintain long-range entity consistency.
Counterfactual Learning: Using training strategies that disentangle visual evidence from prior knowledge (e.g., negative sampling with implausible object-action pairs) to mitigate content fabrication.
Audio-Visual Alignment: Developing methods to verify visual input before integrating audio signals to prevent audio-dominant hallucinations.

In conclusion, the paper argues that solving video hallucinations requires a shift from static image-based reasoning to architectures and training paradigms that explicitly model temporal fidelity and multimodal grounding.