💻 computer science

Revealing Multi-View Hallucination in Large Vision-Language Models

This paper introduces the concept of multi-view hallucination in Large Vision-Language Models, presents the MVH-Bench benchmark to evaluate this issue, and proposes a training-free decoding technique called Reference Shift Contrastive Decoding (RSCD) that significantly improves model performance by suppressing visual interference from different viewpoints or instances.

Original authors: Wooje Park, Insu Lee, Soohyun Kim, Jaeyun Jang, Minyoung Noh, Kyuhong Shim, Byonghyo Shim

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wooje Park, Insu Lee, Soohyun Kim, Jaeyun Jang, Minyoung Noh, Kyuhong Shim, Byonghyo Shim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking at one crime scene photo, you are handed two photos taken from different angles of the same room.

In one photo (View 1), a man in a red shirt is holding a blue cup.
In the other photo (View 2), a woman in a blue shirt is holding a red cup.

Now, imagine you ask a super-smart AI assistant: "What is the person in the red shirt holding?"

A human detective would instantly look at the first photo, ignore the second, and say, "A blue cup."

But the AI in this paper? It gets confused. It looks at both photos at once, gets distracted by the red cup in the second photo, and answers, "A red cup." It mixed up the clues.

The paper you provided is all about fixing this specific type of confusion, which the authors call "Multi-View Hallucination."

Here is the breakdown of their discovery and solution, explained simply:

1. The Problem: The "Confused Detective"

Large Vision-Language Models (LVLMs) are like detectives that can see images and read text. They are getting very good at looking at a single picture and describing it. But when you show them multiple pictures of the same scene from different angles, they start to trip over their own feet.

The authors found two main ways these AI detectives get it wrong:

Cross-Instance Hallucination (Mixing up People): The AI sees a person in a white shirt in Photo A and a person in a black shirt in Photo B. When asked about the person in the white shirt, the AI accidentally grabs the action of the person in the black shirt.
Cross-View Hallucination (Mixing up Angles): The AI sees a watering can pointing left in Photo A and pointing up in Photo B. When asked, "In Photo A, which way is the can pointing?", it looks at Photo B and says "Up."

The Analogy: Imagine you are reading a story with two characters, Alice and Bob. If someone asks, "What is Alice doing?", a confused AI might answer, "Bob is eating an apple," because it saw Bob eating an apple in the next paragraph and couldn't keep the characters straight.

2. The Evidence: The "MVH-Bench" Test

To prove this was a real problem, the researchers built a special test called MVH-Bench.

They took videos with cameras everywhere (like a security system with many angles).
They created 4,800 tricky questions designed to trick the AI.
The Result: Even the smartest AI models (like GPT-4o and LLaVA) failed miserably. They were so easily distracted by the "wrong" photo that they couldn't answer simple questions correctly.

3. The Diagnosis: Why is the AI confused?

The researchers dug into the AI's "brain" (its internal layers) to see what was happening. They found that the AI wasn't just looking at the pictures; it was trying to understand the question by talking to itself.

The Discovery: The AI needs to connect the words in the question (e.g., "red shirt") to the right part of the image. But in the middle layers of the AI, the words were getting jumbled. The AI wasn't fully understanding the question before it started looking at the pictures.
The Metaphor: It's like trying to solve a math problem while someone is shouting a different math problem in your ear. You get distracted and solve the wrong equation.

4. The Solution: "Reference Shift Contrastive Decoding" (RSCD)

The researchers came up with a clever, "training-free" trick to fix this. They didn't need to re-teach the AI; they just needed to change how it thinks while it answers.

They call their method RSCD. Here is how it works, using a simple analogy:

The "Double-Check" Trick:
Imagine the AI is about to give an answer.

Normal Path: The AI looks at the question and the photos and thinks, "Okay, the answer is likely X."
The "Shift" Path: The researchers temporarily "blindfold" the AI's ability to connect the words in the question to each other. They force the AI to understand the question poorly.
- What happens? When the AI doesn't understand the question well, it gets even more confused and grabs the wrong visual clues (the hallucination).
The Comparison: The AI now has two thoughts:
- Thought A (Normal): "The answer is X."
- Thought B (Confused): "The answer is Y (the wrong thing)."
The Fix: The AI subtracts Thought B from Thought A. By seeing what the AI gets wrong when it's confused, it can mathematically push itself away from that wrong answer and lock onto the right one.

The Metaphor: It's like a GPS.

Normal GPS: "Turn left."
Confused GPS: "Turn right" (because it got confused by a similar street sign).
RSCD GPS: The system realizes, "Wait, if I get confused, I turn right. Therefore, to be correct, I must turn left." It uses the confusion to find the truth.

5. The Result

When they tested this "Double-Check" trick on the confused AI models:

The models stopped mixing up people and angles.
Their accuracy jumped by a huge amount (up to 34.6% better than previous methods).
It was fast and didn't require re-training the massive AI models.

Summary

This paper is about teaching AI to pay attention to the right details when looking at multiple pictures.

The Problem: AI gets distracted by similar-looking things in different photos.
The Test: A new game designed to catch these mistakes.
The Cure: A clever trick where the AI intentionally "pretends to be confused" to realize what it shouldn't do, allowing it to find the correct answer.

It's a bit like telling a student, "If you don't read the question carefully, you'll pick the wrong answer. So, let's see what the wrong answer looks like, and then we'll make sure we don't pick it."

1. Problem Definition: Multi-View Hallucination (MVH)

The paper identifies a critical failure mode in Large Vision-Language Models (LVLMs) when processing multi-view images (images of the same scene captured from different viewpoints, e.g., first-person and third-person views).

The Phenomenon: While LVLMs excel at single-view interpretation, they frequently confuse visual information originating from different instances or viewpoints when multiple images are provided. The authors term this Multi-View Hallucination (MVH).
Two Categories of MVH:
1. Cross-Instance Hallucination: The model confuses the queried instance with a different instance in the scene.
  - Example: Asked "What is the person in the white top doing?", the model answers based on the person in the black top.
2. Cross-View Hallucination: The model ignores the specified viewpoint and relies on information from an unqueried view.
  - Example: Asked "In View 1, which direction is the watering can pointing?", the model answers based on the orientation in View 2.
Root Cause: The paper argues that MVH arises from incomplete query understanding. When the model fails to fully integrate the textual context (specifically the instance descriptor and the viewpoint specification) before attending to visual features, it becomes susceptible to "distracting visual cues" from non-target instances or views.

2. Methodology

A. MVH-Bench: A New Benchmark

To systematically evaluate MVH, the authors constructed MVH-Bench, a dataset comprising 4,800 question-answer pairs.

Data Sources: Derived from Ego-Exo4D and LEMMA, datasets containing synchronized multi-view videos of human interactions.
Construction Pipeline:
1. Extraction: Using GPT-4o to extract "Instance-Descriptor" pairs (e.g., Instance: "person in white top", Descriptor: "pointing at flowers") from each view.
2. Generation: Automatically generating Binary (Yes/No) and Multiple-Choice questions by swapping instances and descriptors across views to create "distractors."
3. Verification: Human annotators verify the pairs to ensure correctness and remove trivial cases.
Evaluation Metrics:
- p-Acc (Pair Accuracy): Measures if the model correctly answers both the "Yes" and "No" variants of a question pair.
- q-Acc (Quadruplet Accuracy): Requires the model to answer all four binary questions in a set correctly (stricter consistency check).
- YER (Yes-Error Ratio) & AER (Adversarial Error Ratio): Measure bias toward "Yes" answers or selecting specific adversarial distractors.
- MVH-Score: A composite metric summing all accuracy-based metrics.

B. Analysis of Query Context Formation

The authors investigated where in the LVLM architecture the query context is formed.

Experiment: They applied a mask to block text-to-text attention in specific layers of the Transformer decoder.
Finding: Blocking attention in intermediate layers (e.g., layers 13–20 in LLaVA-OneVision) caused a significant drop in "reference accuracy" (the ability to ground the caption in the correct image). This implies that intermediate layers are critical for synthesizing the full textual query before attending to visual tokens.

C. Proposed Solution: Reference Shift Contrastive Decoding (RSCD)

To mitigate MVH without retraining, the authors propose RSCD, a training-free decoding technique.

Core Idea: Amplify the contrast between the model's original prediction and a prediction generated under "disrupted" understanding.
Mechanism:
1. Reference Shift Mask ( $M_{rs}$ ): In the intermediate layers ( $\Lambda^*$ ), the model masks the top $\rho\%$ of attention weights from text tokens to other text tokens. This intentionally disrupts the formation of the full query context, forcing the model to rely on partial understanding and thus attend to incorrect visual cues (generating "negative logits").
2. Contrastive Decoding: The final logits are adjusted by subtracting the negative logits from the original logits:
  $\text{logit}_{rscd} = (1 + \alpha) \cdot \text{logit}_{orig} - \alpha \cdot \text{logit}_{disrupted}$
3. Result: This process suppresses predictions grounded in non-target visual cues (distractors) and amplifies those grounded in the correct instance/viewpoint.

3. Key Contributions

Definition & Categorization: Formalized the concept of Multi-View Hallucination, distinguishing between Cross-Instance and Cross-View errors.
MVH-Bench: Introduced a rigorous benchmark with 4.8k QA pairs and specialized metrics (p-Acc, q-Acc) to evaluate multi-view robustness.
Mechanism Discovery: Demonstrated that intermediate text-to-text attention layers are crucial for query grounding; disrupting them reveals the model's reliance on visual distractors.
RSCD Algorithm: Proposed a simple, training-free decoding method that significantly outperforms existing hallucination mitigation techniques.

4. Experimental Results

The authors evaluated 15 state-of-the-art LVLMs (both closed-source like GPT-4o and open-source like LLaVA-OneVision and Qwen2.5-VL) on MVH-Bench.

Baseline Performance: All models struggled significantly, with low p-Acc and q-Acc scores, indicating a widespread inability to distinguish between views and instances.
RSCD Performance:
- RSCD consistently outperformed existing methods (VCD, ICD, AvisC).
- Improvements:
  - Qwen2.5-VL-7B: Improved MVH-Score by 21.1 points over the next best method.
  - LLaVA-OneVision-7B: Improved MVH-Score by 34.6 points.
- Efficiency: Unlike other contrastive methods that require re-processing image tokens or modifying inputs, RSCD reuses cached key-value states, resulting in minimal inference overhead (0.89s/query vs. 1.38s for baselines).
Qualitative Analysis: Attention map visualizations confirmed that RSCD successfully shifts the model's attention away from distracting regions (e.g., a sink in the wrong view) toward the correct target region specified in the query.

5. Significance

This work addresses a critical gap in the deployment of LVLMs for embodied AI, digital twins, and surveillance, where multi-view inputs are standard.

Safety & Reliability: By identifying and mitigating MVH, the paper enhances the reliability of AI agents that must act based on specific viewpoints or distinct objects in complex scenes.
Efficiency: RSCD offers a highly efficient, plug-and-play solution that does not require expensive fine-tuning, making it immediately applicable to current and future LVLMs.
Insight: The finding that intermediate text layers are pivotal for visual grounding provides a new direction for architectural analysis and future decoding strategies in multimodal models.