💬 NLP

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs

This paper introduces HalluScope, a benchmark revealing that hallucinations in large vision-language models are primarily driven by textual instruction priors, and proposes HalluVL-DPO, a preference optimization framework that effectively mitigates these prompt-induced hallucinations while preserving visual capabilities.

Original authors: Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny, Mustafa Shukor, Alasdair Newson, Matthieu Cord

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny, Mustafa Shukor, Alasdair Newson, Matthieu Cord

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large Vision-Language Model (LVLM) as a very smart, well-read librarian who has read millions of books and seen millions of pictures. This librarian is incredibly good at describing what they see, but they have a quirky habit: they sometimes trust their memory of how the world usually works more than what they are actually looking at right now.

Here is a simple breakdown of the paper "When Prompts Override Vision," using the story of this librarian.

1. The Problem: The Librarian's "Guessing Game"

Usually, when you show the librarian a picture of a beach, they say, "I see sand and water." Perfect.
But sometimes, you ask a tricky question: "What kind of shoes is the surfer wearing?"

Even if the picture shows a surfer with bare feet, the librarian might confidently say, "They are wearing red flip-flops."

Why? Because in the librarian's massive training data, surfers usually wear flip-flops. Their brain is so used to the pattern "Surfer + Flip-flops" that they ignore the fact that the person in front of them is actually barefoot. They are hallucinating based on a "textual prior" (what they expect to see) rather than the visual evidence.

2. The Investigation: "HalluScope" (The Detective's Magnifying Glass)

The researchers realized that existing tests were too vague. They didn't know why the librarian was making mistakes. Was the librarian's eyesight bad? Or was their brain just too full of assumptions?

They built a new test called HalluScope. Think of this as a three-part detective game:

Round 1 (The Eyes): "Is there a chair in this room?" (The chair is there).
- Result: The librarian gets this right. Their eyes work fine.
Round 2 (The Memory): "Is there a chair in this room?" (The chair is not there, but it's a room that usually has chairs).
- Result: The librarian often says "Yes," guessing based on the room type.
Round 3 (The Trap): "What color is the chair in this room?" (There is no chair).
- Result: This is the worst failure. The librarian doesn't just say "Yes," they confidently describe a "blue wooden chair." The question itself trapped them. By asking about the chair's color, the question assumed the chair existed. The librarian's brain couldn't say, "Wait, there is no chair!" because the question was too persuasive.

The Big Discovery: The researchers found that the librarian's eyes are actually fine! The problem is that the question (the prompt) is too strong. The librarian is so eager to answer the question that they invent details to make the answer fit, ignoring the picture.

3. The Solution: "HalluVL-DPO" (The Librarian's Re-Training)

To fix this, the researchers created a new training method called HalluVL-DPO.

Imagine the librarian is being re-trained with a strict coach.

The Old Way: The coach just said, "Don't lie." (Too vague).
The New Way (HalluVL-DPO): The coach sets up a "Choose Your Own Adventure" game.
- Scenario A: The librarian sees a picture with no chair.
- Question: "What color is the chair?"
- Bad Answer (Rejected): "It's a red chair." (The librarian is hallucinating).
- Good Answer (Chosen): "There is no chair in the picture." (The librarian is grounded in reality).

The coach doesn't just say "Good job." They use a special scoring system. If the librarian gives a "Good Answer" that clearly contradicts the "Bad Answer" (like saying "No chair" vs. "Red chair"), that gets a high score. If the answers are just slightly different versions of the same lie, it gets a low score.

The librarian is then fine-tuned to love the "Good Answers" and hate the "Bad Answers." They learn that it is better to say "I don't see that" than to make something up to please the questioner.

4. The Result: A More Honest Librarian

After this training:

When asked about the non-existent chair, the librarian now says, "There is no chair here."
They stop inventing details just because the question implied they should exist.
Crucially, they didn't get dumber. They are still great at describing real objects, solving math problems, and writing stories. They just stopped "guessing" when they should be "seeing."

Summary Analogy

Think of the AI like a weather forecaster who has memorized that "Summer = Sunny."

The Problem: If you show them a picture of a rainy summer day and ask, "Is it sunny?", they might say "Yes" because their memory says "Summer = Sunny," ignoring the rain in the photo.
The Fix: The researchers taught the forecaster to look at the clouds in the photo first, rather than the calendar. Now, even if you ask, "How hot is the sun?", they will correctly say, "I can't tell you how hot the sun is because it's raining right now."

The Takeaway: The paper proves that AI hallucinations aren't usually because the AI is "blind." It's because the AI is too eager to please the text and relies too much on its "common sense" assumptions. By training them to prioritize what they see over what they expect, we can make them much more reliable.

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant progress in multimodal tasks but remain prone to hallucinations—generating outputs not grounded in visual input. While previous research attributed these failures to limitations in visual backbones or the dominance of language priors, the specific relative contributions of these factors were unclear.

Existing benchmarks (e.g., POPE, CHAIR, MMHAL-Bench) treat hallucination as a monolithic failure, failing to distinguish between:

Perception failures: The model cannot see the object.
Co-occurrence priors: The model hallucinates an object because it statistically often appears with other objects in the scene (e.g., seeing a table and assuming a chair is present).
Instruction-induced presuppositions: The model hallucinates an object because the textual prompt implicitly assumes its presence (e.g., asking "What color is the chair?" when no chair exists).

The authors hypothesize that as visual backbones improve, hallucinations are increasingly driven by textual instruction priors and semantic co-occurrence biases rather than perceptual limitations.

2. Methodology

The paper introduces two primary contributions: a diagnostic benchmark (HalluScope) and a mitigation framework (HalluVL-DPO).

A. HalluScope: A Diagnostic Benchmark

HalluScope is designed to disentangle the causes of hallucinations. It consists of 3,000 semantically diverse images (selected from COCO via K-Center Greedy on caption embeddings) paired with specific question types:

Positive Object Recognition: Queries about objects present in the image.
Random Object Recognition: Queries about objects unrelated to the scene and absent.
Adversarial Object Recognition: Queries about objects that are semantically plausible (high Pointwise Mutual Information with scene objects) but visually absent. This tests reliance on co-occurrence priors.
Adversarial Presupposition Attribute (AdP): Queries that ask for attributes of an adversarial object, implicitly assuming its presence (e.g., "What logo is on the baseball cap?" when no cap is present). This tests reliance on textual instruction priors.

Construction Pipeline:

Object Detection: Uses Florence-2 and Grounding-DINO to verify object presence.
Adversarial Discovery: Computes a co-occurrence graph using PMI scores to find the most "credible" absent objects for a given scene.
Verification: A two-stage pipeline (Grounding-DINO + VLM verification) ensures the selected adversarial objects are truly absent.

B. HalluVL-DPO: Mitigation Framework

To address instruction-induced hallucinations, the authors propose HalluVL-DPO, a fine-tuning framework based on Direct Preference Optimization (DPO).

Dataset Construction: A large-scale synthetic dataset (27.4k images, 100k+ queries) is generated. It includes:
- Counterfactual Presupposition Questions (CPQ): Asking about absent objects.
- True Presupposition Questions (TPQ): Asking about present objects.
- Preference Pairs: For each query, a "chosen" (visually grounded) and "rejected" (hallucinated) response are generated.
- Augmentation Strategies: To avoid reward hacking (e.g., length bias) and ensure diverse hallucinations, the authors use:
  - Contrastive Hint Augmentation: Adding hints to both chosen/rejected prompts.
  - Post-Hoc Hallucination Injection: Forcing the model to hallucinate specific attributes.
  - Model-Assisted Answer Inversion: Using an external LLM to generate incorrect attributes for present objects.
Weighted Preference Optimization: Unlike standard DPO which treats all pairs equally, HalluVL-DPO introduces a sample-informativeness weight ( $w$ $w$ ).
- $w=1$ : Low contrast (near rephrasings).
- $w=2$ : Moderate contrast (both follow wrong presupposition but differ in detail).
- $w=3$ : High contrast (chosen correctly denies presence; rejected assumes presence).
- The loss function weights the log-probability ratio by $w$ , focusing optimization on the most informative samples.

3. Key Contributions

HalluScope Benchmark: The first benchmark to explicitly separate hallucination causes into perception, co-occurrence, and instruction-induced presuppositions.
Empirical Insight: Demonstrates that modern LVLMs have robust visual backbones (high accuracy on random/positive recognition) but fail significantly when textual instructions presuppose absent objects (AdP scores drop drastically).
HalluVL-DPO Framework: A novel DPO approach using weighted sample informativeness and a curated synthetic dataset to align models toward visually grounded responses without degrading general capabilities.
Open Source: Release of the benchmark, construction pipeline, training data, and model checkpoints.

4. Results

The authors evaluated LLaVA-1.5-7B and Qwen2-VL-7B on HalluScope and other standard benchmarks.

HalluScope Findings:

Visual Perception: Models achieved >85% accuracy on Positive and Random recognition, confirming visual backbones are reliable.
Co-occurrence Bias: Accuracy dropped by 8–37% on Adversarial Recognition (objects absent but contextually plausible), indicating reliance on learned priors.
Instruction Bias: Accuracy dropped by 25–85% on Adversarial Presupposition (AdP) tasks. For LLaVA-1.5, AdP was only 5.85% before tuning, showing models almost always hallucinate when the prompt assumes an object exists.

Mitigation Performance (HalluVL-DPO):

LLaVA-1.5: AdP score improved from 5.85% to 83.54%. Adversarial Recognition improved from 58.5% to 76.5%.
Qwen2-VL: AdP improved from 57.96% to 80.40%.
Generalization: The model maintained or improved performance on general multimodal benchmarks (MME, ScienceQA, MM-Vet) and other hallucination benchmarks (CHAIR, POPE, HallusionBench).
Ablation: Removing "Attribute pairs" (generated via answer inversion) caused the most uniform performance drop across all benchmarks, highlighting their importance for fine-grained reasoning.

5. Significance

Paradigm Shift: The paper challenges the notion that hallucinations are primarily a visual perception problem. It establishes that alignment between modalities is the critical bottleneck; models are "over-attentive" to text priors and under-attentive to visual evidence when instructions are leading.
Effective Mitigation: HalluVL-DPO proves that targeted preference optimization using synthetic, high-quality preference pairs can drastically reduce prompt-induced hallucinations without sacrificing general multimodal intelligence.
Reproducibility: By releasing the full pipeline and dataset, the authors enable the community to further diagnose and solve the specific failure modes of "prompt-induced" hallucinations, moving beyond generic hallucination metrics.

In conclusion, the paper argues that as LVLMs become more visually capable, the primary source of error shifts to textual instruction bias. The proposed solution effectively re-aligns the model to prioritize visual evidence over textual presuppositions.