Mechanisms of Prompt-Induced Hallucination in Vision-Language Models

1. Problem Definition: Prompt-Induced Hallucination (PIH)

The paper investigates a specific failure mode in Vision-Language Models (VLMs) known as Prompt-Induced Hallucination (PIH). This occurs when a VLM prioritizes conflicting textual prompts over visual evidence, leading to hallucinations.

• The Phenomenon: In a controlled object-counting setting, if a user asks a model to "Describe the 4 waterlilies" when only 3 exist, the model often hallucinates a fourth flower to satisfy the prompt, rather than correcting the user.
• The Paradox: This is not a failure of numerical reasoning. The models can correctly count objects when asked directly (e.g., "How many waterlilies?"). The error arises specifically when the prompt contains a false premise that contradicts the image.
• Key Observation: PIH is highly dependent on object count. For small numbers (2–4), models often resist the prompt. However, as the number of objects increases (>4), models increasingly conform to the prompt regardless of the discrepancy size, even for extreme mismatches (e.g., describing 59 cats in an image of 9).

2. Methodology

The authors employed a mechanistic interpretability approach to isolate and analyze the internal components responsible for PIH across three state-of-the-art VLMs: LLaVA-OneVision-7B, Qwen2-VL-7B, and Janus-Pro-7B.

A. Experimental Setup

• Dataset: Used CountBench for object counting and Visual CounterFact for a color identification task.
• Prompt Design:
  • Baseline: "How many [objects] are in the image?" (Used to filter for samples where the model knows the ground truth).
  • Misaligned Prompt: "Describe the $N+k$ [objects] in the image," where $N$ is the true count and $k$ is a positive offset (discrepancy).
• Metrics: Measured the rate of "Prompt Matching" (hallucinating the wrong number) vs. "True Count Matching" (correcting the prompt).

B. Intervention: Attention Head Ablation

To identify the causal mechanisms, the authors performed mean ablation on specific attention heads:

1. Head Knockout: They systematically ablated individual attention heads using mean ablation (replacing the head's output with its mean activation) to remove token-specific information while preserving activation magnitude.
2. Ranking: Heads were ranked by their "success rate"—the percentage of samples where ablation caused the model to switch from a hallucinated count to the correct visual count.
3. Grouped Ablation: The top-$m$ heads were ablated simultaneously to test the cumulative effect.

C. Generalization Tests

• Color Task: Applied the same PIH-head ablations to a color identification task (e.g., "Describe the purple banana" when it is yellow) to test if the mechanism generalizes beyond numbers.
• Capability Checks: Evaluated performance on standard benchmarks (CalTech101, MM-Vet, POPE) to ensure ablation did not degrade general multimodal reasoning or instruction following.

3. Key Contributions & Findings

A. Identification of PIH Heads

• Small Subset: A very small set of attention heads (typically 3–10 per model) is responsible for the majority of PIH.
• Localization: These heads are predominantly located in the early layers (Layer 0 and 1) of the language model backbone. This suggests that prompt-induced numerical information is injected and propagated early, before extensive multimodal integration occurs.
• Shared Mechanisms: There is significant overlap in the top PIH heads between Qwen2-VL and LLaVA-OneVision (which share the same Qwen2 language backbone), indicating that PIH is primarily mediated by the language model component rather than vision-specific encoders.

B. Effectiveness of Ablation

• Drastic Reduction in Hallucinations: Ablating the identified PIH heads reduced prompt-induced hallucinations by 40% to 95% across models and tasks without any additional training.
  • Counting: Prompt match rates dropped from ~40–60% to <11%.
  • Color: Prompt-induced hallucinations were reduced by up to 94.25%.
• Restoration of Visual Grounding: The intervention shifted model behavior from prompt reliance to visual reliance. Models began correcting the prompt and reporting the true count/color.
• Preservation of General Capabilities: Ablation did not degrade performance on standard benchmarks (MM-Vet, POPE) or instruction-following tasks where the prompt and image align. In fact, LLaVA-OneVision showed a 4.35% improvement in baseline counting accuracy after ablation.

C. Mechanistic Differences Across Models

While ablation universally reduced PIH, the mechanism of suppression varied by model:

1. LLaVA-OneVision: Exhibited the strongest effect. Ablation suppressed all forms of copying (exact, soft, and format) and induced a massive shift in attention mass toward image tokens (especially in early layers). This model became significantly more visually grounded.
2. Janus-Pro: Primarily suppressed format copying. When correcting prompts, it shifted from copying the prompt's digit format to generating free-form word responses. It showed minimal change in attention mass toward images.
3. Qwen-VL: Interestingly, ablation increased format copying (adhering to the prompt's digit style) while reducing soft copying (content hallucination). Despite this, it still improved visual grounding, suggesting a complex trade-off where the model adheres to the prompt's structure but corrects its content.

4. Significance and Implications

• Causal Understanding: The paper moves beyond correlation to establish a causal link between specific attention heads and hallucination, proving that PIH is a localized routing failure rather than a general lack of visual perception.
• Efficient Mitigation: It demonstrates that hallucinations can be mitigated via targeted, training-free interventions (ablation) rather than expensive fine-tuning or reinforcement learning.
• Architectural Insight: The findings highlight that VLMs often rely on the language backbone to process prompt instructions, sometimes at the expense of visual evidence. The early injection of prompt tokens into the computation stream allows text to override visual data before integration is complete.
• Model-Specificity: The study reveals that while the outcome (reduced hallucination) is consistent, the internal implementation of PIH varies significantly between models, necessitating tailored interpretability approaches for different architectures.

Conclusion

The paper concludes that Prompt-Induced Hallucination is a mechanistically localized failure mode driven by a small set of early-layer attention heads that propagate prompt information. By ablating these specific heads, researchers can substantially restore visual grounding and reduce hallucinations across diverse tasks and models, offering a promising pathway for improving the reliability of VLMs in real-world applications.