System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

Technical Summary: System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

1. Problem: The "Yes-Bias" and the Image-Centric Fallacy

Vision-Language Models (VLMs) frequently suffer from hallucination, specifically a phenomenon known as "yes-bias." In binary (yes/no) tasks—such as object detection, reasoning, or factual QA—models tend to indiscriminately respond with "yes," regardless of the actual visual or textual evidence.

Existing research typically follows an "image-centric hypothesis," which posits that hallucinations stem from insufficient attention allocated to the image modality. Consequently, most mitigation strategies focus solely on increasing image attention. The authors argue this view is incomplete because it ignores the role of the system modality (e.g., <BOS> tokens, system messages) and the textual modality, both of which may hold disproportionate or redundant attention weights.

2. Methodology: The System-Mediated Hypothesis

The authors propose a "system-mediated hypothesis": hallucinations arise because functionally redundant attention weights in the system modality (often acting as "attention sinks") deprive both the image and text modalities of necessary attention.

Experimental Framework:

• Targeted Intervention: The study focuses on the late layers (Q4, layers 25–32) of the LLaVA-1.5 7B text decoder, as these are identified as the primary sites for hallucination.
• Causal Redistribution: Instead of simple scaling, the authors use proportional redistribution. They zero-ablate the attention weights of a "source" modality and reallocate that mass to the "recipient" modalities in proportion to their original weights.
  • System-mediated test: Redistributing from the system to both image and text.
  • Image-centric test: Redistributing from text to image.
• Benchmarks: They utilize six paired-prompt benchmarks (e.g., HallusionBench, Winoground, SugarCrepe) designed to test fine-grained compositionality, where the ground truth changes based on minimal visual or textual differences.
• Metrics: Simple accuracy, paired accuracy (correctness on both prompts in a pair), and "Yes-rate" (the frequency of "yes" responses).

3. Key Contributions

1. Identification of System Attention as a Driver: They demonstrate that redundant system attention in late layers is a primary causal factor in the yes-bias.
2. Formalization of a Holistic Account: They move beyond the image-centric view to a system-mediated view, showing that hallucination is a result of an imbalance across all three modalities.
3. Mechanistic Explanation: They provide evidence that system attention encourages coarse, cross-modal aggregation (global representations), which is useful for general tasks but detrimental to fine-grained, compositional tasks that require local detail.

4. Results

• Superiority of Redistribution: Proportional redistribution of attention from the system modality to both image and text (Q4 system redistribution) significantly outperformed image-only boosting (PAI) and text-suppressing (AD-HH) methods.
• Validation of Hypothesis: In pairwise tests, system-to-text transfers yielded the highest performance gains, followed by system-to-image. This directly contradicts the image-centric hypothesis, which predicts that text-to-image transfers should be more effective.
• Task-Specific Nuance: The intervention improved performance on tasks requiring fine-grained scrutiny (e.g., Winoground) but actually decreased performance on tasks like MME, which rely on high-level, global knowledge. This confirms that system attention facilitates "coarse" reasoning.
• Representational Evidence: The authors showed that redistributing system attention significantly lowers the cosine similarity between image and text representations, proving that system attention was indeed forcing the modalities into a single, coarse, aggregated representation.
• Generalizability: While the effect was most pronounced in LLaVA-7B, similar (though less dramatic) trends were observed in LLaVA-13B and the Qwen2-VL family, suggesting the phenomenon is a structural characteristic of current VLMs.

5. Significance

This paper shifts the paradigm of VLM hallucination research from "the model isn't looking at the image enough" to "the model is looking at the system/sink tokens too much." By identifying system-mediated attention imbalance as a lever, the study provides a new, more effective pathway for mitigating hallucinations. It suggests that future mitigation strategies should not just boost visual attention but should actively manage the redistribution of redundant attention from non-informative system tokens to the informative input modalities.