See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment

1. Problem Statement

Multimodal Large Language Models (MLLMs) suffer from object hallucination, where models generate descriptions of objects not present in the visual input or omit objects that are clearly visible.

• Root Cause: The authors identify that this is not primarily a failure of visual perception (the encoder sees the objects) but a decoding-time attention imbalance.
• Mechanism: During autoregressive decoding, attention mechanisms disproportionately concentrate on visually dominant, large, or frequent objects (e.g., a motorcycle). This "attention collapse" suppresses tokens associated with rare, small, or peripheral objects (e.g., a person standing nearby), causing the model to ignore them or hallucinate non-existent ones to fill the context.
• Limitation of Current Solutions: Simply redistributing attention equally is insufficient because it ignores the disparity in object scale and frequency. Existing methods often require retraining, auxiliary models, or complex architectural changes.

2. Methodology: DOP-OBC

The paper proposes DOP-OBC, a training-free, architecture-agnostic decoding strategy that operates entirely at inference time. It injects "equitable attention" signals directly into the causal attention mask without modifying model weights.

The method consists of three core components:

A. Upper-Triangular Attention Registers

To prevent attention collapse and stabilize long-range generation, the authors introduce a register that absorbs surplus attention mass directed toward future positions.

• They modify the logits ($\mathbf{W}$) by adding an upper-triangular matrix $\mathbf{P}$ with a linear decay slope ($\sigma$).
• This ensures that as decoding progresses, the model maintains a controlled focus on historical context while strictly adhering to causal masking.

B. Dominant Object Penalty (DOP)

This signal softly suppresses attention allocated to visually dominant objects to free up representational capacity for others.

• Calculation: A dominance score $D(o)$ is computed for each object based on normalized size ($\hat{s}$), temporal persistence ($\hat{L}$), and current attention share ($\hat{A}$).
• Penalty: A soft penalty weight $w_{pen} = \exp(-\lambda D(o))$ is applied to the row-wise logit amplitude. Objects with high dominance scores receive a lower amplitude, preventing them from monopolizing the attention distribution.

C. Outlier Boost Coefficient (OBC)

This signal amplifies attention toward rare but confidently detected objects.

• Calculation: A rarity score $r(o)$ is estimated using the Mahalanobis distance between an object's features and an exponential moving average (EMA) of the scene distribution.
• Boost: A boost coefficient $b(o) = \gamma \cdot r(o)$ is applied only if the object's detection confidence $p(o)$ exceeds a threshold $\tau_p$. This ensures that only reliable, rare objects receive amplification.

D. Integration

These signals are integrated into the attention computation as row-wise modulations:

1. Amplitude Scaling: $\alpha_i \leftarrow \alpha_0 \cdot w_{pen}(o_i) \cdot (1 + b(o_i))$
2. Slope Adjustment: $\sigma_i \leftarrow \sigma_0 / (1 + b(o_i))$
3. The modified logits are passed through the standard SoftMax and causal masking.

3. Key Contributions

1. Diagnosis of Inequity: The paper establishes that object hallucination is fundamentally driven by inequitable attention allocation during decoding, where dominant objects suppress peripheral ones.
2. DOP-OBC Framework: A novel, plug-and-play decoding strategy that uses two complementary signals (Penalty for dominant, Boost for outliers) to rebalance attention.
3. Training-Free & Agnostic: The method requires no weight updates, no retraining, and no auxiliary models. It works with any transformer-based MLLM (image or video) by modifying the attention matrix at inference time.
4. Comprehensive Validation: Extensive experiments across image and video benchmarks demonstrate consistent improvements in grounding and reductions in hallucination.

4. Experimental Results

The authors evaluated DOP-OBC on multiple state-of-the-art models (LLaVA-1.5, Video-LLaVA, InstructBLIP, Chat-UniVi, VILA, Video-LLaMA2) across various benchmarks.

• Hallucination Reduction (CHAIR & POPE):

  • On LLaVA-1.5, DOP-OBC reduced CHAIRS (hallucination rate) from 48.0 to 39.7 (a significant drop) and improved POPE-P (Precision) from 82.8 to 89.3.
  • On Video-LLaVA, CHAIRS dropped from 50.2 to 41.2, and POPE-P improved from 85.3 to 87.7.
  • These results outperform prior training-free methods like FixVPE, EDVT, and FarSight.

• General Performance Improvement:

  • Unlike some hallucination mitigation techniques that degrade general reasoning, DOP-OBC improved scores on general benchmarks like MMBench, LLaVA-W, and MM-Vet.
  • In video captioning (evaluated by GPT-4o), the method improved all five dimensions: Correctness, Consistency, Detail, Context, and Temporal understanding.

• Qualitative Analysis:

  • Visualizations of attention maps show that while baselines (like LLaVA-1.5) exhibit "attention collapse" on dominant objects (ignoring a girl next to a motorcycle), DOP-OBC maintains a balanced distribution, correctly grounding both the dominant motorcycle and the peripheral girl.

5. Significance and Conclusion

• Paradigm Shift: The paper shifts the focus from improving visual encoders to correcting decoder attention mechanics. It argues that "fairness" in attention allocation is a practical necessity for faithful multimodal generation.
• Practicality: Because DOP-OBC is a lightweight, inference-time intervention, it can be immediately deployed in existing systems without the computational cost of retraining or the complexity of integrating external guidance models.
• Future Outlook: The authors suggest that future work could integrate uncertainty modeling more deeply and extend this equity-aware decoding to multi-image and long-video dialogue settings.

In summary, DOP-OBC proves that by explicitly penalizing over-attended objects and boosting rare but confident ones, MLLMs can generate more accurate, complete, and grounded descriptions without altering their underlying architecture.