SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding

1. Problem Definition

Object Hallucination remains a critical bottleneck in Vision-Language Models (VLMs). This phenomenon occurs when models generate plausible-sounding but factually incorrect responses regarding visual content (e.g., claiming an object exists when it does not).

• Root Cause: The paper identifies that existing multi-scale VLMs (like LLaVA-NeXT) often integrate visual information indiscriminately across scales. Simply upscaling images increases the number of patches exponentially, leading to information dilution and a failure to align visual attention with specific object masks.
• Theoretical Gap: While reducing patch size (increasing resolution) theoretically improves attention alignment (measured by the Attention Dice Coefficient), indiscriminate scaling introduces excessive background noise, causing the model to struggle with precise object focus.

2. Methodology: SECOND Framework

The authors propose SECOND (Selective and Contrastive Decoding), a training-free framework that mitigates hallucinations by dynamically selecting relevant visual patches and contrasting multi-stage outputs.

A. Selective Multi-Scale Feature Integration

Instead of processing all patches at all scales, SECOND employs an iterative, coarse-to-fine approach:

1. Multi-Stage Architecture: The framework utilizes multiple vision encoders processing images at different resolutions (e.g., $H/4 \times W/4$, $H/2 \times W/2$, and base resolution).
2. Dynamic Patch Selection:
   • At each stage, the model generates visual attention maps.
   • A dynamic mechanism selects the top $p_{select}\%$ of patches for the next, higher-resolution stage based on the entropy of the visual attention ($H(V)$).
   • Logic: Low entropy (clear focus) requires fewer additional patches; high entropy (unclear focus) triggers the selection of more fine-grained patches to refine the object's representation.
   • This ensures that only object-relevant information is propagated to higher resolutions, preserving object density while filtering out background noise.

B. Multi-Stage Contrastive Decoding (CD)

SECOND introduces a novel Multi-Stage Contrastive Decoding strategy, moving beyond the traditional "Amateur vs. Expert" single-step contrast:

• Amateur vs. Expert: Early stages (coarse, sparse information) act as "amateurs," while later stages (fine-grained, dense information) act as "experts."
• Hierarchical Contrast: Instead of just contrasting the final expert output with the initial amateur output, SECOND contrasts outputs across all intermediate stages.
• Logit Formulation: The final logits are calculated by amplifying the differences between stages:
  $$\text{logit}_{SECOND} = \text{logit}_{expert} + \alpha(\text{logit}_{expert} - \text{logit}_{amateur3}) + \beta(\text{logit}_{amateur3} - \text{logit}_{amateur2}) + \gamma(\text{logit}_{amateur2} - \text{logit}_{amateur1})$$
  This process amplifies the distinction between sparse and dense visual information, effectively suppressing hallucinations driven by language priors or background noise.

3. Key Contributions

1. Novel Training-Free Framework: SECOND addresses hallucinations without requiring additional model training or auxiliary vision models, leveraging existing VLM capabilities.
2. Theoretical Insight: The authors provide a theoretical analysis (Theorem 3.2 & 3.3) proving that multi-stage patch selection increases the Attention Dice Coefficient (alignment with ground truth) compared to uniform scaling, thereby reducing hallucination probability ($P_{Hal}$).
3. Multi-Stage Contrastive Decoding: Unlike prior works that focus on suppressing language priors via single-step contrast, SECOND leverages the evolutionary trajectory of visual information across scales to refine object perception.
4. Patch Selection Mechanism: A novel entropy-based dynamic selection strategy that balances the trade-off between information quantity and relevance.

4. Experimental Results

The method was evaluated on multiple VLMs (LLaVA-NeXT, LLaVA-OneVision, Yi-VL) across diverse benchmarks:

• POPE (Perceptual Hallucination Benchmark):
  • SECOND outperformed baselines (including VCD and standard LLaVA-NeXT) in 11 out of 12 experimental configurations.
  • On LLaVA-NeXT (Vicuna-7B), SECOND with CD achieved 89.7% Accuracy and 89.2% F1 on MSCOCO, significantly surpassing the baseline (87.7% Acc, 86.5% F1).
• General Tasks (VQAv2, MMStar, MMBench):
  • SECOND demonstrated consistent improvements in open-ended VQA and reasoning tasks.
  • On MMStar, SECOND improved the score from 37.3 (baseline) to 38.6, proving that improved perception translates to better reasoning without degrading high-level capabilities.
• Ablation Studies:
  • Patch Selection: Dynamic selection outperformed fixed ratios (e.g., 70% fixed) and using all patches (100%), confirming the necessity of selective filtering.
  • Multi-Stage vs. Single-Stage CD: Multi-stage CD showed superior robustness on general tasks (MMStar) compared to single-stage CD, even if single-stage CD performed similarly on POPE.

5. Significance

• Paradigm Shift: The paper challenges the "more resolution is always better" assumption in VLMs, demonstrating that selective integration of multi-scale data is superior to indiscriminate accumulation.
• Efficiency: By being training-free and utilizing existing encoders with interpolation, SECOND offers a computationally efficient path to reducing hallucinations.
• Human-Like Perception: The approach mimics human visual perception (scanning broadly then focusing on details), providing a biologically inspired solution to a technical problem.
• Future Impact: The work highlights the untapped potential of multi-scale applications in VLMs, suggesting that future architectures should prioritize dynamic, attention-guided feature integration over static, uniform scaling.

Code Availability: The authors have made the code publicly available at https://github.com/AIDASLab/SECOND.