Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance

1. Problem Statement

Large Vision-Language Models (LVLMs) suffer from hallucinations, where they generate content that is grammatically coherent but factually incorrect or irrelevant to the visual input. The paper identifies Language-Prior Hallucination as a primary cause. This occurs when the model's generation process is dominated by statistical language priors (patterns learned during pre-training) rather than visual evidence.

• Mechanism of Failure: The authors observe that during the autoregressive decoding process, the logits (probability scores) for hallucinated tokens can erroneously rise and surpass those of the correct, visually-grounded tokens. This often happens because the model accumulates biases from grammar templates and dialogue structures, causing the visual context to fade as the text generation progresses.
• Limitations of Existing Solutions: Current methods to mitigate hallucinations fall into two categories:
  1. Training-based: Require costly data retraining, fine-tuning, or preference alignment (e.g., DPO), which limits scalability.
  2. Training-free (Inference-time): Methods like Contrastive Decoding or attention modification often require multiple forward passes (2x+ inference time), increase GPU memory overhead, or modify internal model architectures, reducing efficiency and generalizability.

2. Methodology: Residual Decoding (ResDec)

The authors propose Residual Decoding (ResDec), a training-free, plug-and-play decoding strategy that leverages historical token information to guide the current decoding step without additional inference costs.

A. Key Insight: Implicit Inference and Logit Evolution

The paper reveals that LVLMs possess implicit multi-token prediction capabilities. Before explicitly decoding an answer, the model's logits for previous tokens already contain signals about the correct answer.

• Observation: By analyzing the evolution of candidate token probabilities over time, the authors found a U-shaped trend in the Jensen–Shannon Divergence (JSD) between adjacent time steps.
• Three Phases of Decoding:
  1. Pre-Semantic Clarity Phase (PSAP): High JSD; the distribution is chaotic as the model navigates ambiguities.
  2. Semantic Anchoring Phase (SAP): Low JSD (the "valley" of the U-shape); the distribution stabilizes, indicating the model has firmly anchored on the core semantics.
  3. Expressive Divergence Phase (EDP): Rising JSD; the model diverges to satisfy linguistic templates and diverse expressions, making it susceptible to language priors.

B. The ResDec Algorithm

ResDec aggregates historical logits from the SAP and EDP phases to form a "residual guidance" stream, which is fused with the current decoding logits.

1. Historical Window Selection: A sliding window $W$ is used to analyze the JSD curve. The algorithm identifies the "valley" (SAP) and the subsequent rise (EDP) to define a Historical Aggregation Window ($\Delta_t$).
2. Confidence-Weighted Aggregation: Not all historical steps are equally reliable. ResDec calculates a confidence metric $C_i$ for each time step $i$ based on the entropy of the token distribution (lower entropy = higher confidence).
   • The historical residual logit is computed as a weighted sum of logits from the selected window:
     $$\text{logit}^{\text{res}}_\theta = \sum_{i \in \Delta_t} \left( \frac{C_i}{\sum C_j} \cdot \text{logit}_\theta(\hat{y}_i) \right)$$
3. Logit Fusion: The final probability distribution is a weighted blend of the original model logits and the historical residual logits:
   $$p_{\text{ResDec}} = \text{Softmax}((1-\alpha)\text{logit}_\theta + \alpha \text{logit}^{\text{res}}_\theta)$$
   Where $\alpha$ balances the influence of the historical guidance.
4. Truncation: A hyperparameter $\beta$ is used to truncate low-probability tokens to further filter out implausible outputs.

3. Key Contributions

1. Mechanism Discovery: The paper elucidates that hallucinations arise because hallucinated tokens erroneously emerge in the logit distribution and surpass genuine tokens during specific decoding stages. It identifies the U-shaped JSD pattern as a marker for stable semantic representation.
2. ResDec Strategy: Introduces a novel, training-free decoding method that requires only a single forward pass. It aggregates historical information using confidence weighting to create a residual stream that corrects language-biased predictions.
3. Efficiency: Unlike contrastive decoding methods that double inference time, ResDec incurs negligible latency and memory overhead (comparable to standard greedy decoding) because it reuses logits naturally generated during the inference process.

4. Experimental Results

The authors evaluated ResDec on three LVLMs (LLaVA-1.5, InstructBLIP, Qwen2.5-VL) across 11 benchmarks.

• Hallucination Benchmarks (POPE, CHAIR, HallusionBench):
  • POPE: ResDec achieved an average improvement of 7.84% in accuracy and 8.01% in F1 score over standard decoding across all models. It outperformed state-of-the-art (SOTA) methods like VCD, OPERA, and MemVR.
  • CHAIR: Showed significant reductions in object hallucinations (e.g., 16.57% improvement on CHAIRS for InstructBLIP).
  • HallusionBench: Achieved SOTA performance with consistent gains across all metrics.
• Comprehensive Benchmarks (MME, MMBench, MMVP, etc.):
  • ResDec improved performance on general reasoning and perception tasks, demonstrating that reducing hallucinations does not come at the cost of general capability. For instance, it improved MME scores by up to 11.65% on Qwen2.5-VL.
• Efficiency:
  • ResDec is only 0.02x slower than standard greedy decoding, whereas other SOTA methods (like OPERA) are significantly slower (up to 3.6x).
  • Memory usage is nearly identical to standard decoding.

5. Significance

• Practicality: ResDec offers a highly scalable solution for hallucination mitigation that can be deployed immediately on existing LVLMs without retraining or architectural changes.
• Theoretical Insight: The work provides a deeper understanding of the temporal dynamics of LVLM decoding, specifically how semantic stability evolves and how language priors disrupt this process.
• Broad Applicability: The method is model-agnostic and decoding-strategy-agnostic (works with Top-K, Nucleus, and Greedy sampling), making it a versatile tool for improving the reliability of multimodal AI systems in real-world applications.