VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

1. Problem Statement

Medical Vision-Language Models (VLMs) are increasingly used for clinical tasks like Visual Question Answering (VQA) and radiology report generation. However, they suffer from hallucinations, where the model generates factually incorrect but clinically plausible responses based on language priors (statistical patterns learned during training) rather than actual visual evidence.

• The Risk: In high-stakes medical scenarios, such hallucinations can lead to misdiagnosis and dangerous automation bias.
• The Limitation of Existing Methods: General-domain hallucination mitigation techniques (e.g., Visual Contrastive Decoding [VCD], DoLA, OPERA) often fail in the medical domain. The paper notes that these methods can degrade performance by over 6% on medical benchmarks. This is attributed to the unique nature of medical VQA: shorter responses, specialized vocabulary, and the critical need for precise anatomical terms where aggressive correction can be harmful.

2. Core Insight

The authors identify a distinct behavioral signature differentiating hallucinated tokens from visually grounded tokens:

• Visually Grounded Tokens: When visual information is degraded (e.g., by adding noise), the probability of these tokens decreases because the model loses the supporting visual evidence.
• Hallucinated Tokens: When visual information is degraded, the probability of these tokens maintains or increases because they are driven by language priors rather than the image content.

3. Methodology: VGS-Decoding

The proposed solution is Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free inference-time method.

A. Visual Grounding Score (VGS)

The method quantifies the visual dependency of each token using a normalized probability difference.

1. Input Distortion: An input image $V$ is distorted to create $V'$ by adding Gaussian and Poisson noise.
2. Probability Comparison: The model generates token probability distributions for both the original image ($P_{orig}$) and the distorted image ($P_{dist}$).
3. VGS Calculation: For each token $t$, the VGS is calculated as:
   $$VGS(t) = \frac{P_{orig}(t) - P_{dist}(t)}{P_{orig}(t) + P_{dist}(t)}$$
   • Range: $[-1, +1]$.
   • Positive VGS: Indicates the token is visually grounded (probability dropped when image was corrupted).
   • Negative VGS: Indicates a potential hallucination (probability remained stable or increased).
   • Zero VGS: Indicates visual independence (e.g., function words).

B. Adaptive Reweighting

Unlike previous methods that use fixed-weight subtraction, VGS-Decoding uses multiplicative reweighting to adaptively adjust token probabilities:
$$P_{final}(t) \propto P_{orig}(t) \times \max(1 + \alpha \cdot VGS(t), \delta)$$

• $\alpha$ (Strength): Controls the intensity of the correction.
• Mechanism:
  • Tokens with positive VGS are amplified (factor $> 1$).
  • Tokens with negative VGS are suppressed (factor $< 1$).
  • Tokens with near-zero VGS remain largely unchanged, preserving linguistic coherence.

C. Computational Complexity

• Requires two forward passes per decoding step (one for original, one for distorted image).
• Results in approximately 2× inference overhead compared to standard greedy decoding.
• Significantly more efficient than beam-search-based methods like OPERA.

4. Key Contributions

1. Identification of Domain Failure: Demonstrated that existing hallucination mitigation methods (VCD, DoLA, OPERA) often degrade performance on medical VQA benchmarks, necessitating domain-specific solutions.
2. Novel Metric (VGS): Introduced a bounded, interpretable metric ($[-1, +1]$) that quantifies token-level visual dependency through normalized probability differences.
3. Training-Free Framework: Proposed VGS-Decoding, which achieves consistent improvements across diverse medical VLMs without requiring additional model training or fine-tuning.

5. Experimental Results

The method was evaluated on two medical VQA benchmarks (VQA-RAD and MIMIC-Diff-VQA) across three diverse models (LLaVA-Med, CheXagent, and MedGemma).

• Performance Gains:
  • VGS-Decoding is the only method that consistently improved performance across all models and datasets.
  • Overall Improvement: Up to +9.12% on MIMIC-Diff-VQA (MedGemma) and +8.16% on VQA-RAD.
  • Open-Ended Recall: Up to +8.98% improvement.
• Comparison with Baselines:
  • Existing methods frequently caused performance drops (e.g., VCD dropped LLaVA-Med performance by -5.93%).
  • VGS-Decoding maintained or improved performance even on highly specialized models like CheXagent, whereas other methods degraded them.
• Ablation Studies:
  • Confirmed that both the VGS metric and the multiplicative reweighting are necessary; using fixed weights (VCD-style) degraded performance.
  • Optimal reweighting strength ($\alpha$) varied by model (1.0 to 2.0), but $\alpha=1.0$ provided consistent gains.

6. Significance

• Clinical Safety: By effectively suppressing hallucinations without compromising the model's ability to generate precise medical terminology, VGS-Decoding reduces the risk of misdiagnosis in automated clinical tools.
• Practicality: As a training-free method with only 2× inference overhead, it is immediately deployable in clinical settings without the computational cost of retraining large models.
• Generalizability: The approach works across models of different sizes (3B to 7B parameters) and architectures, suggesting a robust solution for the broader medical AI community.