Technical Summary: Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs

1. Problem Statement

Multimodal Large Language Models (MLLMs) suffer from object hallucinations, where models generate fluent, coherent outputs that include objects, attributes, or relations absent from the visual input. This failure mode reflects weak visual grounding and poses significant risks in high-stakes domains such as autonomous driving and healthcare.

Contrastive Decoding (CD) has been proposed as a training-free inference strategy to mitigate these hallucinations. The core idea involves contrasting the output distribution of an "expert" model (conditioned on the original input) against a weaker, hallucination-prone "amateur" model (conditioned on a perturbed input, such as a noisy image or altered instruction). Tokens favored by the amateur model are penalized, theoretically suppressing hallucinations. While CD variants like Visual Contrastive Decoding (VCD), Instruction Contrastive Decoding (ICD), and Self-Introspective Decoding (SID) have reported performance gains on benchmarks like POPE, recent work by Yin et al. (2026) questioned whether these gains reflect genuine improvements in visual grounding or are merely artifacts of decoding constraints.

2. Methodology

This study conducts a rigorous reproducibility and extension of the analysis by Yin et al. (2026). The authors evaluate newer models (LLaVA-v1.5-7B/13B and Qwen2.5-VL-7B) across discriminative (POPE, MME) and generative (CHAIR, NoCaps, LLaVA-Bench) benchmarks.

The methodology is structured around two primary claims and several novel investigative experiments:

A. Reproduction of Core Claims

1. Unidirectional Bias: The study tests whether CD gains arise from a unidirectional shift in the output distribution toward "Yes" answers, rather than improved object recognition.
2. Adaptive Plausibility Constraint (APC): The study verifies whether the APC, a component of CD that restricts generation to tokens likely under the expert distribution, effectively degrades sampling-based CD into an approximation of greedy search.

B. Extended Investigative Experiments

To decompose the nature of CD's performance, the authors introduce several novel analyses:

• Flip Analysis: Decomposing "No→Yes" flips into True Positives (genuine corrections) and False Positives (injected hallucinations) to calculate flip precision and TP recall.
• Label Imbalance Stress-Testing: Evaluating CD on POPE subsets with skewed label ratios (25/75 and 75/25 Yes/No) to determine if gains persist when the metric (Balanced Accuracy) is immune to threshold shifts.
• Jaccard Similarity Analysis: Measuring the lexical overlap between APC-augmented sampling and greedy decoding across varying APC thresholds ($\beta$) to quantify the shift toward greedy behavior.
• Per-Token Logit Analysis: Analyzing the contrastive adjustment $d = E - A$ (Expert minus Amateur logits) to determine if the mechanism selectively suppresses hallucinated tokens or amplifies them.
• Gaussian Noise Proxy: Replacing the amateur branch with Gaussian noise to test if the specific structure of the amateur model is necessary for CD's effects.
• Layer-wise Logit-Lens Analysis: Using linear probes on transformer layers to determine if CD's shift is selective (targeting hallucinations) or a uniform translation of the decision margin.

3. Key Results

Reproduction of Claims

• Unidirectional Bias Confirmed (with nuance): For VCD and SID, the study confirms a unidirectional shift toward "Yes" outputs, particularly on adversarial subsets. However, ICD exhibits a reversal, shifting toward "No" while remaining accuracy-neutral. This suggests the direction of the shift depends on the perturbation modality, not the CD paradigm itself.
• APC Degrades to Greedy: The study confirms that the APC collapses sampling into greedy-like decoding. As the threshold $\beta$ increases, the Jaccard similarity between APC-augmented outputs and greedy search rises significantly, while similarity to direct sampling remains flat. At the default $\beta=0.1$ used by VCD, the output is already dominated by the greedy bias.

Extended Findings

• Spurious Gains vs. Genuine Correction:
  • Flip Precision: Non-visual controls (Prompt-Based Adjustment and Output Layer Modification) achieve comparable or superior flip precision to VCD and SID.
  • TP Recall: Non-visual controls correct a higher proportion of false-negative errors than CD methods.
  • Hallucination Injection: CD methods introduce new hallucinations at rates comparable to non-visual controls.
• Label Imbalance Sensitivity: When evaluated on skewed datasets (25/75 or 75/25), CD methods fail to improve Balanced Accuracy. In fact, their performance often degrades relative to the baseline, indicating that their gains on standard POPE are driven by shifting the decision threshold to match the dataset's class balance rather than improving visual grounding.
• Mechanistic Ineffectiveness:
  • Logit Analysis: The contrastive adjustment $d$ does not selectively suppress hallucinated tokens. On LLaVA, it amplifies both real and hallucinated tokens. On Qwen, it suppresses real objects more than hallucinations.
  • Proxy Validation: A Gaussian noise proxy (discarding the amateur branch entirely) yields performance on CHAIR statistically indistinguishable from real CD methods, suggesting the amateur branch provides no unique benefit.
  • Layer-wise Selectivity: While object presence is linearly decodable from intermediate layers (Layer 8), CD's shift is non-selective at all depths. It reinforces the false "Yes" for the vast majority of hallucinations. A single scalar bias added to the decision margin reproduces CD's entire POPE gain, confirming the effect is a threshold shift, not a mechanistic correction.

4. Contributions

The paper makes the following specific contributions:

1. Reproduction: It validates the central claims of Yin et al. (2026) on a newer suite of models (LLaVA-v1.5, Qwen2.5-VL) and across multiple datasets.
2. Decomposition of Gains: It provides granular evidence (flip precision, TP recall, balanced accuracy) that CD's reported improvements are spurious artifacts of output distribution shifts rather than genuine hallucination mitigation.
3. Mechanistic Dissection: Through logit-level and layer-wise analyses, it demonstrates that CD fails to selectively suppress hallucinations and instead applies a uniform translation to the decision boundary.
4. Novel Baselines: It introduces and validates a Gaussian noise proxy and stress-tests CD under extreme label imbalances, offering new tools for auditing inference-time interventions.

5. Significance and Claims

The paper concludes that current contrastive decoding strategies should not be relied upon as effective hallucination mitigation.

• The "Mirage" of Performance: The apparent gains on benchmarks like POPE are shown to be "spurious," arising from the model's output distribution shifting to better match the dataset's class balance or from the APC collapsing sampling into greedy search.
• Lack of Visual Grounding: The study demonstrates that CD does not improve the model's ability to distinguish between present and absent objects. In some cases (e.g., Qwen), it actively penalizes correct objects while sparing hallucinations.
• Call for Rigor: The authors argue that the field must move beyond aggregate accuracy metrics that are sensitive to threshold shifts. They propose that future training-free interventions must be audited for selectivity (does it target hallucinations specifically?) and mechanistic validity (does it correct the underlying generation process?) rather than just benchmark scores.

The work challenges the community to develop more reliable approaches for mitigating hallucinations, emphasizing that current "training-free" fixes may merely be tuning the decision boundary rather than solving the grounding problem.