Technical Summary: Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

Problem Definition

Large Vision-Language Models (LVLMs) have advanced multimodal AI by integrating visual understanding with large language models (LLMs). However, they remain susceptible to hallucination, where models generate fluent text that contradicts visual inputs. Recent studies attribute this to language bias: a tendency for LVLMs to over-rely on internal linguistic priors while neglecting visual signals.

Despite the prevalence of this issue, existing analyses remain largely empirical, lacking a systematic, principled understanding of its root cause. Current mitigation strategies are often categorized as training-free (post-processing) or training-based (fine-tuning), but they frequently fail to address the underlying mechanism driving the bias. The paper posits that the core issue is a modality misalignment during training, where optimization objectives inadvertently prioritize textual improvements over balanced multimodal understanding.

Methodology

1. In-Depth Analysis of Modality Misalignment

The authors conduct a rigorous quantitative analysis of two primary training paradigms: Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO).

• Decomposition of Training Dynamics: The study tracks two key quantities during training:
  • $R$ (Reward): The gain in conditional probability on full multimodal input ($\pi(y|x, v)$).
  • $B$ (Bias): The gain in conditional probability on text-only input ($\pi(y|x)$).
• Key Finding: In both VIT and DPO, the improvement in text-only likelihood ($B$) often rivals or exceeds the improvement in multimodal likelihood ($R$). This indicates that the models are drifting toward behavior resembling conventional language modeling rather than learning to ground responses in visual data.
• Formal Definition: Language bias is formally defined as $B = \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$, quantifying the model's reliance on internal language priors relative to a reference model.

2. Proposed Mitigation Strategies

To address this misalignment, the paper proposes two simple, training-based interventions that require no additional data or auxiliary models.

A. Language Bias Regularization (LBR) for Visual Instruction Tuning

• Context: Applied during the VIT stage where language bias is nascent.
• Mechanism: Introduces a regularization term to the standard Maximum Likelihood Estimation (MLE) loss.
  • Loss: $L'_{VIT} = L_{VIT} + \alpha \cdot |B|$
  • The term $|B|$ penalizes the magnitude of the language bias, constraining the model's text-only output distribution to remain close to the reference model.
• Goal: Prevents the model from over-relying on linguistic modality at the expense of visual grounding.

B. Language Bias Penalty (LBP) for Direct Preference Optimization

• Context: Applied during DPO, where the model has already acquired significant language bias from prior instruction tuning.
• Mechanism: Introduces a penalty term to the DPO objective that actively pushes the language bias $B$ toward negative values.
  • Loss: $L'_{DPO} = L_{DPOM} - \gamma \cdot \log \sigma(B)$
  • This encourages the model to "unlearn" the bias acquired during VIT and strengthens reliance on visual information.
• Goal: Enhances trustworthiness in visually grounded tasks by suppressing the tendency to mimic linguistic patterns in preference data without visual support.

Key Contributions

1. Systematic Analysis: The paper provides a novel perspective on modality misalignment, offering rigorous quantitative evidence that LVLMs systematically underutilize visual signals during standard training paradigms (VIT and DPO).
2. Novel Methods: It introduces LBR and LBP, two versatile, easily deployable methods that integrate seamlessly into existing training pipelines without requiring extra data or models.
3. Comprehensive Evaluation: The authors validate their approach across models of varying scales (LLaVA-1.5 7B/13B, LLaVA-NEXT 3B) and a wide range of datasets, demonstrating strong generalization capabilities.

Experimental Results

General Capabilities (LBR)

• Performance: LBR consistently improves performance across more than ten general-purpose benchmarks, including text-intensive tasks (e.g., ChartQA, TextVQA), visual QA (e.g., GQA, VizWiz), and image captioning (e.g., COCO Captions).
• Comparison: LBR outperforms the vanilla VIT baseline on the vast majority of metrics, validating its ability to foster better multimodal alignment without sacrificing general capabilities.

Trustworthiness and Hallucination (LBP)

• Performance: LBP achieves state-of-the-art performance on hallucination-focused benchmarks, including MMHalBench, AMBER, and Object HalBench.
• Impact:
  • On the challenging MMHalBench, LBP reduces the hallucination rate of LLaVA-v1.5-7B by 27%.
  • The method significantly improves model robustness, particularly in long-form generation where language bias is most pronounced.
  • LBP matches or exceeds the performance of GPT-4V on specific sub-tasks of AMBER and Object HalBench using only 5.7K preference pairs.
• Generalization: LBP maintains or improves general capabilities (e.g., MME, MMBench) while reducing hallucinations, successfully circumventing the typical trade-off between alignment and performance.

Human Evaluation

• A targeted human evaluation on 100 COCO images confirmed that both LBR and LBP significantly reduce hallucinations across six dimensions (Existence, Attribute, State, Number, Action, Relation) compared to baselines.
• Qualitative case studies demonstrate that LBP-aligned models produce factually accurate descriptions, whereas baselines often hallucinate objects, attributes, or relationships.

Significance and Claims

The paper claims that its work offers both a deeper theoretical understanding of language bias and a practical path toward more reliable LVLMs.

• Root Cause Identification: By identifying modality misalignment as the root cause, the work moves beyond superficial fixes to address the fundamental training dynamics that lead to hallucination.
• Efficiency: The proposed methods are highlighted for their simplicity and efficiency; they require no additional data or auxiliary models, making them highly accessible for adoption.
• Trustworthiness: The results suggest that explicitly targeting and reducing language bias is a potent strategy for enhancing the trustworthiness of LVLMs, which is critical for deploying these models in high-stakes, real-world applications where factual grounding is paramount.

The authors conclude that while existing automated benchmarks have limitations in capturing the nuances of language bias, their targeted human evaluations and extensive quantitative results provide strong empirical support for the efficacy of LBR and LBP in advancing the alignment of LVLMs.