Technical Summary: When Language Overwrites Vision

1. Problem Statement

Vision-Language Models (VLMs), particularly those based on decoder architectures, increasingly power high-stakes applications but suffer from a critical failure mode: object hallucination. These models confidently describe visual content that is not present in the input image.

The paper identifies the root cause not merely as a lack of alignment, but as geometric over-alignment.

• The Modality Gap: Visual and linguistic representations naturally occupy disjoint manifolds within a shared latent space.
• The Over-Alignment Mechanism: To satisfy the requirements of cross-modal attention mechanisms in decoder-based VLMs, models aggressively force visual embeddings to overlap with the text manifold.
• The Consequence: This forced collapse injects a statistical linguistic bias into the visual representations. The model exploits a "structural shortcut" where the top principal components (PCs) of the vision embedding space are hijacked to serve as a modality anchor for text. Consequently, fine-grained visual evidence is overshadowed by high-probability textual patterns, causing the model to default to linguistic priors rather than attending to the actual image.

Prior work has either attempted to aggressively close the modality gap (potentially exacerbating over-alignment) or suppressed hallucinations via expensive black-box decoding strategies (e.g., contrastive decoding) without addressing the underlying geometric cause.

2. Methodology

The authors propose a geometric debiasing framework that treats the textual bias as a linearly isolable subspace within the visual embeddings.

A. Mechanistic Analysis

• Text Manifold Definition: The authors define a coordinate system for the text manifold by performing Principal Component Analysis (PCA) on token embeddings derived from pretraining captions. They demonstrate that the top PCs form a universal, dataset-agnostic manifold representing statistical linguistic bias.
• Layer-wise Alignment: Analysis reveals that in standard VLMs, vision tokens aggressively align with this text manifold in early-to-middle decoder layers.
• Visual Overwrite: Linear probing shows that as visual tokens propagate through the network, their decodability degrades because the top PCs act as a "structural mask," overriding true visual signals with textual syntax and priors.

B. Proposed Remedies

The paper introduces two complementary methods to remove this bias:

1. Training-Free Inference Strategy (Post-hoc):

   • At inference time, the method projects visual embeddings orthogonally to the top $k$ principal components of the text manifold.
   • Formula: $\vec{v}_{debiased} = \vec{v} - \sum_{i=1}^{k} ((\vec{v} - \vec{m}) \cdot \vec{k}_i)\vec{k}_i$, where $\vec{v}$ is the vision embedding, $\vec{m}$ is the text manifold mean, and $\vec{k}_i$ are the top PCs.
   • This operation removes the "textual anchor" without requiring additional model training or computational overhead.

2. Bias-Aware Fine-Tuning Paradigm:

   • During the fine-tuning process, the same top-$k$ PC removal is applied to the output of the cross-modal projector before it enters the LLM decoder.
   • This structurally deprives the network of statistical linguistic shortcuts during parameter updates, forcing the model to natively encode visual evidence for grounding.

3. Key Contributions

• Quantitative Characterization of Over-Alignment: The paper provides the first explicit analysis showing that linguistic bias in decoder-based VLMs concentrates in the top principal components of the vision embedding space, actively degrading visual decodability.
• Novel Geometric Framework: A task-agnostic method to isolate and measure the penetration of linguistic bias, identifying that this bias is heavily concentrated in the top PCs of the latent space.
• Training-Free Debiasing: A principled, post-hoc intervention that significantly boosts visual grounding on diverse benchmarks with zero additional computational cost over the base model.
• Bias-Aware Fine-Tuning: A specialized training framework that forces the model to rely on visual evidence by removing statistical linguistic shortcuts during the training loop.

4. Experimental Results

The methods were evaluated on multiple architectures (Qwen2.5-VL-7B, LLaVA-1.5-7B) and benchmarks.

• Hallucination Reduction:

  • POPE: Improved from 87.3 to 88.4 (Qwen) and 82.8 to 83.8 (LLaVA).
  • CHAIR: Significant reductions in hallucination rates. For Qwen2.5-VL-7B, CHAIR dropped from 5.14 to 4.23 (-17.7%); for LLaVA-1.5-7B, it dropped from 9.20 to 6.74 (-26.7%).
  • AMBER: Substantial improvements in both instance-level and sentence-level metrics.
  • Comparison: The training-free method outperformed or matched computationally expensive black-box baselines like VCD, SID, and DMAS.

• Long-Form Captioning (CLAIR Score):

  • Evaluated on VRSBench and TextCaps using the CLAIR score (which penalizes hallucinated content).
  • The bias-aware fine-tuning paradigm consistently outperformed both zero-shot and standard Supervised Fine-Tuning (SFT) baselines.
  • Example: On VRSBench with Qwen2.5-VL-7B, the score improved from 72.30 (Standard SFT) to 74.70.

• Logit Lens Analysis:

  • Tracing latent representations revealed that baseline models decoded visual patches as high-probability syntax markers (e.g., punctuation, prepositions).
  • After geometric debiasing, the models successfully decoded fine-grained visual semantics (e.g., "bench," "refrigerator," "deck"), confirming the removal of the syntactic protocol.

5. Significance and Claims

The paper argues that the prevailing assumption—that closing the modality gap is the correct strategy—is flawed. Instead, a deliberate modality gap should exist to preserve the geometric integrity of visual representations.

• Core Insight: Forcing visual and text spaces to fully overlap contaminates visual representations with spurious statistical regularities of language.
• Implication: By explicitly projecting out the top textual PCs, the model is forced to attend to actual visual evidence rather than relying on linguistic priors.
• Limitations Acknowledged: The authors note that the current decoder-based architecture, which requires compressing fine-grained visual space into a text manifold, may be fundamentally flawed. They suggest future architectures might need separate, modality-specific encoders. Additionally, the method relies on the assumption that textual bias can be perfectly isolated into a linear subspace, which may struggle with highly out-of-distribution domains.

In conclusion, the work demonstrates that hallucinations in VLMs are not just a decoding issue but a representational one caused by geometric over-alignment, and that this can be mitigated through principled geometric debiasing.