Technical Summary: Mitigating Manifold Departure via Uncertainty-Aware Subspace Rectification

1. Problem Statement

Multimodal Large Language Models (MLLMs) frequently suffer from object hallucination, generating content that is fluent and plausible but inconsistent with visual inputs. This phenomenon is primarily attributed to an over-reliance on language priors learned during pre-training, which can override weak or ambiguous visual evidence.

While recent training-free decoding strategies attempt to mitigate this by penalizing language priors (e.g., via contrastive decoding), the authors identify a critical flaw in these approaches: they treat language priors as uniformly harmful. In reality, priors play a dual role:

• Harmful: They induce hallucinations when visual and linguistic evidence conflict (e.g., predicting a "yellow banana" when the image shows a blue one).
• Helpful: They stabilize and sharpen generation when aligned with visual evidence (e.g., reinforcing "yellow" when the image is indeed a yellow banana).

Indiscriminate suppression of priors disrupts the model's internal semantic manifold, pushing hidden states into low-density regions and degrading performance even in valid cases. The authors term this failure mode Manifold Departure.

2. Methodology: Manifold-Guided Adaptive Projection (MGAP)

To address Manifold Departure, the paper proposes MGAP, a geometry-aware, training-free decoding framework. Unlike global linear suppression methods, MGAP performs subspace-selective updates that preserve the orthogonal semantic structure of the model.

2.1. Offline: Label-Free Language Prior Subspace Construction

MGAP first constructs a low-dimensional representation of language priors without requiring labels or image content:

1. Data Collection: The model processes a set of textual queries with a "blind" input (e.g., a null or perturbed image) to extract final-layer hidden states $\{h_{blind}^{(i)}\}$.
2. Subspace Estimation: A centered matrix of these blind states is formed, and Singular Value Decomposition (SVD) is applied. The top-$K$ right singular vectors ($V_{prior}$) define the language-prior subspace, capturing dominant variations induced by linguistic regularities.

2.2. Online: Consistency-Aware Adaptive Projection

During standard multimodal decoding, MGAP intervenes at each step $t$ to selectively attenuate priors only when necessary:

1. Projection: The current hidden state $h_{orig}$ is projected onto the prior subspace: $h_{proj} = V_{prior}V_{prior}^\top h_{orig}$.
2. Consistency Check (Protection): The alignment between the original state and the prior projection is measured via cosine similarity. A mismatch score $\delta = 1 - \cos(h_{orig}, h_{proj})$ is computed. A small $\delta$ implies the state is consistent with the prior (no suppression needed).
3. Uncertainty Gating: Predictive uncertainty is estimated using the Shannon entropy ($H$) of the token distribution. A gating factor $\gamma = \tanh(\lambda H)$ is derived, amplifying intervention only when the model is uncertain.
4. Adaptive Correction: The corrected hidden state is computed as:
   $$h_{cut} = h_{orig} - \alpha \cdot \gamma \cdot \delta \cdot h_{proj}$$
   This formulation ensures that the correction is bounded and selective. If the visual evidence aligns with the prior (low $\delta$) or the model is confident (low $H$), the operation approximates an identity mapping, preserving the semantic manifold.

3. Key Contributions

The paper makes three primary contributions:

1. Identification of Manifold Departure: The authors formally define and analyze "Manifold Departure" as a geometric failure mode where uniform linear suppression pushes hidden states away from the high-density semantic manifold, explaining why existing training-free methods degrade performance even when priors are correct.
2. Proposal of MGAP: A novel decoding framework that learns a language-prior subspace from blind states and applies uncertainty-aware, subspace-selective projection. This allows for the suppression of hallucinations without disturbing orthogonal semantic components.
3. Theoretical and Empirical Validation: The paper provides theoretical proofs showing that MGAP's updates are bounded and subspace-selective. Extensive experiments demonstrate that MGAP achieves superior trade-offs between hallucination mitigation and descriptive fidelity compared to prior baselines.

4. Experimental Results

The authors evaluated MGAP on two representative backbones (LLaVA v1.5-7B and Qwen3-VL-8B) across multiple benchmarks:

• POPE (Object Hallucination): MGAP consistently outperformed vanilla decoding and other training-free baselines (VCD, ICD, HalTrapper, DeCo, MoD) across Random, Popular, and Adversarial splits. Notably, while methods like VCD degraded performance on standard splits, MGAP improved Precision and F1 scores while maintaining high Accuracy.
• CHAIR (Image Captioning): MGAP significantly reduced hallucination rates (lower CHAIRs and CHAIRi scores) compared to all baselines, while simultaneously improving precision. This indicates that hallucination mitigation was achieved without overly conservative generation.
• AMBER Benchmark: MGAP generalized well to this diverse benchmark, showing that the learned prior subspace captures transferable structural properties of hallucinations beyond specific evaluation protocols.
• Efficiency: Unlike contrastive decoding methods that require multiple forward passes (e.g., VCD, ICD), MGAP operates within a single forward pass. It reduced inference latency by approximately 50% compared to contrastive baselines while remaining fully training-free.

5. Significance and Claims

The paper claims that the indiscriminate suppression of language priors is a fundamental limitation of current training-free decoding strategies. By reframing the problem through the lens of representation geometry, the authors demonstrate that Manifold Departure is the root cause of performance degradation in existing methods.

The significance of MGAP lies in its selective intervention mechanism. By modeling priors as a low-rank subspace and applying adaptive, uncertainty-aware gating, MGAP avoids the "blunt instrument" approach of global linear shifts. The authors assert that this approach allows MLLMs to retain the stabilizing benefits of language priors when they are correct, while effectively suppressing them only when they conflict with visual evidence, thereby achieving a more robust and trustworthy decoding process.

The work concludes that hallucination priors are not merely artifacts of specific benchmarks but exhibit transferable structural properties in the representation space, suggesting that selective prior correction is a generalizable mechanism for improving large vision-language models.