Technical Summary: The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation

Problem Definition: Recorruption

While Multimodal Large Language Models (MLLMs) are increasingly integrated with Retrieval-Augmented Generation (RAG) to ground their outputs in external knowledge, this paper identifies a severe, counterintuitive failure mode termed recorruption. Recorruption occurs when the introduction of external documents—even perfectly accurate "oracle" context—causes a capable model to abandon an initially correct prediction derived from visual evidence and flip to an incorrect one.

The authors demonstrate that in high-stakes domains such as medical radiology, social fairness, and geospatial analysis, RAG can actively degrade performance. Instead of confirming the correct answer, the model ignores explicit visual evidence (e.g., pathologies in X-rays or gender cues in images) in favor of textual distractors. This phenomenon is driven by a "zero-sum" competition within the model's unified context window, where massive injections of retrieved text suppress visual signals.

Mechanistic Diagnosis

Through a granular analysis of internal attention matrices at the "bottleneck" layer (the final pre-filling attention operation where the model synthesizes the prompt to generate the first token), the authors identify two primary drivers of recorruption:

1. Visual Blindness (Cross-Modal Dominance): Retrieved text overwhelmingly absorbs the attention budget. This leads to a systemic suppression of Visual Attention Mass ($M_{vis}$) and a sharp decline in Visual Attention Sharpness ($S_{vis}$). The model's attention to the image becomes dangerously diffuse, effectively blinding it to specific visual regions.
2. Textual Positional Bias (Multimodal "Lost-in-the-Middle"): The model exhibits a structural bias toward the boundaries of the text sequence (primacy or recency bias), similar to the "lost-in-the-middle" effect in LLMs. The model mechanically copies text from the beginning or end of the retrieved context rather than performing grounded multimodal reasoning.

The Illusion of Success

A critical finding is the Illusion of Success. The authors show that the internal attention profiles of "successful" RAG outcomes (where the model happens to output the correct answer) and "recorruption" failures are statistically indistinguishable. Both exhibit diffuse visual focus and aggressive boundary bias. Consequently, many seemingly correct RAG outputs are merely positional coincidences where the model's blind textual copying bias aligns with the ground-truth location, rather than the result of robust reasoning.

Methodology: Bottleneck Attention Intervention for Recovery (BAIR)

To address these vulnerabilities without requiring model retraining or fine-tuning, the authors propose BAIR, a parameter-free, inference-time framework. BAIR operates exclusively at the pre-softmax attention matrices during the pre-filling stage, manipulating the attention distribution to restore multimodal grounding. It employs a dual-recovery mechanism:

1. Visual Sharpness and Mass Recovery (VSMR):

   • BAIR first performs a lightweight, uncorrupted reference pass (image + instruction only) to dynamically calculate instance-specific target values for Visual Attention Mass ($M_{target}$) and Sharpness ($S_{target}$).
   • It then modulates the visual logits in the RAG context using a temperature scalar $T$ (solved via bisection search) to restore the target sharpness.
   • A global scalar shift $\alpha$ is applied to restore the target mass, ensuring the visual signal is not diluted by the textual context.
   • A mass interpolation factor $\alpha_v$ allows for the active amplification of visual evidence beyond the baseline.

2. Position-Aware Textual Penalization (PATP):

   • To counteract the "lost-in-the-middle" and boundary biases, BAIR applies a data-driven, asymmetric penalty to the textual logits.
   • It calculates regional logit means for the first 20% and last 20% of the text. If a boundary exhibits a logit spike (indicating excessive attention), a quadratic decay penalty is applied to those specific tokens.
   • This dynamically dampens distracting boundary tokens while preserving central instructional content, without requiring manual hyperparameter tuning for different document lengths.

Experimental Results

The authors evaluated BAIR across three domains:

• Medical Factuality: IU-Chest dataset (MedGemma-4B, CheXagent-8B).
• Social Fairness: FACET dataset (Qwen2.5-VL-7B, DeepSeek-VL-7B).
• Geospatial Factuality: NWPU-RESISC45 dataset (SkySenseGPT-7B, EarthDial).

Key Findings:

• Restoration of Grounding: BAIR successfully restores visual saliency and reduces the Degradation Rate (DR)—the proportion of initially correct predictions corrupted by RAG—across all models and datasets.
• Improved Correction-to-Degradation Ratio: While standard RAG often improves Correction Rate (CR) at the cost of high DR, BAIR significantly improves the CR/DR ratio. For instance, on the FACET dataset with Qwen2.5-VL-7B, applying BAIR to Standard RAG increased the CR/DR ratio from 4.92 to 10.84.
• Complementarity: BAIR functions as a calibration layer that can be integrated on top of existing intervention methods (e.g., LongLLMLingua, MS-PoE, MAD-RAG), consistently shifting their performance trajectories toward higher accuracy and lower degradation.
• Efficiency: BAIR introduces negligible computational overhead (approx. 5–10% runtime increase) and does not increase peak GPU memory usage, as it avoids retraining and operates solely at inference time.

Significance and Claims

The paper claims to provide both a critical diagnostic lens and a practical cure for hidden vulnerabilities in multimodal RAG architectures. Its significance lies in:

1. Formalizing Recorruption: Identifying that external context can paradoxically degrade performance by overriding correct visual perceptions, a phenomenon distinct from standard hallucination.
2. Exposing the Illusion of Success: Demonstrating that high RAG accuracy metrics can be misleading, often stemming from positional coincidences rather than genuine multimodal understanding.
3. Parameter-Free Intervention: Offering a lightweight, architecture-agnostic solution (BAIR) that restores visual grounding and suppresses textual bias without the cost of model retraining or fine-tuning.

The authors emphasize that while BAIR improves reliability, it is an assistive tool. In high-stakes domains like medicine, it should not replace human clinical judgment, and its deployment requires human oversight and domain-specific safety constraints.