Technical Summary: Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

1. Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in cross-modal retrieval tasks involving multiple images. However, these models suffer from severe position bias, where prediction accuracy is dominated by the input order of images rather than their semantic relevance to the query.

• The Phenomenon: In multi-image retrieval tasks (e.g., selecting the correct image from a sequence of $N$ candidates), MLLMs exhibit irrational preferences for specific positions (e.g., the 7th position) while neglecting others, regardless of the visual content. This leads to drastic performance fluctuations: accuracy can reach ~99% if the ground truth aligns with a preferred position but collapse to 0% if it falls into a neglected one.
• Root Cause: This bias stems from the autoregressive architecture inherited from Large Language Models (LLMs). While temporal positional encodings and causal attention masks are beneficial for text generation, they introduce unintended structural asymmetries when applied to non-temporal visual sequences, making predictions highly sensitive to input ordering.
• Limitations of Existing Solutions:
  • Prompting: Instruction-based or Chain-of-Thought (CoT) prompting fails to adapt to the dynamic nature of this bias.
  • Static Calibration: Methods like PriDe assume position bias is a static, content-independent offset (a global prior). However, empirical analysis reveals that bias in MLLMs is conditional, shifting dynamically based on the target's location. Static priors fail to resolve this conditional ambiguity.

2. Key Insight: Logit-Attention Divergence

Through empirical analysis, the authors identify a critical phenomenon termed Logit-Attention Divergence:

• Observation: MLLMs often "look at" the correct answer (internal attention maps are well-aligned with the relevant visual evidence) but "say" the wrong one (output logits are heavily biased by positional priors).
• Implication: The failure is not in semantic perception but in decision calibration. Strong structural priors during the decoding stage override correct internal features, leading to erroneous predictions despite accurate visual grounding.

3. Methodology: Attention-Guided Debiasing

The paper proposes a training-free, inference-time framework that leverages intrinsic attention signals to correct position bias. The method requires only a minimal calibration set (e.g., 5 samples) and negligible computational overhead.

3.1. Conditional Bias Estimation

Instead of a global static prior, the method estimates conditional position bias using a counterfactual statistical approach:

• Symmetrized Calibration: A small calibration set is constructed via cyclic permutations of candidate images and ground truth indices. This ensures the ground truth traverses every position uniformly.
• Decomposition: The observed generation probability is modeled as a product of a position-dependent bias term ($P_{bias}$) and a target-specific visual signal ($P_{vis}$).
• Estimation: By analyzing the ratio of observed probabilities across permutations, the method isolates the structural bias component conditioned on the ground truth position.

3.2. Attention Prior Estimation

Recognizing that raw attention scores are also confounded by structural artifacts (e.g., "attention sinks"), the method estimates a static attention prior:

• The average attention distribution over the symmetrized calibration set is computed to capture the underlying structural bias in the attention mechanism itself.
• This prior serves as a baseline to calibrate the visual evidence during inference.

3.3. Inference-Time Correction

The debiasing process follows a Bayesian-inspired framework:

1. Adaptive Layer Selection: The model identifies the top-$K$ Transformer layers with the highest visual evidence strength (sum of attention probabilities assigned to candidate images) to extract the most informative signals.
2. Intrinsic Visual Evidence Retrieval: The raw attention scores from selected layers are corrected by removing the static attention prior (in the log-domain), yielding a purified visual posterior distribution ($\pi$) that represents the model's true belief about the ground truth location.
3. Expected Bias Correction: Since the true ground truth is latent during inference, the method constructs a dynamic mixture prior by marginalizing the pre-computed conditional bias profiles over the inferred visual posterior $\pi$.
4. Final Calibration: The estimated dynamic prior is subtracted from the raw output logits to recover the intrinsic visual probability for the final prediction.

4. Experimental Results

The method was evaluated on MS-COCO-based benchmarks across three MLLM backbones (Qwen2.5-VL-3B, LLaVA-OneVision-8B, InternVL3-8B) under Random and Adversarial (hard negative) settings.

• Performance Gains: The proposed method achieves state-of-the-art performance, improving accuracy by over 40% compared to baselines (e.g., Vanilla, PriDe, Self-Consistency).
  • On LLaVA-OneVision (N=4, Random), accuracy increased from 67.52% (Vanilla) to 98.66%.
  • On Adversarial settings, it improved from 49.56% to 71.06%.
• Permutation Invariance: The method drastically reduces the Recall Standard Deviation (RStd) and increases Consistency (Cons.), demonstrating that predictions are driven by visual evidence rather than input order.
• Scalability: The method maintains higher accuracy with a more gradual decline as the candidate pool size ($N$) increases from 2 to 12, whereas baselines suffer rapid degradation.
• Generalization: The approach generalizes effectively across domains (Flickr8k to MS-COCO) and difficulty levels (Random to Adversarial) without task-specific recalibration.
• Efficiency: The method operates within a single forward pass. While it incurs a slight latency overhead due to counterfactual bias estimation and attention extraction, it avoids the high costs of ensemble methods.

5. Significance and Claims

The paper claims the following contributions and significance:

• Identification of Logit-Attention Divergence: It reveals a fundamental limitation in existing logit-level calibration methods, showing that internal attention often remains semantically aligned even when outputs are corrupted by bias.
• Training-Free Framework: It introduces a practical, attention-guided debiasing mechanism that requires no model retraining and minimal calibration data (as few as 5 samples).
• Restoration of Robustness: The method effectively restores permutation invariance in multi-image retrieval, suggesting that MLLMs possess latent visual perception capabilities that are currently masked by structural decoding priors.
• Broader Implications: The findings suggest that reported low accuracy in certain MLLM benchmarks may reflect positional bias rather than genuine capability limitations, as the models internally "see" the correct answers but fail to express them.

The work highlights the untapped potential of internal attention mechanisms for enhancing the robustness of MLLMs in complex, multi-image contexts.