Technical Summary: Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Models

Problem Statement

While Large Vision-Language Models (LVLMs) demonstrate strong performance on single-image tasks, their capabilities degrade significantly when processing multiple images simultaneously. Although prior work has noted this performance drop, the underlying mechanisms have remained poorly understood.

The authors empirically identify a specific phenomenon termed cross-image information leakage. In multi-image settings, the model's visual representations become entangled, causing it to mix visual cues from different images rather than interpreting them independently. As illustrated in the paper's motivation, when presented with two images (e.g., one with yellow tulips and another with red roses) and a query about the first image, the model often fails to isolate the target. Instead of selecting the correct option ("yellow tulips"), it incorrectly chooses a merged option ("yellow tulips and red roses").

This leakage is attributed to the standard processing of multiple images as a sequence of concatenated visual tokens. The inter-image causal attention mechanism causes tokens from different images to interact strongly, mixing latent representations and leading to hallucinations where content from non-target images contaminates the response.

The paper also notes that a naive solution—processing each image independently and aggregating results—fails because it sacrifices the inter-image relationships (e.g., comparisons, cross-references) that are essential for complex multi-image reasoning tasks.

Methodology: FOCUS

To address this challenge without requiring additional training or architectural modifications, the authors propose FOCUS, a training-free, architecture-agnostic inference method. The method operates on the observation that LVLMs reason well over single images and aims to guide the model to focus on one image at a time while preserving the positional context of the multi-image input.

FOCUS consists of three sequential steps:

1. Visual Masking:
   For a set of $N$ input images, the method generates $N$ partially masked inputs and one fully masked reference.

   • For each target image $I_i$, all other images $I_j$ ($j \neq i$) are corrupted with random noise (scale $\lambda$), while $I_i$ remains clean.
   • A fully noise-masked input ($I_{noise}$) is also created where all images are corrupted.
   • This forces the model to attend primarily to the single clean image in each forward pass while maintaining the original image order.

2. Image-wise Focused Inference:
   The model performs inference on each of the $N$ partially masked inputs to obtain a logit distribution $\ell_i$ for each target image $I_i$. Additionally, a noise reference logit distribution $\ell_{noise}$ is computed using the fully masked input. This step captures the model's reasoning when focused on specific images within the joint context.

3. Contrastive Aggregation:
   The final output is derived by aggregating the focused logits while suppressing residual noise artifacts. The formula used is:
   $$\ell_{final} = \sum_{i=1}^{N} (\ell_i - \alpha \cdot \ell_{noise})$$
   Here, $\alpha$ is a hyperparameter tuned via validation. Subtracting the noise reference ($\ell_{noise}$) helps isolate and remove spurious visual signals that persist even after masking, ensuring the final prediction relies on the clean image content rather than the noise-induced artifacts.

Key Contributions

• Identification of Cross-Image Information Leakage: The paper provides empirical evidence that multi-image performance degradation is closely linked to the entanglement of visual representations across images, leading to merged, incorrect responses.
• FOCUS Method: A novel, training-free inference strategy that mitigates leakage by using noise masking and contrastive logit aggregation. It preserves inter-image relationships (unlike independent inference) while preventing content mixing.
• Generalization: The method is shown to be effective across diverse LVLM families (InternVL3, Qwen2.5-VL, LLaVA-OneVision, Qwen3-VL) and scales (from <7B to >30B parameters).
• Video Understanding Extension: The authors demonstrate that FOCUS generalizes beyond static multi-image inputs to video understanding tasks (Vinoground), suggesting the mechanism addresses a fundamental issue in temporal and sequential visual reasoning.

Experimental Results

The authors evaluated FOCUS on five multi-image benchmarks: Winoground, VisMin (Easy and Hard), Mantis-Eval, MuirBench, and MIRB.

• Performance Gains: FOCUS consistently improved Image Score (I) and Group Score (G), which directly reflect multi-image reasoning capabilities, while leaving Text Score (T) largely unchanged.
  • On Winoground, gains reached up to +18.8 Image score and +16.8 Group score.
  • On VisMin-Easy, improvements were substantial, e.g., InternVL3-2B saw an Image Score increase from 29.96 to 74.33.
  • On Mantis-Eval, accuracy improved by up to +5.5 percentage points.
  • On MuirBench and MIRB, consistent, albeit smaller, gains were observed across models.
• Comparison with Alternatives: FOCUS outperformed alternative approaches, including:
  • Independent Processing: Processing images separately and concatenating text, which failed to capture inter-image relationships.
  • Conventional Contrastive Decoding (VCD): Standard contrastive methods designed for single images degraded performance in multi-image settings.
  • SoFA: A method targeting position bias, which did not consistently improve performance, indicating that position bias and information leakage are distinct problems.
• Ablation Studies: Removing noise injection or the noise reference subtraction step significantly degraded performance, confirming that both components are essential.
• Latency: The method introduces a moderate latency overhead (requiring $N+1$ inference passes for $N$ images), which the authors note is a justifiable trade-off for the accuracy gains.

Significance and Claims

The paper claims that FOCUS offers a practical and generalizable inference-time solution for enhancing multi-image reasoning. By empirically characterizing cross-image information leakage and providing a method to mitigate it without retraining, the work addresses a critical limitation in current LVLMs.

The authors emphasize that their approach:

1. Does not require additional training data or architectural changes.
2. Works across a wide range of model sizes and families.
3. Extends to video understanding, suggesting the problem of visual entanglement is not limited to static images.
4. Provides a general solution for improving the reliability of multi-image reasoning, which is essential for real-world applications where users provide multiple images simultaneously.

The paper concludes that while FOCUS significantly improves performance, it is not a panacea for all long-context or retrieval-heavy tasks, where the bottleneck may lie in locating relevant information rather than disambiguating visual content. However, for tasks requiring the integration and comparison of multiple visual inputs, FOCUS offers a robust enhancement.