Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval

1. Problem Statement

The paper addresses two critical limitations in current Large Multimodal Model (LMM)-based Unified Multimodal Retrieval (UMR) systems:

• Semantic Alignment Deviation: Existing methods primarily rely on sample-level contrastive learning (distinguishing positive vs. negative pairs). This often fails to capture subject-level semantics, causing models to misallocate attention to semantically irrelevant regions rather than the specific visual subjects mentioned in the text query.
• Visual Modality Neglect: LMMs tend to over-rely on textual cues, leading to an imbalance where visual information is underutilized. The models cluster heavily in "text-biased" regions of the embedding space, failing to leverage complementary visual knowledge effectively.

2. Methodology: SSA-ME Framework

The authors propose Salient Subject-Aware Multimodal Embedding (SSA-ME), a framework designed to enhance fine-grained representation learning through saliency-aware modeling. The architecture consists of three main stages:

A. Multimodal Feature Extraction

The framework uses a standard LMM pipeline (Vision Encoder + Projector + Language Model) but adopts the Explicit One-token Limitation (EOL) principle. It processes interleaved image-text inputs using specific prompt templates to generate a unified multimodal embedding.

B. Saliency-Guided Attention Alignment (SGA)

This module addresses semantic alignment deviation by forcing the model to focus on relevant visual subjects.

1. Subject Extraction: A prompt-based LMM (e.g., Qwen2.5-VL) analyzes the query and its corresponding positive sample to extract a set of salient nouns ($S$) that appear in both and are visually recognizable.
2. Segmentation: A visual expert model (Segment Anything Model - SAM) generates binary segmentation masks for these extracted subjects in the query image.
3. Saliency Map Generation: Binary masks are smoothed using Gaussian kernels to create continuous saliency maps ($A$). A confidence validation mechanism (using CLIP similarity) filters out low-confidence subjects.
4. Loss Function: A Saliency-Guided Loss ($L_{SG}$) is introduced to minimize the Kullback-Leibler (KL) divergence between the model's internal attention distributions and the generated saliency maps, ensuring the model attends to the correct regions.

C. Saliency-Driven Feature Regeneration (SDR)

This module addresses visual modality neglect by recalibrating visual features.

1. Feature Regeneration: The model extracts the final layer's visual attention patterns, flattens them, and uses them as weights to re-aggregate the visual feature tokens.
2. Fusion: The regenerated, saliency-weighted visual summary ($\hat{F}_v$) is fused (via addition) with the original last-token feature ($F_{last}$) to form the final multimodal embedding. This ensures the final representation heavily incorporates the most salient visual information.

Training Objective: The total loss combines the standard contrastive loss ($L_{Con}$) with the saliency-guided loss:
$$L_{total} = L_{Con} + \alpha \cdot L_{SG}$$

3. Key Contributions

1. Subject-Level Perspective: The paper identifies and systematically analyzes "semantic alignment deviation" and "visual modality neglect" as fundamental flaws in sample-level optimization, proposing a shift toward subject-level reasoning.
2. Novel Components (SGA & SDR):
   • SGA: Aligns model attention with semantically meaningful regions using a distillation approach from LMMs and SAM.
   • SDR: Recalibrates visual features based on saliency maps to prevent the model from ignoring visual inputs.
3. State-of-the-Art Performance: The method achieves new benchmarks on the MMEB (Multimodal Embedding Benchmark), demonstrating that explicit subject modeling significantly improves retrieval accuracy and generalization.

4. Experimental Results

The authors evaluated SSA-ME on the MMEB benchmark, which includes 20 in-distribution (IND) and 16 out-of-distribution (OOD) datasets across four tasks: Classification, VQA, Retrieval, and Visual Grounding.

• Performance Gains:
  • SSA-ME (Qwen2VL-2B): Achieved an overall score of 66.0%, surpassing the baseline VLM2Vec (2B) by +5.9%. It showed massive improvements in Classification (+6.7%) and VQA (+8.8%).
  • SSA-ME (Qwen2VL-7B): Achieved an overall score of 69.0%, outperforming the 7B baseline by +3.2% and other strong baselines like MMRet.
• Generalization: The model showed robust performance on OOD datasets, with gains of +7.2% over the baseline, indicating strong adaptability to unseen data distributions.
• Attention Balance: Qualitative analysis (Figure 3) showed that SSA-ME significantly reduced the distance between visual and textual attention weights (up to 99.5% reduction in retrieval tasks), effectively mitigating visual modality neglect.
• Ablation Studies:
  • Both SGA and SDR components contributed positively, with their combination yielding the best results.
  • Applying attention alignment to the final layer of the transformer was found to be most effective compared to early or middle layers.
  • Semantic relevance filtering was crucial for removing noise and improving grounding accuracy.

5. Significance

This work is significant because it moves beyond the standard "contrastive learning" paradigm in multimodal retrieval. By explicitly modeling salient subjects, SSA-ME solves the "black box" issue of where the model is looking, leading to:

• Interpretability: The model's attention maps align with human-identifiable objects.
• Robustness: Better handling of complex queries requiring fine-grained visual-textual correspondence.
• Efficiency: The ability to regenerate features from a subset of salient tokens maintains high performance without processing the entire token set unnecessarily.

The paper establishes that incorporating subject-level reasoning is a critical step toward achieving truly universal and balanced multimodal understanding.