Technical Summary: Evaluating and Steering Modality Preferences in Multi-modal LLMs

Problem Statement

Multi-modal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing heterogeneous data (text, images, video). However, a critical gap remains in understanding whether these models exhibit modality preference—a systematic tendency to favor one modality over another when processing conflicting multi-modal contexts. Existing evaluation methods often isolate modalities (e.g., testing text-only vs. image-only inputs) or rely on Shapley values, which fail to reflect realistic scenarios where modalities co-occur and potentially conflict. Furthermore, it is unclear if these intrinsic preferences are controllable or if they correlate with downstream task performance.

Methodology

1. The MC2 Benchmark

To rigorously evaluate modality preference, the authors introduce MC2 (Modality Context Conflict), a benchmark designed to create controlled evidence-conflict scenarios.

• Data Construction: The dataset comprises 2,000 samples across eight tasks (e.g., counting, attribute recognition, sentiment analysis). It is derived from TDIUC and MS-COCO.
• Conflict Mechanism: For each sample, a visual context ($c_v$) and a textual context ($c_t$) are constructed to provide contradictory evidence regarding a specific question ($q$).
• Validation: The construction pipeline ensures that models can accurately comprehend both modalities in isolation (achieving >95% accuracy on single-modality tests) before being tested on the conflict. This eliminates confounding factors like poor visual recognition or internal knowledge bias.
• Annotation: A semi-automated framework using LLMs (DeepSeekV3, GPT-4o-mini) generates distractor answers and conflicting text, followed by human verification to ensure semantic inconsistency between modalities.

2. Evaluation Metric: Vision Ratio

The authors define the Vision Ratio to quantify preference:
$$\text{Vision Ratio} = \frac{S_{\text{vision}}}{S_{\text{vision}} + S_{\text{text}}}$$
Where $S$ represents the proportion of samples where the model's response aligns with the visual or textual context, respectively. A ratio > 0.5 indicates a preference for vision; < 0.5 indicates a preference for text.

3. Analysis of Drivers

The study investigates factors driving these preferences:

• Attention Patterns: The authors correlate the Vision Ratio with the Attention Ratio (visual attention vs. total attention), finding that uneven attention distribution drives preference.
• Scale and Training: Experiments show that larger models generally exhibit stronger visual preferences. Additionally, fine-tuning with multi-modal conflict data shifts preferences, suggesting that exposure to conflicting contexts during training influences these biases.

4. Representation Engineering for Steering

Based on the finding that modality preference is encoded in latent representations, the authors propose a training-free steering method:

• Probing: The model is queried with instructions to prioritize either vision or text. The difference in hidden states ($x^t_{i,\ell} - x^v_{i,\ell}$) at the last token across layers is computed to identify the modality preference direction vector ($u^\ell_t$).
• Layer Selection: Analysis reveals that layers 20–23 (in models like Qwen2.5VL) exhibit the most stable and significant preference signals.
• Steering: During inference, a scaled steering vector ($s^\ell = w \cdot u^\ell$) is added to the hidden states at the selected layer. The weight $w$ is calculated to align the mean of the probed direction with the original distribution, ensuring the intervention is effective without disrupting language capabilities.

Key Results

Modality Preferences in MLLMs

• Prevalence: All 20 tested MLLMs exhibit clear modality preferences.
• Text Bias: Most models (e.g., LLaVA1.5-7B) show a strong text preference (Vision Ratio ~13.4%), aligning with the "language prior" hypothesis.
• Visual Preference: Models like Qwen2.5VL and InternVL3 show a stronger visual preference, which increases with model scale.
• Correlation: There is a strong positive correlation ($\rho = 0.964$) between the Vision Ratio and visual understanding performance on downstream benchmarks (e.g., MathVista, TallyQA), suggesting preference is a reliable indicator of capability.

Effectiveness of Steering

The proposed representation engineering method successfully steers models toward a desired modality without fine-tuning:

• MC2 Performance: The method outperforms baselines (Zero-shot, Instruction Design, Chain-of-Thought, Few-shot) in adjusting the Vision Ratio. For instance, steering Qwen2.5VL-7B toward text increased its text-alignment score from 35.4% to 63.6%.
• Downstream Tasks:
  • Reasoning: Improved accuracy on MathVista (62.3% vs. 59.0% baseline) and TallyQA (76.6% vs. 74.4%).
  • Hallucination Mitigation: On the PhD benchmark (specifically the subset with misleading text), steering toward vision significantly reduced hallucinations caused by conflicting text.
  • Translation: On AmbigCaps (multi-modal machine translation), steering toward text improved BLEU scores by reducing reliance on spurious visual cues.

Internal Mechanism

Attention analysis confirms that the steering method successfully alters the model's attention distribution. When steered toward text, the attention weight on text tokens increases significantly (e.g., from ~55% to ~83%) while visual attention decreases, validating that the intervention operates at the representation level.

Significance and Claims

The paper claims three primary contributions:

1. Evaluation Framework: The introduction of MC2 provides the first controlled, large-scale benchmark to systematically evaluate modality preference in MLLMs, decoupling preference from understanding capabilities.
2. Insight into Latent Space: The study reveals that modality preferences are steerable and identifiable within the latent representations of MLLMs, offering a new perspective on multi-modal reasoning mechanisms.
3. Training-Free Control: The proposed probing and steering method enables explicit, controllable adjustment of modality preference without additional data or fine-tuning. This enhances performance on downstream tasks where specific modality reliance is critical (e.g., avoiding text-induced hallucinations in visual reasoning).

The authors conclude that understanding and controlling modality preference is essential for developing more reliable and robust multi-modal systems, particularly in scenarios where conflicting information is present.