Information Router for Mitigating Modality Dominance in Vision-Language Models

1. Problem Statement: Modality Dominance and Information Disparity

Vision-Language Models (VLMs) have achieved state-of-the-art performance but suffer from modality dominance, where the model relies disproportionately on a single modality (usually text) while ignoring others (e.g., vision).

• Limitations of Prior Work: Existing methods attempt to fix this by steering the model's attention allocation (e.g., forcing the model to look at visual tokens). However, these approaches implicitly assume that all input modalities already contain sufficient information.
• The Core Issue: In real-world scenarios, modalities often differ significantly in information density and signal-to-noise ratios. For example, visual tokens may be sparse or ambiguous due to occlusion or viewpoint limitations, while text tokens are often dense and semantically explicit.
• Consequence: Simply adjusting attention cannot resolve the underlying lack of information. If a visual token is uninformative, forcing the model to attend to it does not help; the model will still default to text priors, leading to hallucinations or failures when visual evidence is degraded.

2. Methodology: MOIR (Multi-modal Information Router)

The authors propose MOIR, an information-level fusion method designed to explicitly reduce information disparity before the tokens are processed by the Large Language Model (LLM) decoder.

Key Components:

1. Token Informativeness Scoring:

   • MOIR analyzes modality-specific token sequences ($F_A, F_B$) generated by encoders.
   • It performs Singular Value Decomposition (SVD) on the token representations to identify the contribution of each channel (dimension) to the principal components of the representation space.
   • An informativeness score ($S_d$) is calculated for each channel. Channels with low scores are identified as "less informative" (noisy, redundant, or weak).

2. Cross-Modal Information Routing:

   • Instead of just re-weighting attention, MOIR selectively routes complementary information from a stronger modality to the weak channels of a weaker modality.
   • For a token in Modality A with low-informative channels, MOIR injects information from the corresponding channels in Modality B.
   • This is governed by learnable routing gates ($\alpha$) that control the injection amount:
     $$\bar{F}_A = \alpha \cdot F_B + (1 - \alpha) \cdot F_A$$
   • This process constructs information-dense token representations before they enter the LLM decoder.

3. Integration:

   • MOIR operates as a plug-in module between the modality-specific encoders and the LLM decoder.
   • It uses Parameter-Efficient Fine-Tuning (LoRA) to adapt the routing mechanism without altering the core model architecture.

3. Key Contributions

1. Novel Architecture: Proposes MOIR, a fusion method that explicitly exchanges information across modalities to balance token informativeness, rather than just manipulating attention weights.
2. Theoretical Re-evaluation: Revisits the definition of modality dominance, arguing that forcing attention shifts without addressing the evidential status (information density) of tokens is unreliable, especially when one modality is noisy.
3. Empirical Validation: Provides extensive experiments showing that modifying information availability leads to more balanced modality usage and improved robustness.

4. Experimental Results

The authors evaluated MOIR on three benchmarks: ScienceQA (Image-Text), VizWiz (Image-Text), and MMBench-Video (Video-Audio-Text), using backbones like LLaVA-1.5 and Qwen2.5-VL.

• Downstream Performance:
  • MOIR maintains or improves accuracy across all benchmarks.
  • Significant gains were observed in VizWiz and MMBench-Video, particularly in tasks requiring open-ended reasoning and temporal visual understanding, where visual evidence is indispensable.
• Modality Balance (MDI & AEI):
  • MDI (Modality Dominance Index) and AEI (Attention Efficiency Index) were significantly reduced, indicating a more balanced reliance on both modalities.
  • Effective Rank: The effective rank of token representations ($\Delta I$ and $\Delta T$) increased, proving that MOIR successfully enhanced the information density of both modalities rather than suppressing one.
• Robustness to Degradation:
  • In a robustness test where images in VizWiz were replaced with Gaussian noise, the baseline model remained insensitive to the change (predicting the same answer ~62% of the time), indicating heavy reliance on text priors.
  • MOIR significantly reduced this unchanged rate, demonstrating that the model became sensitive to the visual input and relied less on spurious language shortcuts.
• Qualitative Improvements:
  • On MMBench-Video, baseline models often hallucinated or mimicked question wording. MOIR produced responses grounded in actual visual events (e.g., correctly identifying specific plugins or soccer match actions).

5. Significance and Conclusion

This paper shifts the paradigm for mitigating modality dominance from attention steering to information enrichment.

• Core Insight: Modality dominance is often a symptom of information disparity (one modality being "empty" or "noisy"), not just an attention bias.
• Impact: By explicitly routing complementary information to fill gaps in sparse tokens, MOIR enables VLMs to perform genuine multi-modal reasoning even when one modality is degraded.
• Future Direction: The work highlights that interventions at the information level (modifying token content) are a necessary and complementary strategy to architectural or training-dynamic changes for building reliable, robust multi-modal systems.