🤖 AI

Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders

This paper challenges the assumption that integrating multiple vision encoders inherently improves Multimodal Large Language Models by demonstrating through systematic masking and new metrics that pervasive encoder redundancy often allows for performance gains and efficiency when specific encoders are removed.

Original authors: Yizhou Wang, Song Mao, Yang Chen, Yufan Shen, Yinqiao Yan, Pinlong Cai, Ding Wang, Guohang Yan, Zhi Yu, Xuming Hu, Botian Shi

Published 2026-02-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhou Wang, Song Mao, Yang Chen, Yufan Shen, Yinqiao Yan, Pinlong Cai, Ding Wang, Guohang Yan, Zhi Yu, Xuming Hu, Botian Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of detectives to solve a complex mystery. You decide that to be absolutely sure, you shouldn't just hire one detective; you should hire five. You figure that if one misses a clue, another will catch it. You hire a specialist in fingerprints, a specialist in footprints, a specialist in voice analysis, a specialist in psychology, and a generalist.

You pay for all five salaries, you give them all a giant office to work in, and you expect the case to be solved faster and better than if you had just one person.

This paper is about a team of researchers who walked into that office, looked at the five detectives, and said: "Wait a minute. We only really need two of you. The other three are just standing around, or worse, they're arguing with each other."

Here is the breakdown of their discovery, using simple analogies:

1. The Big Misconception: "More is Better"

In the world of Artificial Intelligence (specifically Multimodal Large Language Models, or MLLMs), there is a popular belief that if you want a computer to "see" and "understand" images, you should feed it through multiple vision encoders (the "detectives").

The Old Idea: Different encoders are trained on different things. One is good at reading text (OCR), another is good at recognizing objects, and another is good at understanding the "vibe" of a picture. The theory was that combining them would create a super-brain that sees everything perfectly.
The Reality Check: The researchers found that this is often a waste of money and energy. Just like hiring five detectives when one is doing all the work, adding more encoders often adds redundancy. They are all looking at the same clues, or worse, they are confusing the main brain with conflicting information.

2. The Experiment: The "Silent Treatment"

To prove this, the researchers played a game of "Silent Treatment" (which they call Masking).

Imagine the five detectives are sitting at a table. The researchers told one detective to stop talking and just sit there silently (masking the encoder). Then they asked the team to solve a problem.

The Result: In many cases, the team solved the problem just as well, or sometimes even better, with one detective missing!
The Shock: Sometimes, when they silenced a specific detective, the team's performance actually improved. It turned out that detective was just making noise and distracting the others.

3. The New Tools: Measuring "Usefulness"

The researchers invented two new ways to measure how helpful each detective really is:

CUR (Conditional Utilization Rate): Think of this as a "Value Score."
- If a detective has a score of 90%, it means they are the star player. If you remove them, the team fails.
- If a detective has a score of 0%, they are useless. You could fire them, and the team wouldn't notice.
- If a detective has a negative score, they are actually hurting the team. They are the one spreading rumors that confuse everyone.
IG (Information Gap): This measures the "Team Imbalance."
- A small gap means everyone is contributing equally (a balanced team).
- A huge gap means one person is doing 90% of the work while the others are sleeping. The researchers found that in many AI models, the gap is huge. One encoder does almost all the heavy lifting, while the others are just dead weight.

4. The Findings: Specialized vs. General

The study found two distinct types of tasks:

The "Specialist" Tasks (Like Reading Charts or Text):
- Analogy: Imagine a task that requires reading tiny handwriting.
- Finding: Only one specific detective (encoder) is good at this. The others are useless here. If you have a model with 5 encoders, but only 1 can read, the other 4 are just taking up space.
The "General" Tasks (Like "What is in this picture?"):
- Analogy: A task like "Is there a dog in this photo?"
- Finding: Here, all the detectives are basically saying the same thing. They are interchangeable. Having five of them is like having five people shouting "It's a dog!" at the same time. You only need one to get the answer.

5. The Solution: The "Lean Team"

The most exciting part of the paper is the solution. The researchers showed that you can build a smaller, cheaper, and faster AI model by just picking the best 1 or 2 encoders and firing the rest.

Performance: These "Lean Teams" (models with fewer encoders) still got 90% to 95% of the answers correct compared to the giant 5-encoder models.
Cost: Because they are smaller, they train 34% faster and run 20% faster on your phone or computer.
Energy: They use significantly less electricity.

The Takeaway

For a long time, the AI world thought, "If we just add more eyes, the AI will see better."

This paper says: "No. If you have too many eyes, they just get in each other's way. You don't need a crowd; you need the right few."

By identifying which "eyes" are actually useful and which are just redundant, we can build AI that is just as smart, but much cheaper, faster, and more efficient. It's a move from "bigger is better" to "smarter is better."

1. Problem Statement

Recent Multimodal Large Language Models (MLLMs) increasingly adopt architectures integrating multiple distinct vision encoders (e.g., CLIP, SigLIP, SAM, DINO, EVA-02) under the assumption that diverse pre-training objectives yield complementary visual signals, thereby improving performance.

However, the authors challenge the heuristic that "more encoders are better." They identify a critical, underexplored issue: Encoder Redundancy. This phenomenon occurs when multiple encoders provide overlapping, conflicting, or irrelevant visual cues, leading to:

Diminishing Returns: Marginal performance gains despite significant increases in computational cost.
Inefficiency: Wasted training resources and increased inference latency.
Performance Degradation: In some cases, the presence of certain encoders actively harms performance due to conflicting signals that fusion mechanisms cannot resolve.

2. Methodology

The paper proposes a systematic framework to quantify and diagnose encoder redundancy through encoder masking and novel metrics.

A. Experimental Protocol: Systematic Masking

The authors perform ablation studies on representative multi-encoder MLLMs (e.g., Eagle, Cambrian-1, DeepSeek-VL).

Masking Strategy: They systematically mask individual encoders (replacing their output with a zero tensor) and evaluate all possible $2^n$ combinations of encoder subsets.
Benchmarks: Evaluation covers four distinct task categories: General VQA, Knowledge-based, OCR & Chart, and Vision-Centric tasks.

B. Proposed Metrics

To move beyond qualitative observations, the authors introduce two principled, model-agnostic metrics:

Conditional Utilization Rate (CUR):
Measures the marginal contribution of a specific encoder $E_i$ given the presence of all other encoders $E_n$ .
$u(E_i) = \frac{\text{acc}(f_{E_n}) - \text{acc}(f_{E_n \setminus \{E_i\}})}{\text{acc}(f_{E_n})}$
- High Positive CUR: The encoder is indispensable and unique.
- Near-Zero CUR: The encoder is redundant (its information is covered by others).
- Negative CUR: The encoder is detrimental (introducing noise or conflict).
Information Gap (IG):
Quantifies the disparity in utility among encoders within a single model.
$\Delta_{gap}(E_n) = \max_{i} u(E_i) - \min_{j} u(E_j)$
- A large IG indicates severe imbalance (one or two encoders dominate while others are useless).
- A small IG suggests balanced, complementary contributions.

C. Attention Analysis

The authors analyze the attention scores of the LLM's final layer to verify if the model actually attends to the features of all encoders or if it implicitly relies on a subset, confirming the redundancy observed via CUR.

3. Key Results

A. Pervasive Redundancy and "Graceful Degradation"

Performance Resilience: Removing specific encoders often results in graceful degradation rather than catastrophic failure. In some cases, removing encoders improves performance.
- Example: Masking 3 encoders in Eagle-X5 7B reduced performance by less than 4%.
- Example: A 3-encoder subset of Cambrian-1 8B outperformed the full 4-encoder model by 3.5%.
Detrimental Encoders: Some encoders exhibit negative CUR (e.g., SigLIP in Cambrian-1 8B on Vision-Centric tasks had a CUR of -16%), actively harming the model.

B. Task-Specific Specialization vs. General Redundancy

OCR & Chart Tasks: Show high specialization. A single encoder (e.g., EVA-02 or ConvNext) often dominates with a CUR > 70-90%. Other encoders contribute minimally.
General & Knowledge Tasks: Show high redundancy. Encoders are largely interchangeable, with low CUR values and low Information Gaps.

C. Efficiency Breakthroughs

Dual-Encoder Variants: Models with only 1 or 2 carefully selected encoders recover >90% of the full model's performance on non-OCR tasks.
Resource Savings:
- Training: A dual-encoder variant of Eagle-X5 7B reduced training time by 34.6% (saving ~34 GPU hours) while retaining 94% of the performance.
- Inference: Masking 3 redundant encoders reduced end-to-end latency by 19.5% with negligible accuracy loss (<4%).
- Compute: Vision-side FLOPs dropped by ~38% when reducing from 5 to 2 encoders.

D. Impact of Model Scale

Larger LLMs (e.g., 13B vs. 8B) exhibit higher redundancy (larger Information Gaps). While they are more robust to encoder loss, they are architecturally less efficient because they rely heavily on a dominant encoder while ignoring others.

4. Key Contributions

First Systematic Quantification: Provides the first rigorous, quantitative study confirming that encoder redundancy is a pervasive and predictable characteristic of modern multi-encoder MLLMs.
New Diagnostic Metrics: Introduces CUR and IG as standard tools to measure encoder utility and imbalance, enabling data-driven architecture design.
Efficient Design Framework: Demonstrates that dual-encoder architectures can achieve near-state-of-the-art performance with significantly reduced computational costs, challenging the "more is better" paradigm.
Empirical Evidence: Shows that specific encoders are strictly necessary for specific tasks (e.g., OCR), while others are largely interchangeable or harmful for general tasks.

5. Significance and Implications

Paradigm Shift: The paper fundamentally challenges the current trend of stacking multiple encoders. It suggests that future MLLM development should focus on encoder selection and pruning rather than aggregation.
Cost-Performance Trade-off: Offers a practical pathway to deploy high-performance MLLMs in resource-constrained environments by removing redundant components without sacrificing capability.
Architectural Guidance: Provides actionable diagnostics for researchers to identify which encoders are truly necessary for their target domains, preventing the "bloat" of multimodal systems.
Theoretical Insight: The findings align with the Platonic Representation Hypothesis, suggesting that encoders trained on similar web-scale data converge to similar feature spaces, making additional encoders redundant unless they offer highly specialized, non-overlapping capabilities (like OCR).

In conclusion, the paper argues that quality of encoder selection outweighs quantity, and that systematic redundancy analysis is essential for building the next generation of efficient, effective multimodal AI.