🤖 AI

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

This paper introduces a comprehensive retraining-based evaluation framework for multi-encoder Vision-Language Models that reveals encoder rankings differ from prior masking methods, proposes a "Capacity-Necessity" decomposition to identify optimal encoder pairs, and links performance to pre-projector effective rank, thereby offering principled guidelines for efficient multi-encoder design.

Original authors: Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a super-smart robot that can see the world and talk about it. To give this robot "eyes," you have five different types of cameras available:

ConvNeXt: A general-purpose, all-rounder camera.
EVA-02: A high-end camera great at recognizing objects.
CLIP: A camera trained to match pictures with words.
Pix2Struct: A camera specialized in reading text and charts.
SAM: A camera good at outlining shapes.

The big question researchers asked is: If we can't afford to use all five cameras at once, which ones should we pick to get the best performance?

The Old Way vs. The New Way

Previously, scientists tried to figure this out by taking a robot that already had all five cameras, turning one off, and seeing how much the robot's performance dropped. They thought, "If turning off Camera A makes the robot stupid, then Camera A is the most important."

The Problem: This is like testing a sports team by removing a player during the game. The remaining players might panic, or the team strategy might fall apart because they weren't trained to play without that specific person.

The New Way (This Paper): The authors decided to build 31 different robots from scratch. Some had just one camera, some had two, some had three, and so on. They trained each robot independently to see how it actually performed. This is like training a new team specifically for the players they have, rather than just removing a player from an existing team.

The Big Discoveries

1. The "Star Player" Surprise

When they tested the robots, they found a shocking difference.

The Old Method said EVA-02 was the best single camera.
The New Method said ConvNeXt was actually the best single camera.

It turns out, the "best" camera depends entirely on how you train the robot. You can't just look at a camera in isolation; you have to see how it behaves when it's part of a team.

2. The "Two-Headed" Strategy

The most surprising finding was about how many cameras you actually need.

The Myth: "More cameras are always better."
The Reality: You only really need two.

They found that a robot with just ConvNeXt and CLIP performed almost as well as a robot with all five cameras combined. Adding a third or fourth camera barely improved the score. The fifth camera only helped with very specific, hard tasks (like spotting tiny details in complex images).

The Best Pairing:
You might think the best pair is the two "strongest" cameras. But the paper found that's wrong.

The Wrong Pair: The two strongest cameras (ConvNeXt + EVA-02).
The Right Pair: The strongest camera (ConvNeXt) + a "chameleon" camera (CLIP).

The Analogy: Think of ConvNeXt as a sturdy anchor that holds the team together. CLIP is the flexible teammate who changes their style to fit the anchor. When you put them together, they grow stronger than they were alone. But if you pair two "anchors" together, they just step on each other's toes.

3. The "Brain Space" Explanation (Why it works)

Why does the ConvNeXt + CLIP pair work so well? The authors looked at the "brain space" (mathematically called effective rank) inside the robot's brain where the cameras talk to the language part.

The Anchor (ConvNeXt): It keeps its own unique "voice" even when working with others. It doesn't get squished.
The Chameleon (CLIP): When it works with ConvNeXt, its "voice" actually gets bigger and more complex. It expands its capabilities.

The best teams are made of one member who stays steady and one member who grows when they work together.

The Takeaway

If you are designing a multi-camera AI system:

Don't just pick the "strongest" individual cameras.
Don't assume adding more cameras always helps (you hit a point of diminishing returns very quickly).
Look for a Stable Anchor (like ConvNeXt) and a Flexible Partner (like CLIP) that grows when paired with the anchor.

This approach saves money and computing power because you can build a robot that is 97% as smart as the "super robot" but uses only two cameras instead of five.

Technical Summary: Beyond Encoder Accumulation

Problem Statement

As Large Vision-Language Models (LVLMs) scale to incorporate multiple heterogeneous visual streams, understanding how diverse encoders interact under joint training is critical for principled design. However, current LVLMs lack robust tools to evaluate these interactions. Existing methods, such as Inference-time Masking (IM) on fixed checkpoints, suffer from two primary limitations:

Training-Training Mismatch: Evaluating encoder roles by masking branches at inference time departs from the actual training pipeline, leading to unreliable conclusions about encoder contributions.
Incomplete Enumeration: No prior method has evaluated the full set of possible encoder subsets, making it difficult to identify parameter-efficient configurations before training.

Consequently, designers struggle to determine which encoders to include, often relying on heuristics that may not reflect the true optimization landscape of joint training.

Methodology

The authors propose a Training-time Removal (TR) protocol to re-examine encoder roles. Using the Eagle-X5 architecture (Vicuna-7B decoder with five vision encoders: ConvNeXt, EVA-02, CLIP, Pix2Struct, and SAM) as a testbed, the study:

Retrains all 31 non-empty subsets of the five encoders from scratch under a unified pipeline.
Evaluates these 31 models on the 16-benchmark Cambrian-1 suite (covering General, Knowledge, OCR&Chart, and Vision-Centric tasks).
Contrasts these results with Inference-time Masking (IM) results from prior work (Wang et al.) to highlight discrepancies between inference-time counterfactuals and training-time realities.
Decomposes encoder contributions into two axes: Capacity and Necessity.
Analyzes the pre-projector effective rank of each encoder to understand representation-level signals independent of parameter count.

Key Contributions

1. Attribution Protocol: Training-time Removal (TR) vs. Inference-time Masking (IM)

The study establishes that IM and TR yield systematically different rankings.

Ranking Discrepancy: Under IM, EVA-02 ranks first; under TR (ground truth), ConvNeXt ranks first.
Stability: TR exhibits lower cross-context variance than IM for most encoders, suggesting that retraining subsets provides a more stable measurement of individual contributions.
Design Impact: The choice of protocol changes practical recommendations. For instance, the IM-based heuristic suggests a top pair of ConvNeXt+EVA-02, whereas the TR-based optimal pair is CLIP+ConvNeXt.

2. Capacity–Necessity Decomposition

The authors introduce a two-axis framework to disentangle encoder roles:

Capacity: The score an encoder achieves on its own (singleton performance) relative to the full pool.
Necessity: The performance drop when an encoder is removed from the full pool (leave-one-out drop).
Findings: These axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal. Instead, the optimal strategy is pairing a Universal Core (high Capacity, high Necessity) with an Adaptive Complement (moderate Capacity, low Necessity, but rank-expanding under joint training).
Taxonomy: This decomposition yields a five-class taxonomy:
- Universal Core: ConvNeXt (strong across all families).
- Adaptive Complement: CLIP (rank expands under joint training).
- Vision-Centric Specialist: EVA-02.
- Niche Specialist: Pix2Struct (OCR&Chart).
- Replaceable: SAM.

3. Representation-Level Mechanism: Effective Rank

At a fixed parameter count, residual score variation is explained by the pre-projector effective rank.

Three Regimes:
1. Singleton: Score is predicted by the encoder's own effective rank.
2. Pairs: Score is predicted by the change in the complement's rank under joint training ( $\Delta$ rank). The best pair (CLIP+ConvNeXt) combines an anchor whose rank survives intact with a complement whose rank expands.
3. Full Pool: Score saturates when the mean per-encoder rank stops growing, indicating a shared projector budget constraint.
Implication: Higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder–projector interface.

Results

Diminishing Returns: Adding encoders yields strongly diminishing returns. A well-chosen two-encoder model (CLIP+ConvNeXt) recovers 97% of the full five-encoder model's score. The third and fourth encoders contribute negligible gains, with only the fifth recovering a small margin for Vision-Centric tasks.
Optimal Pairing: The best pair is CLIP + ConvNeXt, not the pair of the two highest-Capacity encoders (ConvNeXt + EVA-02). This pair closes 97% of the gap between a single CLIP encoder and the full pool.
Task-Specific Saturation:
- OCR&Chart and General tasks saturate with two encoders.
- Vision-Centric tasks continue to benefit up to the full five-encoder pool.
- Knowledge tasks are dominated by the LLM prior, rendering visual encoder contributions minimal.
Rank vs. Parameters: Parameter count is a coarse predictor. Encoders with similar parameter counts (e.g., CLIP, EVA-02, SAM) show wide variations in effective rank and singleton scores. Effective rank is a more precise descriptor of representation quality.

Significance and Claims

The paper claims to expose a methodological gap in multi-encoder LVLM design, specifically the reliance on fixed-checkpoint approximations (IM) rather than training-time counterfactuals.

Practical Guidance: The study offers concrete primitives for compact-pool design: select a high-Capacity anchor and an Adaptive Complement whose rank expands under joint training, rather than simply selecting the strongest individual encoders.
Distillation Implications: The findings suggest that distilling a "Replaceable" encoder is low-risk, but distilling an "Adaptive Complement" is challenging because its contribution relies on dynamic encoder–projector interactions (rank expansion) that a frozen student cannot reproduce.
Future Relevance: As foundation models for world modeling and physical AI fuse more heterogeneous streams, the authors argue that full enumeration retraining will become infeasible. The proposed Capacity–Necessity decomposition and rank analysis provide a framework for estimating these properties with minimal retraining, potentially via lightweight surrogates based on pre-projector geometry.

The authors remain modest, noting that the "Anchor with Rank-Expanding Complement" heuristic is descriptive for the Eagle-X5 pool and requires further validation as a generic rule for unseen pools.