💻 computer science

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

CoME-VL is a modular vision-language framework that scales performance by fusing complementary contrastive and self-supervised visual encoders through entropy-guided aggregation and RoPE-enhanced cross-attention, achieving state-of-the-art results on diverse understanding and grounding benchmarks.

Original authors: Ankan Deria, Komal Kumar, Xilin He, Imran Razzak, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ankan Deria, Komal Kumar, Xilin He, Imran Razzak, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world not just by reading a book, but by looking at a photograph and describing what it sees.

For a long time, these robots (called Vision-Language Models) had a blind spot. They were like a student who had read a million books about art but had never actually looked at a painting closely. They could tell you, "This is a picture of a dog," but if you asked, "Where exactly is the dog's nose?" or "How many legs are visible?", they would often guess wrong or hallucinate details.

This paper introduces a new robot brain called CoME-VL. Think of it as a "Super-Team" approach that fixes these blind spots by combining two very different types of "eyes."

The Two "Eyes" (Encoders)

Most previous robots used only one type of eye, which was great at recognizing what things were (semantics) but bad at knowing where they were (spatial grounding).

CoME-VL uses two different cameras simultaneously:

The "Big Picture" Eye (SigLIP): Imagine this eye is like a novelist. It's trained to read captions and understand the story. It's excellent at saying, "That's a golden retriever playing fetch." It understands the meaning of the scene perfectly.
The "Architect's" Eye (DINO): Imagine this eye is like a surveyor or an architect. It doesn't care about the story; it cares about shapes, edges, boundaries, and geometry. It's great at saying, "The dog's nose is exactly at pixel coordinates (38, 52)." It understands the structure of the scene.

The Problem: If you just paste the notes from the novelist and the surveyor together, you get a messy, redundant report. The robot gets confused by too much information.

The Solution: The "Smart Mixer"

CoME-VL doesn't just mash the two eyes together. It uses a clever three-step process to blend them perfectly:

1. The "Entropy Filter" (Choosing the Best Notes)

Not every page of the novelist's book or every sketch of the surveyor is useful.

The Analogy: Imagine the novelist writes 27 chapters. The first few are vague, and the last few are repetitive. The surveyor draws 24 sketches; the early ones are blurry, but the middle ones are perfect.
What CoME-VL does: It analyzes the "confusion level" (entropy) of every single layer. It picks the specific chapters from the novelist that have the most interesting story details and the specific sketches from the surveyor that have the clearest lines. It ignores the boring or repetitive parts.

2. The "Orthogonal Mixer" (Removing Redundancy)

Even after picking the best notes, the novelist and surveyor might say the same thing in slightly different ways.

The Analogy: If the novelist says, "The dog is brown," and the surveyor says, "The dog has brown fur," you don't need to write both.
What CoME-VL does: It uses a mathematical trick called Orthogonal Projection. Think of this as a "de-duplication" filter. It forces the two sets of information to be as different as possible, ensuring the robot gets new information from each eye rather than hearing the same fact twice. This keeps the robot's brain light and efficient.

3. The "GPS Aligner" (RoPE)

The novelist's notes and the surveyor's sketches are on different grids. One might be a 24x24 grid, the other 14x14.

The Analogy: It's like trying to overlay a map of New York on a map of London without aligning the compass.
What CoME-VL does: It uses a special "GPS" system (called RoPE-enhanced Cross-Attention) that aligns the two grids perfectly. It tells the robot: "The 'brown fur' note from the novelist corresponds exactly to the 'brown patch' on the surveyor's sketch at this specific location."

The Result: A Robot That "Sees" and "Points"

When you put this all together, the result is a robot that is significantly better at two things:

Understanding: It knows what is in the picture (thanks to the Novelist).
Grounding: It can point to exactly where it is, count objects accurately, and draw boxes around them (thanks to the Surveyor).

Real-world example:
If you ask a standard robot, "Where is the red flag?", it might say, "There is a red flag in the image."
If you ask CoME-VL, it says, "The red flag is located at coordinates X: 5.5, Y: 41.0," and it gets it right almost every time.

Why This Matters

Previous methods tried to solve this by making the robot's brain bigger and slower, which is expensive. CoME-VL is like upgrading the robot's software rather than its hardware. It gets smarter by knowing how to combine different types of intelligence efficiently, without needing to process a massive amount of extra data.

In short: CoME-VL is the first robot that successfully combines the "soul" of a storyteller with the "precision" of a surveyor, allowing it to not just see the world, but truly understand and navigate it.

1. Problem Statement

Current Vision-Language Models (VLMs) predominantly rely on a single vision encoder (typically CLIP or SigLIP variants) trained via contrastive image-text objectives. While effective for global semantic alignment and retrieval, these single-encoder architectures suffer from specific limitations:

Semantic-Spatial Trade-off: Contrastive encoders excel at semantic understanding but often lack fine-grained spatial localization, precise object boundaries, and detailed geometric cues.
Grounding Deficiencies: Single-encoder models struggle with tasks requiring precise spatial reasoning, such as pointing, counting, and bounding box prediction, often leading to object hallucinations or inaccurate coordinate generation.
Redundancy in Multi-Encoder Fusion: Previous attempts to combine multiple encoders often result in redundant features and a significant increase in the number of visual tokens processed by the Large Language Model (LLM), leading to high computational costs and memory overhead.

The paper asks: How can we effectively scale the fusion of complementary visual representations (contrastive vs. self-supervised) to improve both visual understanding and grounding without overburdening the LLM?

2. Methodology: CoME-VL

The authors propose CoME-VL (Complementary Multi-Encoder Vision-Language), a modular framework that integrates a contrastively trained encoder (SigLIP2) with a self-supervised encoder (DINOv3). The architecture consists of three core components:

A. Entropy-Guided Multi-Layer Aggregation

Instead of using only the final layer of an encoder, CoME-VL aggregates features from multiple layers across both encoders.

Layer Selection via Entropy: The authors analyze token entropy across encoder depths. They find that SigLIP2 maintains high entropy (diverse semantic features) across all layers, while DINOv3 exhibits a sharp drop in entropy in deeper layers (10–23), indicating a shift toward concentrated, strong spatial features.
Strategy: The model aggregates all layers (0–27) from SigLIP2 for semantic richness and selects layers 10–23 from DINOv3 for spatial grounding.
Soft Aggregation: Learnable weights are applied to these selected layers to adaptively combine features.

B. Orthogonality-Regularized Mixing (OL)

To prevent redundancy when fusing features from multiple layers and encoders:

Orthogonal Projections: Each selected layer's output is passed through a lightweight Orthogonal Layer (OL). This projects features into a subspace constrained to be orthogonal ( $Q^T Q = I$ ).
Goal: This ensures that the mixed features occupy complementary subspaces, reducing information overlap and encouraging the model to learn distinct, non-redundant representations from different depths.

C. RoPE-Enhanced Cross-Attention Alignment

To fuse the heterogeneous token grids from SigLIP2 and DINOv3 efficiently without increasing the LLM's context length:

Cross-Attention Mechanism: Instead of concatenating all visual tokens (which increases sequence length), CoME-VL uses a cross-attention interface where SigLIP tokens act as Queries and DINO tokens act as Keys/Values.
RoPE Integration: Rotary Positional Embeddings (RoPE) are injected into the attention computation. This allows the model to account for relative spatial offsets between the two different token grids, ensuring spatially coherent matching.
Efficiency: The output is a compact set of fused visual tokens injected into the LLM, avoiding the computational explosion associated with concatenating multiple encoder outputs.

3. Key Contributions

Complementary Multi-Encoder Framework: Introduction of CoME-VL, which successfully bridges the gap between contrastive (SigLIP) and self-supervised (DINO) representations, leveraging the former for semantics and the latter for spatial grounding.
Entropy-Guided Layer Selection: A principled method for identifying optimal layer ranges in different encoders based on token entropy analysis, moving beyond arbitrary depth selection.
Orthogonality-Regularized Fusion: A novel module to suppress redundancy in multi-layer fusion, ensuring that combined features remain distinct and informative.
Efficient Alignment: A RoPE-enhanced cross-attention mechanism that aligns heterogeneous encoder features efficiently, maintaining the LLM's context length while improving spatial precision.
State-of-the-Art Performance: Demonstration that multi-encoder fusion, when done correctly, significantly outperforms single-encoder baselines on both understanding and grounding tasks.

4. Experimental Results

The model was evaluated on the PixMo benchmark (focusing on counting, pointing, and chart understanding) and RefCOCO (referring expression comprehension).

Visual Understanding (PixMo): CoME-VL achieved an average improvement of +4.9% over the Molmo baseline across understanding tasks (Charts, Diagrams, Tables).
Grounding Tasks:
- Pointing: Achieved 58.56% (at 3px) and 75.94% (at 5px) accuracy, significantly outperforming Molmo (53.79%/68.94%) and other SOTA models like Qwen2-VL.
- Counting: Attained 87.83%, surpassing InternVL2-8B (74.05%).
- RefCOCO: Achieved 92.57% on the validation set, outperforming the previous best feature-merging method (CLIP-to-DINO) by +0.84%.
Efficiency: While CoME-VL has a slight inference overhead (1.52s/sample vs. 1.26s for Molmo), it is substantially more efficient than naive concatenation methods like COMM (2.2s/sample).

5. Significance

Redefining VLM Architectures: The paper challenges the prevailing "single-encoder" paradigm, proving that combining complementary visual signals (semantic vs. spatial) yields superior performance.
Solving the Grounding Bottleneck: By explicitly addressing the lack of spatial precision in contrastive encoders via DINO integration, CoME-VL enables VLMs to perform precise coordinate-level tasks (pointing, bounding boxes) without hallucination.
Scalability: The proposed fusion strategy (orthogonality + cross-attention) offers a scalable path for integrating multiple encoders without the prohibitive computational costs of token concatenation, making it feasible for deployment in resource-constrained environments.
Open Source: The authors have released the code, model weights, and project resources, fostering further research in multi-encoder fusion.

In conclusion, CoME-VL demonstrates that complementary multi-encoder learning, guided by entropy analysis and regularized by orthogonality, is a critical step toward building more robust, grounded, and capable Vision-Language Models.