🤖 machine learning

Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

This paper challenges conventional model selection criteria for Vision-Language Models by demonstrating that the structural similarity between vision and language modalities, measured via Gromov-Wasserstein distance, is a superior predictor of alignment performance compared to encoder size or zero-shot accuracy.

Original authors: Muyang Li, Yucheng Liu, Jianbo Ma, Elliot Osborne, Bo Han, Tongliang Liu

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muyang Li, Yucheng Liu, Jianbo Ma, Elliot Osborne, Bo Han, Tongliang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate "multilingual" translator. You have a brilliant text expert (a Large Language Model, or LLM) who knows everything about words, and you want to hire a visual expert (a Vision Encoder) to teach them how to see.

The big question is: Which visual expert should you hire?

For a long time, people thought the answer was simple: "Hire the biggest expert" or "Hire the one with the highest test scores." But this paper says, "Actually, that's a bad rule of thumb."

Here is the story of what the researchers found, explained simply.

1. The "Wrong" Way to Choose

The researchers gathered 18 different top-tier visual experts. They tried pairing each one with the same text expert to see which combination worked best.

They tested the old rules:

Rule A: Pick the biggest model (most parameters).
Rule B: Pick the one with the highest "zero-shot" accuracy (the one that gets the most questions right without any extra training).

The Result: These rules failed miserably. Sometimes the biggest model performed poorly. Sometimes the "smartest" model on its own was terrible when paired with the text expert. It turns out, being good at your own job doesn't mean you'll be good at working with this specific partner.

2. The Real Problem: Speaking Different "Languages"

The researchers realized the issue wasn't about how smart the visual expert was, but about compatibility.

Imagine the text expert speaks a language where sentences are structured like poems.

Visual Expert A speaks a language structured like math equations.
Visual Expert B speaks a language structured like poems.

Even if Visual Expert A is a genius mathematician, it will be a nightmare to translate their thoughts into the text expert's poetic language. The "translation layer" (the projector) has to work incredibly hard to bridge that gap, and the final result is clunky.

Visual Expert B, however, already thinks in poems. The translation is easy, and the final team works beautifully.

The paper calls this Structural Similarity. It's not about what they know, but how they organize their thoughts.

3. The Solution: The "Gromov-Wasserstein" (GW) Distance

To measure this compatibility without actually hiring and training every single candidate (which would take months and cost a fortune), the researchers invented a new test called the Gromov-Wasserstein (GW) distance.

Here is a metaphor for how it works:

The Old Way (Naive Distance): Imagine you have two maps. One is of New York, and one is of Tokyo. If you try to measure the distance between "Central Park" and "Times Square" on the New York map, and compare it to "Shibuya" and "Tokyo Tower" on the Tokyo map, the numbers won't match because the cities are different sizes and shapes. This is why old methods fail.
The GW Way: Instead of looking at the absolute locations, GW looks at the relationships. It asks: "Is the relationship between Park and Times Square (e.g., 'they are 10 minutes apart') the same as the relationship between Shibuya and Tokyo Tower?"
If the internal geometry (the way points relate to each other) is similar between the two maps, the GW distance is low. This means the two experts "think" in similar shapes, making them easy to pair up.

4. The Proof

The researchers ran a massive experiment:

They calculated this GW distance for all 18 visual experts.
They then actually trained the full systems (the "hard way") to see who really won.

The Findings:

The GW distance was the only metric that correctly predicted the winners.
The correlation was strong: The lower the GW distance, the better the final AI performed.
The old rules (size and accuracy) had almost no connection to the final success.

5. Why This Matters

This paper gives us a "training-free" shortcut.

Before: You had to train 18 different AI models for weeks to find the best one.
Now: You can run this GW calculation in about one minute (on a single powerful computer) and know exactly which visual expert will work best with your text model.

Summary Analogy

Think of building a VLM like forming a dance duo.

Old Wisdom: "Pick the dancer with the most awards and the biggest muscles."
New Wisdom: "Pick the dancer whose rhythm and style naturally match your partner."

The paper proves that rhythm (structural similarity) matters much more than muscle (model size) or awards (accuracy) when it comes to creating a successful AI team. They provided a new tool (GW distance) to measure that rhythm instantly, saving time and money while building better AI.

Technical Summary: Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

1. Problem Statement

Vision-Language Models (VLMs) typically follow a "pre-training then fine-tuning" paradigm, where a projector aligns the representation spaces of a vision encoder and a Large Language Model (LLM). Despite the widespread adoption of this paradigm, there is a lack of principled understanding regarding which pre-trained vision encoders are most suitable for a specific LLM.

Current practices often rely on heuristics such as selecting the largest model or the one with the highest zero-shot accuracy. However, the authors demonstrate through comprehensive experiments that these metrics:

Consistently fail to identify the optimal vision encoder for VLM alignment.
Show only weak to moderate correlation with the final VLM performance after full fine-tuning.
Do not account for the "compatibility" or structural similarity between the visual and textual representation spaces, which is hypothesized to be the critical factor in alignment difficulty.

2. Methodology

The paper proposes using the Gromov-Wasserstein (GW) distance as a training-free, inference-only metric to measure the structural similarity between the distribution of visual representations and textual representations.

Core Concept: Gromov-Wasserstein Distance

Unlike standard distance metrics that require a common ambient space, GW distance is designed to compare two metric-measure spaces (mm-spaces) that may reside in different domains with different dimensions and measures.

Intuition: Instead of comparing absolute positions of points (which is ill-posed across modalities), GW distance compares the internal relationships (pairwise distances) within each space. It seeks an optimal transport plan (coupling) $\pi$ that minimizes the distortion between the pairwise distances of the visual space ( $d_X$ ) and the textual space ( $d_Y$ ).
Formulation: The GW distance is defined as the infimum of the expected loss over all possible couplings $\pi$ :
$GW := \inf_{\pi \in \Pi} \sum_{x,x',y,y'} L(d_X(x, x'), d_Y(y, y')) \pi_{x,y}\pi_{x',y'}$
where $L$ is a penalty function (using $\ell_1$ distance in this work).

Implementation Details

Scale Matching: To handle inherent scale differences between modalities, the authors apply median-ratio matching. They normalize the pairwise distance matrices of the visual modality so that their median off-diagonal values match those of the text modality.
Optimization: The optimal transport plan $\pi^*$ is solved using a conditional gradient scheme (following [40]), treating the GW objective as a quadratic functional over the transport polytope.
Model Selection Algorithm:
- Extract visual features (CLS tokens) and textual features (hidden states) for a sample of image-text pairs.
- Compute pairwise distance matrices for both modalities.
- Normalize distances and compute the GW distance.
- Select the vision encoder with the minimum GW distance to the target LLM, predicting it will yield the best VLM performance.

3. Theoretical Analysis

The paper provides a theoretical justification linking GW distance to the learnability of cross-modal mappings.

Lipschitz Bound: The authors prove that the Lipschitz constant ( $L_{g^*}$ ) of the Bayes-optimal cross-modal projector $g^*$ is bounded by the $\infty$ -norm GW distance ( $GW_\infty$ ) between the modalities.
Implication: A smaller GW distance implies a smaller upper bound on the Lipschitz constant. Since generalization bounds for neural networks often depend on the Lipschitz constant (or spectral norm), a smaller GW distance suggests that the optimal mapping is inherently easier to learn with a hypothesis of lower functional complexity.

4. Experimental Results

The authors conducted extensive experiments involving 60+ full VLM training runs using 18 diverse pre-trained vision encoders and two base LLMs (Qwen-2.5-7B-Instruct and LLaMa-3.1-8B-Instruct).

Key Findings

Superior Selection Accuracy: The GW-based selection strategy consistently identified the optimal vision encoder (the one yielding the highest average score across 9 benchmarks) in both "Small" (<500M params) and "Large" groups.
Correlation with Performance:
- GW distance achieved a Pearson correlation of 0.6568 and Spearman correlation of 0.5341 with final VLM performance.
- The linear fit between GW distance and performance explained 43% of the variance ( $R^2 = 0.4314$ ).
- In contrast, traditional baselines like Zero-Shot Accuracy ( $R^2 \approx 0.21$ ), Representational Similarity Analysis (RSA), and Canonical Correlation Analysis (CCA) showed weak to moderate correlations. Notably, MutualNN showed a high Pearson correlation but a negative Spearman correlation, indicating a spurious relationship.
Robustness: The effectiveness of GW distance was verified across different base LLMs and different pre-training datasets (LAION/Conceptual Captions vs. CC3M), suggesting the metric captures a generalizable pattern of modality compatibility.
Efficiency: The GW estimation is a pure inference task. Computing the metric for 1,000 image-text pairs takes approximately 1 minute on a single GPU, compared to ~8.5 hours (68 GPU hours) required for a full VLM training run.

5. Significance and Claims

The paper claims to provide the first systematic investigation into the factors governing vision encoder selection for VLMs, moving beyond naive heuristics.

Novelty: It introduces the concept of distributional compatibility between modalities as the key determinant of VLM success, formalized via GW distance.
Practical Impact: It offers a computationally efficient, training-free method to predict VLM performance before committing to expensive full-parameter fine-tuning.
Theoretical Contribution: It establishes a formal link between the geometric structure of representation spaces (measured by GW distance) and the learnability of the alignment projector.

The authors conclude that while current heuristics (size, zero-shot accuracy) are insufficient, measuring the structural similarity of representation spaces via GW distance provides a robust, theoretically grounded, and empirically validated proxy for model selection in Vision-Language Models.