💬 NLP

Language-Conditioned Visual Grounding with CLIP Multilingual

This paper isolates the visual encoder as a consistent factor across thirteen languages to demonstrate that multilingual visual grounding disparities stem primarily from text-branch limitations and spatial misalignment rather than signal collapse, revealing that scaling the visual model improves some low-resource languages while exacerbating others and confirming the method's energy-efficient viability.

Original authors: J. de Curtò, Mauro Liz, I. de ZarzÃ

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: J. de Curtò, Mauro Liz, I. de ZarzÃ

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that can look at a picture and find specific things in it, like a "car" or a "pedestrian." You can tell this robot what to look for in many different languages. But, the researchers in this paper discovered that when you speak to the robot in a less common language (like Basque or Luxembourgish), it often points to the wrong spot in the picture, even though it seems to be "looking" just as hard as it does when you speak English.

Here is a simple breakdown of what they did and what they found, using some everyday analogies.

The Experiment: A Controlled Test

The researchers wanted to figure out why the robot makes mistakes in some languages. Is it because the robot's "eyes" (the visual part) don't work well with certain languages? Or is it because the robot's "brain" (the text part that understands words) is weaker in those languages?

To test this, they built a special setup:

The Eyes: They used the exact same camera and vision system for every single language.
The Brain: They swapped out only the part that understands the words, changing it for 13 different languages (from common ones like Spanish and French to rare ones like Basque and Luxembourgish).

This is like giving the same pair of glasses to 13 different people and asking them to describe the same photo. If the descriptions are wrong, we know the problem isn't the glasses; it's the person's ability to describe what they see.

The Big Findings

1. The Problem is in the "Translator," Not the "Camera"
They found that even with the exact same camera, the robot was much worse at finding objects when using low-resource languages.

The Analogy: Imagine a tour guide who knows a city perfectly. If you ask the guide in English, they point to the Eiffel Tower correctly. If you ask the same guide in a language they barely know, they might point to a random tree instead. The guide's eyes haven't changed; their knowledge of the language is the weak link.
The Result: The "penalty" (the drop in performance) was entirely because of the text-processing part of the AI, not the visual part.

2. Bigger Brains Don't Always Fix the Problem
Usually, when AI models get bigger and more powerful, they get better at everything. The researchers tested a small model and a model that was 7 times bigger.

The Surprise: For some languages (like Arabic and Chinese), getting a bigger brain helped. But for others (Basque and Luxembourgish), the bigger brain actually made things worse.
The Analogy: Think of it like a library.
- If a language is "tokenizer-disadvantaged" (like Arabic), it just needs a bigger library to hold more words. A bigger model helps.
- If a language is "corpus-coverage" disadvantaged (like Basque), it means the library has almost no books in that language to begin with. Giving the librarian a bigger library doesn't help if the books aren't there. In fact, the bigger librarian might just get more confident in pointing at the wrong things because they have more "confidence" but no better data.

3. The Robot is "Confidently Wrong"
This is the most important discovery. When the robot fails in these languages, it doesn't just get "quiet" or unsure. It gets loud and confident, but it points to the wrong place.

The Analogy: Imagine a GPS navigation system.
- Signal Collapse (What they didn't find): The GPS says, "I don't know where you are," and shows a blank screen.
- Spatial Misalignment (What they did find): The GPS says, "You are here!" with 100% confidence, but it points to a house three streets away.
The Consequence: Because the robot is so confident, you can't just tell the system, "If you aren't sure, don't show the result." The system is sure, but it's sure about the wrong thing.

Energy Efficiency: A Cheap Solution

Finally, the researchers measured how much electricity this process used.

The Result: This method is incredibly energy-efficient. It uses about 20 to 50 times less energy than the fancy, talking AI models (like the ones that write essays or chat with you).
The Takeaway: If you need a system that can quickly point out objects in a picture in many languages without burning a lot of power, this "dense grounding" method is a very practical, energy-saving choice.

Summary

The paper proves that for these AI models, the problem with rare languages isn't that the AI can't "see" the image; it's that the AI doesn't "understand" the words well enough to match them to the right spot. Making the AI bigger doesn't fix this if the training data (the books in the library) is missing. And because the AI is often confidently wrong, we can't just rely on its confidence level to know if it's telling the truth.

Technical Summary: Language-Conditioned Visual Grounding with CLIP

Problem Statement

Multilingual Vision-Language Models (VLMs) exhibit systematic performance disparities across languages, particularly penalizing low-resource languages and non-Latin scripts. However, the mechanism driving this "multilingual penalty" remains ambiguous. It is unclear whether the degradation stems from the visual encoder's reaction to non-English tokens, the text branch's inability to form robust cross-lingual representations, or interactions between these components. Furthermore, existing evaluation methods often conflate genuine perceptual disparities with measurement bias, as they typically score outputs against English-centric references or vocabularies. This paper addresses the need to isolate the source of the penalty and characterize its failure mode (e.g., signal collapse vs. spatial misalignment) without relying on English-anchored metrics.

Methodology

The authors propose a controlled "dense multilingual CLIP probe" designed to isolate the multilingual penalty to the text branch.

Experimental Design: The study holds the visual encoder identical across 13 typologically diverse languages while varying only the XLM-RoBERTa text branch. This ensures that any observed cross-language divergence originates downstream of the visual stage.
Models & Scale: Two CLIP architectures were evaluated, spanning a 7× gap in visual encoder parameters:
- Base: XLM-R base + ViT-B/32 (~87M visual parameters).
- Large: XLM-R large + ViT-H/14 (~632M visual parameters).
Dataset: 210 images sampled from the BDD100K corpus, evaluated against 11 concepts (e.g., car, pedestrian, traffic light) across 13 languages (including Arabic, Basque, Luxembourgish, Chinese, and various European languages). This yields 2,310 paired observations per non-English language.
Dense Feature Extraction: Following the MaskCLIP approach, the authors extract dense per-patch features from the frozen visual encoder to generate spatial similarity maps, rather than relying on single-image embeddings.
Metrics: To avoid English bias, agreement is measured symmetrically between the target language map and the English reference map using:
1. Cluster-mask IoU: Measures spatial overlap of high-similarity regions.
2. Top-percentile IoU: Scale-invariant overlap at high thresholds.
3. Spearman Rank Correlation: Pixel-wise agreement on relative ordering.
4. Peak-Similarity Ratio: Compares maximum activation strength to distinguish between signal collapse (weak activation) and spatial misalignment (strong activation in the wrong region).
Energy Measurement: Inference energy was measured using the AI Energy Score protocol (NVML-based sampling on an NVIDIA H200), reporting Wh per 1,000 queries.

Key Results

1. The Penalty Persists and Is Localized to the Text Branch

When the visual encoder is held constant, a significant performance gap remains between high-resource (HR) and low-resource (LR) languages.

Quantitative Gap: The HR–LR cluster-mask IoU gap is +0.114 at the base scale and widens to +0.143 at the large scale.
Statistical Significance: Wilcoxon tests confirm this disparity with $p < 10^{-300}$ at both scales.
Implication: Since the visual encoder is identical, the deficit is strictly attributable to the text branch, ruling out visual encoder interaction as the primary cause.

2. Scaling Effects Are Heterogeneous

Increasing the visual encoder scale by 7× does not uniformly close the gap; it separates failure modes based on corpus coverage versus tokenizer fertility.

Corpus-Coverage Failures (Basque, Luxembourgish): These languages, which are under-represented in the pretraining corpus, worsen with scaling ( $\Delta = -0.056$ and $-0.076$ respectively). The larger model amplifies the lack of signal rather than correcting it.
Tokenizer-Fertility Failures (Arabic, Chinese): These languages, which have substantial corpus presence but suffer from tokenization inefficiencies, improve with scaling ( $\Delta = +0.033$ and $+0.039$ ).
Conclusion: Scaling alone cannot rescue languages absent from the pretraining distribution; it only benefits those where the signal exists but is hindered by tokenization or capacity limits.

3. Failure Mode: Spatial Misalignment, Not Signal Collapse

The study identifies that the dominant failure mode is spatial misalignment, not a loss of signal strength.

Peak Similarity: The mean peak-similarity ratio relative to English remains high (0.94 at the large scale), indicating that the model activates strongly.
IoU Drop: Despite high peak activation, the cluster-mask IoU drops sharply.
Implication: The model activates on the wrong region of the image rather than failing to activate. Consequently, confidence-based filtering relying solely on peak similarity is insufficient to detect language-induced grounding errors.

4. Energy Efficiency

The dense-CLIP grounding approach is highly energy-efficient, consuming 3.4–3.9 Wh per 1,000 queries. This is roughly 20–50× lower than the inference budgets reported for autoregressive multilingual VLMs under the same measurement protocol, positioning it as a viable substrate for energy-aware deployment.

Significance and Claims

The paper claims three primary contributions:

Mechanistic Isolation: By holding the visual encoder constant, the study definitively localizes the multilingual penalty in dense grounding to the text branch, distinguishing it from visual encoder limitations.
Failure Mode Characterization: It demonstrates that multilingual grounding errors are primarily spatial misalignments (high signal, wrong location) rather than signal collapses. This challenges the efficacy of standard confidence thresholds in multilingual systems.
Methodological Rigor: The use of symmetric, language-agnostic scoring on dense similarity maps removes the English-anchored measurement bias inherent in previous VLM evaluations, providing a cleaner metric for cross-language agreement.

The authors conclude that while scaling helps tokenization-limited languages, it exacerbates issues for corpus-scarce languages. They suggest that future robust multilingual systems require targeted data curation for under-represented languages and the implementation of cross-language agreement checks (rather than just peak-similarity checks) to ensure spatial grounding reliability.