🤖 AI

Blind to Position, Biased in Language: Probing Mid-Layer Representational Bias in Vision-Language Encoders for Zero-Shot Language-Grounded Spatial Understanding

This paper reveals that Vision-Language Encoders' final-layer embeddings suffer from positional insensitivity and language-dependent geometric shifts, prompting the proposal of a mid-layer representational bias probing method that significantly improves zero-shot referring image segmentation and text-to-image retrieval performance.

Original authors: Na Min An, Inha Kang, Minhyun Lee, Hyunjung Shim

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Na Min An, Inha Kang, Minhyun Lee, Hyunjung Shim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Smart but Clueless" AI

Imagine you have a brilliant AI assistant named VLE (Vision-Language Encoder). This AI has read millions of books and seen millions of pictures. It is incredibly smart at understanding the general idea of things. If you show it a picture of a dog and ask, "Is that a dog?", it says "Yes!" with 100% confidence.

However, this paper reveals that VLE has two major personality flaws when you ask it to point to specific things:

It is "Spatially Blind": It knows what a "red ball" is, but it doesn't care where the red ball is. If you say, "Find the red ball on the left," VLE might point to the red ball on the right because it's so focused on the concept of "red ball" that it ignores the direction.
It is "Language Biased": It speaks English fluently, but if you ask the same question in German, Chinese, or Korean, it gets confused. The meaning of the words gets "scrambled" in its brain, causing it to lose its way.

The researchers discovered that this isn't because the AI is stupid; it's because of where in its brain it's looking.

The Analogy: The "Final Exam" vs. The "Draft Notes"

Think of the AI's brain as a factory with many floors (layers).

The Middle Floors (Mid-Layers): Here, the workers are taking detailed notes. They are looking at the specific parts of the image. They know exactly where the "left" is and they are still processing the sentence structure of different languages carefully.
The Top Floor (Final Layer): This is where the "Final Report" is generated. To make the report clean and easy to read, the workers summarize everything into one big, abstract idea.
- The Problem: In the process of summarizing, they throw away the "left/right" details to save space. They also accidentally translate the German and Chinese notes into a weird, distorted version of English before writing the final report.

The paper's discovery: The AI actually has the correct spatial and multilingual information, but it's hiding it on the middle floors. The "Final Report" (the standard output) is just too summarized to be useful for precise tasks.

The Solution: B2G (Biased to Grounded)

The researchers built a tool called B2G. Instead of asking the AI for its "Final Report," B2G acts like a detective who sneaks into the middle floors of the factory to grab the raw, detailed notes before they get summarized.

Here is how B2G works in two steps:

1. Fixing the "Blindness" (The Spatial Map)

The Problem: When the AI looks at the middle floors, it sometimes gets the direction backward (like looking in a mirror). It thinks "Left" is "Right."
The Fix: B2G uses a simple trick called negation. It flips the AI's internal map upside down.
- Analogy: Imagine the AI is drawing a map where the "Target" is a dark spot in the middle of a white page, but the AI accidentally made the Target a white spot on a black page. B2G just inverts the colors, instantly making the Target pop out clearly.
Result: The AI can now see exactly where the object is, even if the instruction says "the cup on the far left."

2. Fixing the "Bias" (The Multilingual Centroid)

The Problem: If you ask the same question in 10 different languages, the AI's internal representation of that question drifts apart. It's like asking 10 people to describe a "dog," but they all describe it slightly differently, making it hard to find the real dog.
The Fix: B2G takes the question, translates it into 10 languages, and asks the AI to process all 10 versions at the same time on the middle floors. Then, it calculates the average (the centroid) of all those answers.
- Analogy: Imagine 10 people are trying to describe a location. One says "near the tree," another says "by the oak," another says "under the leaves." B2G takes all their descriptions, finds the common ground (the "centroid"), and creates one super-accurate, language-neutral instruction.
Result: The AI stops caring about the specific language and focuses purely on the meaning. It works just as well in Korean as it does in English.

Why Does This Matter?

Before this paper, if you wanted an AI to find "the cat sitting on the blue chair" in a photo, you often had to:

Train the AI on thousands of specific examples (expensive and slow).
Or, accept that it would fail if you asked in a different language.

With B2G:

No Training Needed: It works instantly on any pre-trained AI.
Better Precision: It improves accuracy by a huge margin (up to 7-8% better in tests), which is massive in the world of AI.
Universal: It works for English, German, Chinese, Korean, and many other languages without needing to retrain the model.

The Takeaway

The paper teaches us that don't just look at the AI's final answer. Sometimes, the most useful information is hidden in the "middle of the process." By peeking into the AI's intermediate thoughts and correcting its biases, we can make these powerful tools much more precise, fair, and useful for everyone, regardless of what language they speak.

In short: The AI wasn't broken; it was just looking at the wrong part of its own brain. B2G simply told it, "Hey, look here instead!" and suddenly, it could see perfectly.

1. Problem Statement

The paper addresses two critical limitations in Zero-Shot Referring Image Segmentation (RIS) when using pre-trained Vision-Language Encoders (VLEs) like CLIP or SigLIP:

Spatial Blindness: While VLEs excel at global semantic alignment, their final-layer embeddings often discard fine-grained positional information. This leads to models selecting background regions or incorrect objects when queries contain spatial cues (e.g., "left," "closest to").
Multilingual Bias: In multilingual settings, semantically equivalent sentences in different languages (e.g., English vs. German) exhibit systematic geometric shifts in the final embedding space. This causes the model to perform poorly on non-English queries, even if the visual target remains identical.

Existing training-free methods typically rely on final-layer embeddings, which are optimized for high-level abstraction but are suboptimal for pixel-level grounding and cross-lingual consistency. The authors hypothesize that intermediate (mid) layers of VLEs retain the necessary spatial structure and language-stable geometry that are lost in the final abstraction.

2. Methodology: Biased to Grounded (B2G)

The authors propose B2G, a training-free framework that operates on frozen VLEs using a Probe $\rightarrow$ Select $\rightarrow$ Ground paradigm. It does not require fine-tuning, gradient backpropagation, or architectural changes.

A. Spatial Bias Probing & P-Map Generation

Observation: Intermediate vision layers are sensitive to spatial arrangement, whereas final layers are position-invariant. However, raw cosine similarity in mid-layers often exhibits "opposite visualization" (foreground patches have negative correlation with the query).
Solution:
1. Sign Inversion: The authors apply a deterministic sign flip to patch embeddings before projection ( $\cos(-u, v) = -\cos(u, v)$ ). This corrects the anti-correlation, restoring foreground-localized activations.
2. P-Map Construction: A dense patch-level similarity map (P-Map) is generated by comparing these corrected mid-layer image patches with a hybrid text embedding.
3. Layer Selection: A spatial equivariance probe (using tile permutation) identifies the optimal mid-layer where spatial structure is best preserved.
4. Reranking: The P-Map is converted into a spatial guidance mask. Candidate masks from an off-the-shelf segmentor (e.g., Mask2Former) are reranked by combining the baseline image-text similarity with the overlap score against the P-Map guidance.

B. Multilingual Bias Probing & Centroid Injection

Observation: Final-layer text embeddings for the same sentence in different languages drift apart geometrically, distorting cross-modal alignment.
Solution:
1. Layer Selection: The framework probes the text encoder across $N$ parallel translations of a query. It calculates cross-language consistency (cosine similarity between embeddings of different languages) at each layer to find the language-stable layer ( $l^*_{txt}$ ).
2. Centroid Injection: At the selected mid-layer, a multilingual centroid is computed by averaging the embeddings of all $N$ translations.
3. Stabilization: This centroid replaces the end-of-text (EOT) token representation of the anchor language query. The modified representation is then propagated through the remaining layers. This creates a language-agnostic feature that neutralizes geometric drift before the final alignment bottleneck.

C. Hybrid Text Features

The method also introduces Context Tokens (CT), fusing global sentence-level features with local noun-phrase features augmented by context tokens (e.g., adjectives or prepositions following the noun) to improve spatial disambiguation.

3. Key Contributions

Diagnostic Insight: The paper reveals that VLEs are not inherently "blind" to position or language; rather, the final-layer abstraction interface suppresses these cues. Intermediate layers preserve the necessary structural and geometric stability.
Training-Free Framework (B2G): A plug-and-play solution that improves zero-shot RIS without retraining. It leverages mid-layer representations via a lightweight reranking mechanism (for spatial) and centroid injection (for multilingual).
Unified Approach: It simultaneously addresses spatial grounding and multilingual robustness, two problems often treated separately.
Novel Probing Mechanisms: Introduction of the Tile-Permutation Probe (to find spatially sensitive layers) and the Cross-Language Consistency Probe (to find language-stable layers).

4. Experimental Results

The method was evaluated on standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) and multilingual extensions (10 languages).

Standard Zero-Shot RIS:
- Outperformed strong baselines (Global-Local, HybridGL) across various backbones (CLIP ViT-B/32, ViT-B/16, SigLIP, DFN).
- Improvements: +2.15 mIoU on RefCOCO, +5.47 mIoU on RefCOCO+, and +5.75 mIoU on RefCOCOg (using CLIP ViT-B/16).
- Achieved state-of-the-art results on the challenging PhraseCut dataset.
Multilingual Zero-Shot RIS:
- Significantly improved performance across 9 translated benchmarks.
- Improvements: Average IoU@50 increased from 45.13 to 53.41; mIoU increased from 42.42 to 49.62.
- Rescued "collapsed" languages (e.g., German, Chinese) that performed poorly with baseline VLEs, while maintaining or improving English performance.
Zero-Shot Text-to-Image Retrieval:
- Improved Recall@1 on multilingual retrieval tasks, demonstrating that the method corrects the underlying geometric drift in the text encoder.
- Key Finding: The method acts as a structural regularizer, improving weak languages without sacrificing strong ones (no "English trade-off").
Efficiency:
- While multilingual probing adds some inference cost (approx. 1.5x–1.7x latency due to parallel translation processing), the spatial reranking is computationally lightweight compared to gradient-based saliency methods (e.g., Grad-CAM).

5. Significance and Impact

Paradigm Shift: Challenges the prevailing assumption that final-layer embeddings are the optimal interface for all VLE tasks. It demonstrates that intermediate representations are crucial for fine-grained spatial and cross-lingual reasoning.
Practicality: Offers a highly effective, training-free solution for deploying VLEs in real-world, multilingual, and spatially complex scenarios (e.g., embodied AI, robotics, medical imaging) without the cost of fine-tuning massive models.
Generalizability: The findings suggest that probing mid-layers could benefit other multimodal tasks requiring structural reasoning, such as visual question answering (VQA) or dense captioning.

In conclusion, B2G successfully unlocks latent spatial and multilingual capabilities within frozen VLEs by strategically bypassing the biased final-layer abstraction, providing a robust, training-free pathway for precise language-grounded understanding.