💬 NLP

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

This study demonstrates that textual supervision significantly enhances the geospatial representations in vision-language models compared to vision-only architectures, highlighting the critical role of language as a complementary modality for improving spatial accuracy in geospatial AI.

Original authors: Marcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon, Luiz Felipe Vecchietti, Bryan Nathanael Wijaya, Cheng Yaw Low, Virgilio Almeida, Meeyoung Cha

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon, Luiz Felipe Vecchietti, Bryan Nathanael Wijaya, Cheng Yaw Low, Virgilio Almeida, Meeyoung Cha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Do AI Eyes Know Where They Are?

Imagine you have two friends: Alex and Sam.

Alex is a pure visual expert. He has seen millions of photos but has never read a single word. He knows what a "cat" or a "pyramid" looks like, but he doesn’t know the names of places or languages.
Sam is a visual expert who also reads books. Sam has seen millions of photos and read the captions that go with them. Sam knows that the photo of the pyramid is called the "Step Pyramid of Djoser" and that it is in "Saqqara, Egypt."

The researchers wanted to know: If you show both Alex and Sam a picture of a famous landmark, can they guess exactly where on Earth it is? And more importantly, how do they figure it out inside their "brains" (the AI models)?

The Experiment: The "Where Are We?" Test

The team took several types of AI models:

Vision-Only Models (Like Alex): These only look at pixels.
Vision-Language Models (Like Sam): These look at pixels and process text.

They showed these models thousands of images from different categories:

Easy stuff: Famous landmarks (Eiffel Tower), streets, and buildings.
Hard stuff: Close-ups of food, random objects, or people’s faces.

They then asked the AI’s internal layers to predict the latitude and longitude (the exact GPS coordinates) of the image.

The Findings: Text is a Secret Map

1. Sam (Vision-Language) is much better at geography than Alex (Vision-Only).
Even though Alex has seen the same images, Sam is significantly better at guessing the location. Why? Because when Sam read the captions during training, he learned that "Eiffel Tower" goes with "Paris, France." The text acted like a secret map that helped the visual part of the brain understand where things are.

2. The "Brain Layers" behave differently.

In Alex (Vision-Only): The location information is hidden deep in the back of his brain. You have to look at the very last layers of his thinking process to find the GPS data.
In Sam (Vision-Language): The location information appears early on, right when the text and image meet. But here’s the twist: if you don’t ask Sam a question, he starts to forget the location as he processes the image further. However, if you prompt him with a question like "Guess the latitude and longitude," he suddenly remembers the location clearly in the later stages of his thinking.

3. Some pictures are easier to place than others.
Both Alex and Sam are good at placing landmarks, streets, and signs. They are terrible at placing close-ups of food, drinks, or random objects. This makes sense—there are no geographic clues in a picture of a cup of coffee unless you can read the label on the cup (which helps Sam, but not Alex).

4. The "GPS" is stored in a tiny corner of the brain.
The researchers found that the AI doesn’t spread the location data across its entire brain. Instead, it packs the GPS information into a small, specific subset of its internal features. It’s like finding a specific drawer in a huge library where all the maps are kept.

The "Magic Trick": Swapping Locations

The researchers did a fun experiment to prove that the AI really knows the location.

They took a picture of the Step Pyramid in Egypt. They looked inside the AI’s brain, found the specific "drawer" that held the location data for Egypt, and swapped it with the location data for Rome, Italy (from a picture of the Trevi Fountain).

When they asked the AI to describe the pyramid, it didn’t say, "This is the Step Pyramid in Egypt."
It said, "This is the Step Pyramid in Rome, Italy."

The AI kept the visual description of the pyramid (because they didn’t touch the visual parts of the brain) but changed the location because they swapped the "location drawer." This proves that the AI separates what an object is from where it is.

Why This Matters (According to the Paper)

Better AI for Real-World Tasks: If you want an AI to help with tasks that need location awareness (like identifying which country a photo was taken in), using models that understand text (Vision-Language) is much more effective than using models that only see images.
Privacy Risks: This is a double-edged sword. Because these models are so good at guessing locations from images, bad actors could potentially use them to figure out where people are just by looking at their photos. The paper warns that this raises serious privacy and safety concerns.
Bias: The models are much better at guessing locations in Europe and North America because they have seen more photos from there. They struggle with places in Africa, South America, and Oceania. This means the AI’s "geography knowledge" is uneven and biased toward wealthy regions.

In Summary

The paper shows that language helps AI see geography. By reading text captions, AI models learn to associate visual features with specific places on Earth. This makes them much better at geolocation than models that only look at images. However, this power comes with risks, including privacy violations and geographic bias.

Technical Summary: Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Problem Statement
The paper addresses the underexplored dimension of geospatial understanding in machine learning systems, specifically investigating whether vision-only architectures and vision-language models (VLMs) implicitly encode geolocation information (latitude and longitude) during pretraining and fine-tuning, even without explicit geospatial supervision. While prior work has demonstrated that models can be trained for geo-localization, this study focuses on the latent features acquired by existing foundation models. The authors aim to determine the extent to which these models internalize global location knowledge and how architectural choices and training modalities (vision-only vs. multimodal) influence the emergence of these geospatial representations.

Methodology
The authors conduct a systematic analysis using linear probing to extract and evaluate geospatial representations from the hidden layers of various model families.

Models Evaluated: The study examines three categories of models:
- Vision-only architectures: ViT, ViT Masked Autoencoder (MAE), DINOv2, and Web-SSL models.
- Vision-Language Models (VLMs): CLIP, MetaCLIP, LLaVA-1.5, Qwen2.5, and Gemma 3.
- For each family, multiple size variants are tested to assess the impact of scale.
Datasets:
- YFCC100M Subset: A 4M-image subset from the MediaEval 2016 Placing Task, clustered into 40 semantic categories (e.g., people, landmarks, objects) using k-means clustering on ResNet-152 embeddings.
- Google Landmarks: A subset of 580k images of globally recognized landmarks with geolocation coordinates from OpenStreetMap.
- Sampling Strategy: To mitigate spatial bias toward Europe and North America, the authors partitioned the globe into non-overlapping geocells based on administrative boundaries and sampled images to ensure balanced geographic coverage.
Probing Technique:
- The authors employ layer-wise linear probing using ridge regression to predict latitude and longitude from the residual stream representations ( $A^{(l)}$ ) of each layer $l$ .
- Performance is measured using the coefficient of determination ( $R^2$ ).
- The study analyzes where in the network depth these representations emerge and how they change when textual prompts are introduced.
Intervention Analysis:
- The authors perform representation swapping experiments on Qwen2.5-VL-3B. By replacing the top geospatially relevant feature dimensions of a source image’s residual stream with those of a target image, they test if the model’s generated text can be steered to reflect the target location while preserving other semantic content.

Key Contributions

Comparative Analysis of Geospatial Structure: The paper demonstrates that VLMs and large-scale multimodal foundation models exhibit substantially stronger geospatial structure in their latent representations compared to vision-only architectures.
Layer-wise Representation Dynamics: The study reveals distinct patterns in where geospatial information is encoded:
- Vision-only models tend to exhibit stronger geospatial representations in their deepest layers.
- VLMs show better geospatial representations in the early layers of their language model blocks. Without textual prompts, these signals often degrade or stagnate in later layers.
Role of Textual Supervision: The authors show that prompting VLMs with geospatial queries allows geospatial information to propagate to later layers, sometimes increasing representation quality. This suggests that language acts as an effective complementary modality for encoding spatial context.
Compactness of Geospatial Features: The analysis indicates that geospatial information is concentrated in a compact subset of the latent space (approximately 40% of dimensions), rather than being uniformly distributed.

Results

Performance Disparity: VLMs significantly outperform vision-only models in geolocation prediction. For instance, on the Landmarks dataset, VLMs achieved $R^2$ values up to 0.8, whereas vision-only models generally remained below 0.3. Even smaller VLMs (e.g., CLIP-base) outperformed much larger vision-only models (e.g., DINOv2-giant).
Cluster-wise Variability: Performance varied by image cluster. Landmarks, streets, and buildings were consistently highly localizable. Objects and food were the least localizable. Notably, VLMs showed localizability for "signs and text" clusters, a capability not observed in vision-only models.
Layer Dynamics: In vision-only models, $R^2$ increased progressively with layer depth. In VLMs, $R^2$ increased initially but then stagnated or decreased in later layers when no prompt was present. When prompted with "Guess the latitude and longitude," the performance in VLMs remained stable or increased in later text layers, with Qwen2.5-VL-7B achieving an $R^2$ of 0.88 on landmarks.
Representation Swapping: The authors successfully steered model generation by swapping geospatial dimensions. For example, swapping representations from an image of the Step Pyramid of Djoser with those of the Trevi Fountain caused the model to generate text describing the pyramid as being in Rome, Italy. However, this intervention led to instability in longer text generations.
Downstream Task Impact: Fine-tuning for country identification showed that models with stronger implicit geospatial representations (higher probe $R^2$ ) achieved better downstream performance, corroborating the utility of these representations.

Significance and Implications
The paper claims that textual supervision is a primary factor in learning strong geospatial representations, enhancing the efficiency of learning compared to scaling vision-only models alone. This supports the view that multimodal learning is a key direction for advancing geospatial AI.

The authors highlight significant ethical and privacy implications. The ability of these models to extract precise location data poses risks for stalking, mass surveillance, and geographic disparities in performance (with lower accuracy in underrepresented regions). Consequently, the paper argues for robust regulatory policies, transparency in model use, and explicit user consent to ensure safe deployment. The findings suggest that leveraging VLM representations can improve sample-efficient pipelines for geospatial tasks, particularly where labeled data is scarce.