💻 computer science

UNIGEOCLIP: Unified Geospatial Contrastive Learning

UNIGEOCLIP is a unified multimodal contrastive learning framework that aligns five geospatial modalities through all-to-all alignment and a scaled latitude-longitude encoder, enabling superior cross-modal retrieval and reasoning across arbitrary combinations of data types.

Original authors: Guillaume Astruc, Eduard Trulls, Jan Hosang, Loic Landrieu, Paul-Edouard Sarlin

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guillaume Astruc, Eduard Trulls, Jan Hosang, Loic Landrieu, Paul-Edouard Sarlin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a specific neighborhood in a city to a friend who has never been there.

If you only give them GPS coordinates (like "40.7128° N, 74.0060° W"), they have a dot on a map, but no idea what it looks like.
If you only send them a satellite photo, they see the rooftops but can't smell the coffee shop or hear the traffic.
If you only send a street-level photo, they see the sidewalk but don't know the layout of the whole block.
If you just write a text description ("It's a busy area near a park"), it's vague.

Usually, AI models are like specialists: one model is great at reading maps, another is great at recognizing buildings from above, and a third is great at reading text. They rarely talk to each other.

UNIGEOCLIP is like a "Universal Translator" that forces all these different ways of seeing the world to speak the same language.

The Big Idea: The "All-to-All" Party

Most previous AI models tried to connect everything to one "main character" (usually the satellite image). Think of it like a party where everyone has to talk to the host, but the guests can't talk to each other.

UNIGEOCLIP changes the rules. It throws a massive party where everyone talks to everyone.

The Satellite Image talks to the Street View.
The Street View talks to the Text Description.
The Text talks to the Elevation Map (a 3D model of the ground).
And the GPS Coordinates talk to all of them.

By forcing these five different "languages" (Satellite, Street View, 3D Elevation, Text, and GPS) to align perfectly, the AI learns a single, unified "mental map" of the world. If you show it a street photo, it instantly understands the satellite view, the text description, and the exact coordinates, even if it has never seen that specific photo before.

The Secret Sauce: The "Smart GPS"

One of the paper's biggest innovations is how it handles GPS coordinates.

Usually, AI treats a location like a simple number (e.g., "40.7"). But the world isn't flat; it's curved, and neighborhoods have complex structures. A simple number can't capture that.

The authors built a Scaled Latitude-Longitude Encoder. Think of this as a "super-magnifying glass" for maps. Instead of just looking at a location as one dot, this encoder looks at it through multiple lenses at once:

The Wide Lens: It sees the big picture (the whole city).
The Zoom Lens: It sees the neighborhood layout.
The Micro Lens: It sees the specific street block.

By combining these views, the AI doesn't just know where a place is; it understands the structure of the place. This is why the model can predict things like "Is this a rich neighborhood?" or "Is this a park?" just by looking at the GPS coordinates alone.

Why Does This Matter? (Real-World Magic)

Because the AI now understands the world so holistically, it can do some cool tricks:

The "Sherlock Holmes" Search: You can upload a photo of a street corner, and the AI can instantly find the matching satellite photo or the exact GPS address, even if the photo was taken in a different year or from a weird angle.
The "Time Traveler": The model was trained on data from 2017–2024 but tested on 2023 data. It didn't get confused by new buildings or changed roads. It learned the essence of the city, not just a snapshot of it.
The "Mind Reader": If you give it a text description like "a busy area with a Whole Foods and a park," the AI can point you to the exact location on a map, or find the satellite image that matches that description.
The "3D Vision": It can look at a flat 2D map and understand the 3D shape of the terrain (hills, buildings, valleys) because it learned to connect the flat image with the 3D elevation data.

The Analogy of the "Universal Passport"

Imagine every place on Earth has a Universal Passport.

Before UNIGEOCLIP, a place had five different passports (one for the map, one for the photo, one for the text), and they didn't match.
UNIGEOCLIP creates one single passport that contains all that information.
If you show the AI a street photo (one page of the passport), it can instantly read the other pages (the map, the text, the 3D shape) because they are all stamped with the same unique code.

In Summary

UNIGEOCLIP is a breakthrough because it stops treating maps, photos, text, and coordinates as separate things. It fuses them into one giant, intelligent brain that understands the Earth in 3D, with text, and from every angle simultaneously. This makes it much better at finding places, understanding cities, and predicting how our world changes over time.

1. Problem Statement

Current geospatial representation learning suffers from fragmentation and limited modality integration. Existing approaches generally fall into three categories, each with significant limitations:

Embedding Fields: Map coordinates to latent vectors but act as static "snapshots" of a region, failing to model dynamic changes or integrate rich semantic data.
Multimodal Fusion Models: Combine multiple sensors (e.g., satellite, SAR) into a single representation, which prevents cross-modal comparison or retrieval between different data types.
Contrastive Approaches (e.g., GeoCLIP, SatCLIP): Typically align geographic coordinates with only one observation modality (usually top-down imagery) and largely ignore textual information or ground-level perspectives.

The Core Challenge: There is a lack of a unified framework that can jointly align heterogeneous geospatial data (aerial imagery, street-level views, elevation models, text, and raw coordinates) into a single embedding space to enable seamless reasoning, retrieval, and comparison across arbitrary modality combinations.

2. Methodology

UNIGEOCLIP proposes a massively multimodal contrastive framework that aligns five complementary modalities in a single unified embedding space without relying on a central "pivot" modality.

A. Architecture

The system employs modality-specific encoders ( $\phi_m$ ) for five data types:

Aerial Imagery & Street-Level Views: Encoded using the image encoder from SigLIP-2.
Text Descriptions: Encoded using the SigLIP-2 text encoder. These descriptions are automatically generated from georeferenced data, capturing land use, landmarks, and context.
Digital Surface Models (DSM): A dedicated Vision Transformer (ViT) with register tokens is trained from scratch to encode elevation and geometric structure.
Geographic Coordinates (GPS): A novel Scaled Latitude-Longitude Encoder is proposed.
- Preprocessing: Coordinates are mapped to a planar representation using the Equal Earth Projection to mitigate spherical distortions.
- Encoding: Instead of simple MLPs, the encoder uses Random Fourier Features (RFF) with multiple spectral bandwidths ( $\sigma_k$ ).
- Multi-Scale Fusion: Each frequency projection is treated as a token. These tokens are processed through self-attention blocks (Transformer layers) to enable interaction between different spatial scales. The final output is a pooled $D$ -dimensional embedding.

B. Supervision: All-to-All Contrastive Learning

Unlike prior works that use a pivot modality (e.g., ImageBind aligns everything to images), UNIGEOCLIP uses a fully holistic, all-to-all contrastive objective.

Loss Function: The model minimizes the average InfoNCE loss over all ordered pairs of modalities $(m, n)$ .
Mechanism: For a batch of $B$ samples, every modality of a specific location is contrasted against every other modality of the same location (positive pairs) and against all other locations (negative pairs).
Result: This enforces a single shared latent space where any modality can retrieve or reason about any other modality.

3. Key Contributions

UNIGEOCLIP Framework: The first purely contrastive framework to jointly align five georeferenced modalities: Street View, Aerial Imagery, DSMs, Text, and GPS coordinates.
Scaled Latitude-Longitude Encoder: A novel coordinate encoder that outperforms standard Fourier-feature and MLP-based embeddings by explicitly modeling multi-scale spatial dependencies via self-attention.
Holistic Alignment Strategy: Moving away from pivot-based alignment to an all-to-all formulation, ensuring text, elevation, and visual sensors are "primary citizens" in the embedding space.
Comprehensive Evaluation: Demonstrated state-of-the-art performance across cross-modal retrieval, zero-shot geolocation, and diverse downstream tasks (land-cover segmentation, socio-economic inference).

4. Experimental Results

The model was trained on a large-scale dataset covering the continental USA (approx. 800k S2 cells) with a strict temporal split (training on 2017–2024, testing on 2023).

A. Cross-Modal Retrieval

Performance: UNIGEOCLIP significantly outperforms retrained baselines like GeoCLIP and ScalingGeoloc.
Key Metric: In cross-view retrieval (Street View $\to$ Aerial), it achieved 88.2% Acc@100m when using all modalities, compared to 24.6% for GeoCLIP.
Ensembling: Combining modalities (e.g., Aerial + DSM + Text) consistently yielded higher retrieval accuracy than any single modality, proving the complementarity of the data sources.
Generalization: The model maintained competitive performance on an Out-of-Distribution (OOD) dataset from Amsterdam, demonstrating robustness to geographic shifts.

B. Downstream Tasks

Satellite Image Encoder: Achieved 97.0% OA on solar panel detection and 66.3 mIoU on land-cover segmentation, outperforming specialized Earth observation models (e.g., AnySat, Panopticon) and general vision models (e.g., SigLIP-2) despite being trained without task-specific supervision.
Coordinate Encoder: On 27 regression tasks (health, socio-economic, environmental indicators), the encoder achieved a mean $R^2$ of 57.0, substantially beating AlphaEarth (29.0) and GeoCLIP (49.8).
DSM Encoder: In land-cover segmentation on DSM data, the linearly probed UNIGEOCLIP encoder reached 72.0% accuracy, far surpassing U-Net (45.5%) and ViT-B (39.1%) trained from scratch.

C. Ablation Studies

Modality Contribution: Performance improved monotonically as more modalities were added to the contrastive training.
Encoder Depth: Increasing the depth of the coordinate encoder (from 0 to 12 Transformer blocks) significantly improved retrieval and regression performance, confirming that complex geographic structure requires non-linear interactions across scales.
Visualization: PCA and t-SNE visualizations showed that UNIGEOCLIP produces spatially structured embeddings that reflect urban features (e.g., Central Park), whereas baselines produced smoother, position-driven gradients.

5. Significance

UNIGEOCLIP represents a paradigm shift in geospatial AI by moving from specialized, single-modality models to a unified, multimodal foundation.

Robustness: By learning a shared manifold, the system can infer missing modalities or reason across them (e.g., retrieving a location from a text description or an elevation map).
Efficiency: It demonstrates that contrastive pre-training can distill rich structural and semantic information into encoders, enabling strong downstream performance without massive task-specific fine-tuning.
Scalability: The framework provides a scalable foundation for large-scale Earth observation, urban planning, and socio-economic inference, bridging the gap between ground-level, aerial, and textual data.

The reference implementation is available at gastruc.github.io/unigeoclip.