🤖 AI

CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities

To address limitations in current urban representation evaluations, such as spatial leakage and lack of generalization, the authors propose CityRep, a unified benchmark featuring spatially structured splits and a standardized framework to rigorously assess models across multiple cities, tasks, and modalities.

Original authors: Junyuan Liu, Xinglei Wang, Zichao Zeng, Jiazhuang Feng, Quan Qin, Ilya Ilyankou, Guangsheng Dong, Tao Cheng

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junyuan Liu, Xinglei Wang, Zichao Zeng, Jiazhuang Feng, Quan Qin, Ilya Ilyankou, Guangsheng Dong, Tao Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand how cities work. You have many different "teachers" (AI models) who have studied cities in different ways: some looked at satellite photos, others read lists of shops and restaurants, and some just memorized street addresses.

The problem is that until now, there was no fair way to see which teacher is actually the best. Most tests were like giving the robot a pop quiz where the answers were hidden right next to the questions. If the robot memorized the neighborhood it was tested on, it got a perfect score, but it couldn't actually understand the city. It was like a student who memorized the answer key instead of learning the subject.

CityRep is a new, fair "Olympics" for these city-understanding robots. Here is how it works, broken down simply:

1. The Problem: Cheating on the Test

In the past, researchers tested these AI models by randomly picking city blocks for training and testing. Because cities are connected (what happens on one street often happens on the next), the AI could just "peek" at the training data to guess the test answers. This made the scores look amazing, but the models were actually terrible at understanding new, unseen cities.

The Analogy: Imagine a student taking a math test. If the teacher puts the answer key for Question 5 right next to Question 6, the student gets a 100%. But if you move the student to a different classroom with a new test, they might fail. CityRep stops this "peeking."

2. The Solution: The "CityRep" Benchmark

The authors created a unified testing ground called CityRep. Think of it as a standardized driving test that happens in 8 different cities (London, New York, Singapore, etc.) with 8 different challenges (predicting traffic, population, air quality, etc.).

To make it fair, they introduced three main rules:

Rule #1: The Universal Translator (Spatial Alignment)
Some AI models speak "Pixel" (satellite images), some speak "Address" (street numbers), and some speak "Region" (neighborhoods). You can't compare them directly. CityRep acts as a translator. It takes the output from any model and converts it into the same format needed for the specific test. It's like converting all the answers into a single language before grading them.
Rule #2: The "No Peeking" Rule (Spatial Splits)
Instead of randomly mixing up the test questions, CityRep divides the city into big, separate blocks (like a grid). It trains the AI on one set of blocks and tests it on a completely different set of blocks that are far away.
The Analogy: Instead of testing a student on a neighborhood they just studied, you test them on a neighborhood they have never visited. If the model still gets a good score, it actually understands the city, not just the map.
Rule #3: The Multi-Task Challenge
A model might be great at predicting where people live but terrible at predicting air pollution. CityRep tests them on 8 different things:
- Morphology: What is the land used for? (Houses vs. factories)
- Demographics: How many people live there, and how old are they?
- Economy: How much money is made there? (GDP, Nighttime lights)
- Environment: Is the air dirty? Is the ground hot?

3. What They Found

The researchers tested 11 different "teachers" (AI models) using this new, fair system. Here is what happened:

The "Peeking" Score vs. The Real Score: When they used the old, unfair "random" method, the scores were high and the rankings looked one way. When they switched to the new "no peeking" method, the scores dropped, and the rankings changed completely. Some models that looked like winners were actually just good at memorizing.
The Big Winners: The models that learned from satellite imagery (looking at the whole city from space) generally performed the best across all tasks. They seem to have the most "general knowledge" about how cities function.
No One-Size-Fits-All: Even the best models struggled with certain cities or certain tasks. A model that is great at predicting population in New York might struggle with the same task in Mumbai. This proves that you can't just pick one model and say it's the "best" for everything; you have to test it in many different places.

The Bottom Line

CityRep is a tool to stop researchers from over-hyping their AI models. It forces them to prove that their models can actually understand the complex, messy reality of cities, not just memorize a specific map. By using this fair, block-based testing method, we can finally see which models are truly ready to help us build better, smarter cities in the future.

Where to find it: The authors have released all the data, the code, and the tools for free on GitHub so anyone can run these tests themselves.

Technical Summary: CityRep: A Unified Benchmark for Urban Representations

1. Problem Statement

Urban representation learning aims to encode heterogeneous urban observations (e.g., remote sensing, street-view imagery, points of interest, and geographic coordinates) into general-purpose spatial embeddings. These embeddings are intended to serve as foundational components for diverse downstream tasks and emerging urban foundation models. However, the evaluation of these representations suffers from three critical limitations:

Narrow Scope: Existing evaluations typically focus on one or two cities, a limited number of tasks, or a single label type. This restricts the ability to assess whether a representation is genuinely general-purpose or merely effective under specific, narrow conditions.
Incompatible Spatial Supports: Representation models operate on disparate spatial supports (raster grids, vector regions, H3 cells, or continuous coordinates), while downstream tasks often define units differently. Without a standardized alignment mechanism, fair comparison across heterogeneous models is impossible.
Spatial Leakage in Evaluation: Most studies rely on random train-test splits. In urban contexts, where data exhibits strong spatial autocorrelation, random splits allow models to achieve inflated performance by interpolating between nearby training and test samples rather than truly generalizing to unseen areas. This leads to over-optimistic conclusions about model transferability.

2. Methodology: The CityRep Framework

To address these gaps, the authors propose CityRep, a unified benchmark designed to evaluate urban representations across data modalities, cities, and tasks using spatially structured protocols. The framework consists of three core components:

A. Spatial Unit-Agnostic Evaluation & Alignment

CityRep introduces a standardized alignment module to make heterogeneous representations comparable. Instead of forcing all models onto a single universal grid, the framework aligns native representations to the specific spatial units of downstream tasks ( $U_{c,t}$ ) for each city ( $c$ ) and task ( $t$ ).

Mechanism: For raster or region-level embeddings, the system aggregates embeddings within task units or shares embeddings across coarser units. For entity-level embeddings (e.g., POIs), it first aggregates entities to an intermediate support (H3 cells) before mapping to task units to avoid missing features. For coordinate encoders, it queries the encoder at representative coordinates of task units.
Goal: This allows models based on rasters, regions, entities, or coordinates to be evaluated under the same downstream interface.

B. Spatially Structured Split Protocol

To mitigate spatial leakage, CityRep replaces random splits with block-based spatial splits.

Protocol: The spatial extent of a task is partitioned into non-overlapping blocks (e.g., a $10 \times 10$ grid). Train, validation, and test sets are formed by assigning entire blocks to disjoint subsets, rather than individual samples.
Rationale: This ensures that test samples are spatially separated from training samples, forcing models to demonstrate true generalization to unseen geographic areas rather than local interpolation.

C. Multi-City, Multi-Task Benchmark Suite

CityRep spans 8 cities (London, New York, Singapore, Sydney, Mumbai, Nairobi, Jakarta, Cape Town) and 8 downstream tasks organized into four urban domains:

Morphology: Land-use classification (LUC) and Road-density regression (RDE).
Demographics: Population regression (POP) and Age-distribution prediction (AGE).
Economy: GDP regression and Nighttime lights (NTL) regression.
Environment: PM2.5 regression and Land-surface-temperature (LST) regression.

The benchmark evaluates 11 representative models, including position encoders (PE), POI-based methods (Place2Vec, CaLLiPer), coordinate encoders (Space2Vec, SatCLIP), map-entity models (CityFM, Urban2Vec), and large-scale remote-sensing foundation models (TESSERA, AlphaEarth, AETHER).

3. Key Results

The authors conducted a large-scale empirical study with the following findings:

Impact of Split Protocols: Performance is highly sensitive to the split protocol. Random splits consistently inflate scores (particularly for dense raster tasks like population and LST) and alter model rankings compared to spatial splits. This confirms that random splits measure local interpolation rather than transfer to unseen areas.
Model Performance: Large-scale pretrained geospatial representations (AETHER, AlphaEarth, TESSERA) achieved the strongest overall transfer across tasks and cities, suggesting that broad spatial coverage and large-scale pretraining are valuable for heterogeneous urban prediction. However, specialized or simpler models (e.g., CityFM for LST, CaLLiPer for age distribution) remained competitive on specific tasks, indicating that input modality alone does not dictate performance; architecture and alignment strategy matter.
Contextual Variability: Model rankings and performance gaps vary significantly across cities and tasks. No single city is consistently easy or difficult for all prediction problems. For instance, a city might be easy for socioeconomic prediction but difficult for environmental prediction. This underscores that single-city evaluations provide an incomplete picture of model robustness.
Linear Probe Stability: Results remained broadly consistent when evaluated with a linear downstream head versus a non-linear MLP, suggesting that the superiority of remote-sensing foundation models is not solely an artifact of downstream model capacity.

4. Significance and Claims

The paper positions CityRep as a necessary step toward the development of urban foundation models. Its significance lies in:

Standardization: It provides a reproducible, unified interface for comparing heterogeneous urban representations, moving beyond narrow, single-setting comparisons.
Rigorous Evaluation: By enforcing spatially structured splits, it offers a more realistic assessment of generalization, preventing the overestimation of model capabilities common in current literature.
Diagnostic Insight: The benchmark reveals that urban representation quality is not static; it fluctuates based on urban context, task domain, and data availability. This challenges the notion of a single "best" model for all urban scenarios.
Resource Availability: The authors release datasets, evaluation pipelines, processed benchmarks, and diagnostic tools to facilitate fair comparison and future research.

The authors modestly note that while CityRep advances evaluation, it is constrained by the quality of available public geospatial data and does not yet fully investigate how multimodal data scarcity shapes global embeddings. They frame CityRep as a research benchmark to guide the development of general-purpose urban representations rather than a decision-making system for direct policy deployment.