CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing

1. Problem Statement

Urban socioeconomic sensing—the quantification of indicators like income, education, health, and crime from visual data—is critical for sustainable urban planning and policy-making. While traditional deep learning methods have been used for this task, they face limitations in handling unstructured multi-modal data, generalizing across different countries, and interpreting culturally nuanced aspects of urban environments.

Although Large Vision-Language Models (LVLMs) possess the theoretical capability to integrate visual and textual modalities and generalize globally, there is a lack of systematic benchmarks to evaluate their actual performance in predicting complex socioeconomic indicators. Existing works often suffer from limited geographic coverage, narrow indicator diversity, and a lack of unified evaluation frameworks.

2. Methodology

A. Dataset Construction (CityLens)

The authors constructed CityLens, a comprehensive multi-modal benchmark designed to evaluate LVLMs.

• Scope: Covers 17 globally distributed cities across 6 continents (including the US, UK, China, and cities in Africa, South America, and Asia).
• Indicators: Defines 11 prediction tasks spanning 6 socioeconomic domains: Economy, Education, Crime, Transport, Health, and Environment.
  • Examples: GDP, House Price, Population, Bachelor Ratio, Violent Crime, Mental Health, Building Height, etc.
• Data Modalities: Each prediction unit (region) consists of 1 satellite image (4.7m resolution from Esri) and 10 street view images (sourced from Google, Baidu, and Mapillary APIs).
• Data Processing: Ground-truth labels are aggregated from heterogeneous sources (e.g., Census Tracts, MSOA, WorldPop) to match the visual regions. The dataset includes both a proprietary version (Google/Baidu) and an open-source version (Mapillary) for reproducibility.

B. Evaluation Paradigms

The paper proposes three distinct evaluation methods to probe different capabilities of LVLMs:

1. Direct Metric Prediction: The model is prompted to output a specific numerical value for an indicator (e.g., "What is the GDP?") based on the images. This tests the model's ability to perform zero-shot numerical regression.
2. Normalized Metric Estimation: Indicator values are normalized to a scale of 0.0–9.9. The model estimates this relative score. This tests coarse-grained spatial knowledge and the ability to rank regions without requiring precise numerical grounding.
3. Feature-Based Regression: The LVLM acts as a feature extractor. It scores 13 predefined visual attributes (e.g., Greenery, Building Facade, Vehicle density) for each street view image. These scores are aggregated into a feature vector and fed into a LASSO regression model to predict the ground-truth socioeconomic indicator. This isolates the model's perceptual capability from its numerical reasoning.

C. Experimental Setup

• Models: Benchmarked 17 state-of-the-art LVLMs, including open-source models (Gemma 3, Qwen2.5-VL, Llama 4, Mistral) and proprietary models (Gemini 2.0/2.5, GPT-4o/4.1, MiniMax).
• Metrics: Performance is measured using $R^2$ (Coefficient of Determination) and nRMSE (Normalized Root Mean Squared Error).
• Ablation Studies: The authors analyzed the impact of input modalities (satellite vs. street view), image quantity, Chain-of-Thought (CoT) prompting, and different vision encoders.

3. Key Contributions

1. Largest Socioeconomic Benchmark: CityLens is the most extensive benchmark to date in terms of geographic coverage (17 cities), indicator diversity (11 indicators, 6 domains), and model scale (17 LVLMs).
2. Unified Evaluation Framework: Introduces a three-pronged evaluation strategy (Direct, Normalized, Feature-Based) to comprehensively diagnose model strengths and weaknesses in urban sensing.
3. Comprehensive Analysis: Provides deep insights into how model architecture, training data, and prompt design affect performance, revealing that current LVLMs struggle with abstract socioeconomic reasoning despite strong visual perception.
4. Open Resources: The code, data (including Mapillary-based open-source version), and evaluation scripts are publicly released to foster reproducibility.

4. Key Results

• Overall Performance: Current LVLMs face significant challenges. While they show promise in visually grounded tasks (e.g., Building Height, $R^2 \approx 0.59$), they perform poorly on abstract indicators like Mental Health and Bachelor Ratio (often $R^2 \approx 0$ or negative).
• Evaluation Paradigm Comparison:
  • Feature-Based Regression consistently outperforms Direct and Normalized prediction. This suggests LVLMs are better at extracting structured visual features than acting as direct numerical predictors.
  • Direct vs. Normalized: Tasks with clear visual correlates (e.g., House Price) favor Direct prediction, while tasks with weak visual signals (e.g., GDP, Population) sometimes benefit from Normalized estimation, though performance remains low overall.
• Model Architecture & Scale:
  • Scaling Laws: Increasing model size does not guarantee better performance (e.g., Gemma3-27B performed worse than Gemma3-12B on GDP).
  • Architecture Matters: Models with specific training or architectures (e.g., Gemma series) outperformed others with similar parameter counts.
  • Vision Encoders: LVLMs initialized with CLIP as the vision encoder generally produced the most informative features for socioeconomic tasks compared to DINOv2 or SigLIP.
• Input Modalities:
  • Street View > Satellite: Contrary to some prior beliefs, street view images alone often outperformed satellite images or the combination of both, likely due to richer semantic cues (façades, signage) at the human scale.
  • Image Quantity: Performance improved progressively as the number of street view images increased (from 1 to 20).
• Reasoning & Prompting:
  • Chain-of-Thought (CoT): CoT prompting improved performance for complex tasks like House Price but degraded performance for tasks relying on direct visual features (e.g., Drive Ratio).
  • Reasoning Models: Specialized "thinking" models did not consistently outperform standard models, suggesting urban sensing requires visual-contextual inference rather than pure logical reasoning.
• Fine-Tuning Potential: Supervised fine-tuning (SFT) on domain-specific data significantly boosted performance (e.g., Fine-tuned Qwen2.5-VL-7B achieved $R^2 > 0.6$ on several tasks), indicating that the bottleneck is often a lack of domain adaptation rather than fundamental model incapability.
• Bias: The study identified a geographic bias, where models performed significantly better on "Global North" cities compared to "Global South" cities, highlighting issues in generalization across cultural and economic contexts.

5. Significance

• Diagnostic Tool: CityLens provides a unified framework to diagnose why LVLMs fail in urban sensing, distinguishing between perceptual failures (missing visual cues) and reasoning failures (misinterpreting context).
• Future Directions: The results suggest that future research should focus on:
  • Developing domain-specific LVLMs via fine-tuning.
  • Designing prompts that better mimic human reasoning patterns for urban contexts.
  • Addressing geographic and cultural biases to ensure equitable global application.
  • Exploring hybrid approaches where LVLMs extract features for traditional regression models rather than acting as end-to-end predictors.
• Ethical Implications: The authors emphasize that while the benchmark is for research, the technology must not be used for real-world policing or resource allocation without rigorous ethical review and validation, given the sensitivity of socioeconomic data.

In conclusion, CityLens establishes a new standard for evaluating LVLMs in urban science, revealing that while these models have strong perceptual foundations, they currently lack the robustness and domain-specific grounding required for accurate socioeconomic prediction without further adaptation.