From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models

Technical Summary: From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models

Problem Statement
Image geolocalization—the task of inferring geographic coordinates or addresses from visual data—is a critical capability for applications in crisis response, digital forensics, and location-based intelligence. While traditional computer vision approaches (classification-based and retrieval-based) have addressed this task, they often struggle with generalization to unseen environments, lack interpretability, and fail to incorporate multimodal semantic cues (e.g., text, cultural markers) that humans use intuitively. Although Large Language Models (LLMs) have demonstrated advanced visual reasoning and world knowledge, their specific capabilities and limitations regarding fine-grained image geolocalization remain underexplored. Existing benchmarks often rely on established datasets that may be contaminated by LLM training corpora or lack standardized metrics for spatial reasoning and geographic bias.

Methodology
The authors introduce IMAGEO-Bench, a systematic framework designed to evaluate the image geolocalization abilities of LLMs across accuracy, computational cost, geographic fairness, and reasoning processes.

• Dataset Construction: To mitigate data contamination and ensure diversity, the authors compiled three distinct datasets:
  1. Dataset-GSS (Global Streetscape Set): 6,152 high-quality street-level images from 396 cities across 123 countries, derived from the NUS Global Streetscapes dataset.
  2. Dataset-UPC (U.S. POIs Crowdsourced Set): 2,929 images of Points of Interest (POIs) spanning all 50 U.S. states and the District of Columbia, sourced from Google Maps.
  3. Dataset-PCW (Privately Collected Wild Set): 220 original photo–address pairs collected by the authors to serve as a held-out validation set with no overlap with public training sources.
• Task Formalization: The benchmark defines a structured end-to-end task where models must output a JSON tuple containing: (1) Image information (environment, scene type, setting), (2) Reasoning (landmarks, text/signage, cultural indicators, spatial context), and (3) Geolocalization (confidence score, address, and coordinates). Models are restricted from accessing external APIs or GPS metadata during inference.
• Evaluation Metrics: Performance is measured via identification rate, geographic accuracy (country, state, city levels), great-circle distance error (Haversine formula), and computational cost (tokens and monetary cost).
• Diagnostic Analysis: The study employs post-hoc logistic and ridge regression to analyze the correlation between specific visual features (e.g., urban vs. rural, presence of landmarks) and model performance. Additionally, word-cloud analysis and text-masking ablation studies are conducted to understand reliance on textual cues.

Key Results
Experiments were conducted on 10 state-of-the-art LLMs, including both open-source (e.g., Llama series) and closed-source (e.g., GPT-4, Gemini, Claude) models.

• Performance Disparities: Closed-source models generally outperform open-source counterparts. Gemini-2.5-pro achieved the best results on Dataset-GSS with a median distance error of 4.2 km and 92.4% country-level accuracy. o3 and gpt-4.1 also demonstrated strong performance. In contrast, open-source models like llama-4-17b and llama-3.2-11b exhibited significantly higher errors (e.g., 127.0 km and 66.6 km median errors on GSS, respectively).
• Geospatial Bias: A pronounced geographic bias was observed. Models performed significantly better in high-resource regions with dense training data and distinctive visual cues, such as North America, Western Europe, and Australia. Performance degraded sharply in underrepresented areas (e.g., parts of Latin America, Africa, and Asia). Within the U.S., states like California and Florida showed higher accuracy compared to others.
• Feature Dependence: Regression diagnostics revealed that successful geolocalization is primarily dependent on recognizing urban settings, outdoor environments, street-level imagery, and identifiable landmarks. Models struggle significantly with rural, natural, and indoor scenes where distinctive visual anchors are absent.
• Textual Reliance: Text-masking ablation studies showed that performance on global street scenes (Dataset-GSS) drops drastically (22–28% accuracy loss) when textual cues are removed, indicating a heavy reliance on OCR-visible elements. This dependency is less pronounced in the U.S. POI dataset (Dataset-UPC), where text is often generic.
• Cost vs. Accuracy: A trade-off exists between cost and performance. While open-source models like llama-3.2-11b are highly cost-efficient, they sacrifice accuracy. Gemini-2.5-flash offered a notable balance, achieving 41.7% city accuracy at a low cost ($0.00141 per request).

Significance and Claims
The paper claims that IMAGEO-Bench provides a rigorous lens for assessing the spatial reasoning capabilities of LLMs, moving beyond simple accuracy to evaluate reasoning transparency and geographic fairness. The authors assert that while LLMs offer a promising avenue for overcoming the limits of conventional geolocalization (particularly with out-of-distribution data), they are not yet equipped to generalize robustly across diverse global geographies. The study highlights that current models rely heavily on discrete, regulative visual anchors (signage, landmarks) rather than holistic scene understanding. The authors conclude that future work must address geographic bias and explore strategies to improve performance in underrepresented regions to build more equitable and globally robust geolocation systems.