EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery

This paper introduces EarthSpatialBench, a comprehensive benchmark comprising over 325K question-answer pairs designed to evaluate and expose the limitations of multimodal large language models in performing complex, quantitative spatial reasoning on Earth imagery, including distance, direction, topology, and diverse geometric object representations.

The Gist

Imagine you have a super-smart robot assistant that can look at a photo and chat with you about it. You might ask, "What's that building?" or "Is that a dog?" and it would answer correctly. But now, imagine you ask it something much trickier: "How far is that red house from the river, and how many other houses are within a 10-minute walk from it?"

Most of today's smart AI assistants would stumble. They are great at naming things, but terrible at understanding space, distance, and geometry on a map.

This paper introduces a new tool called EarthSpatialBench to test exactly how good these AI robots are at "reading the map." Here is the breakdown in simple terms:

1. The Problem: The AI is "Map-Blind"

Think of current AI models like a tourist who has never used a compass or a ruler.

• They can see a picture of a city and say, "That's a park."
• But if you ask, "Is the park inside the fence, or is the fence around the park?" or "How many meters is the park from the road?", they often guess wildly.
• Real-world tasks (like helping during a flood or planning a new city) require precise answers, not just guesses.

2. The Solution: A "Map-Reading" Final Exam

The authors created EarthSpatialBench, which is like a giant, rigorous final exam for AI, specifically designed for satellite and drone photos of the Earth.

Instead of just asking "What is this?", the exam asks three types of hard questions:

• The Ruler Test (Distance): "How far is that house from the river?" (The AI needs to give a number, not just "near" or "far").
• The Compass Test (Direction): "Is that building to the North-East or South-West of the silo?" (The AI needs to calculate angles).
• The Puzzle Test (Topology): "Is this road cutting through the park, or is the park inside the road loop?" (The AI needs to understand how shapes fit together).

3. The Exam Materials: A Giant Box of Puzzles

To make this exam fair and tough, they built a dataset with 325,000 questions based on real satellite images.

• The Objects: They didn't just use simple boxes. They used polygons (shapes that look like actual park boundaries), polylines (wiggly lines for rivers and roads), and boxes (for buildings).
• The References: Sometimes the AI has to find an object because you described it ("The only red house"), and sometimes because you gave it exact coordinates ("The house at these GPS points").

4. The Results: The AI is Still a "Novice"

The researchers tested the world's smartest AI models (like GPT-5, Gemini, and Claude) on this exam. Here is what they found:

• Good at Chatting, Bad at Math: The AIs are great at saying "Yes" or "No" to simple questions, but they struggle to give exact numbers for distances or angles. It's like a student who can tell you "it's far away" but fails when asked to measure it in inches.
• The "Grounding" Gap: This is the biggest issue. To answer a math question about a map, the AI first has to find the object on the picture. If the AI can't accurately point to the "red house" on the image, it can't calculate the distance to the river. The study found that many AIs are "hallucinating" (imagining) where things are, which ruins their math.
• Visual vs. Text: When the researchers gave the AI a picture with a red circle drawn around the target object, the AI got better at finding it. But when they just gave text instructions, the AI got confused. This shows the AI is still learning how to connect words to pixels.

5. Why Does This Matter?

Imagine a future where AI helps save lives during a hurricane.

• Current AI: "I see some water. Maybe people are in trouble?"
• Future AI (with this benchmark): "I see 15 houses flooded within 50 meters of the river. The nearest road is 200 meters away. Send rescue boats to these exact coordinates."

The Takeaway

EarthSpatialBench is a wake-up call. It shows that while AI is getting smarter at "seeing" and "talking," it is still clumsy at "measuring" and "navigating."

The authors hope that by giving AI this tough "map-reading" exam, developers will build better robots that can one day truly understand the physical world, helping us plan cities, monitor the environment, and respond to disasters with precision. Until then, we shouldn't trust an AI to drive a rescue boat just yet!

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in natural image understanding, their spatial reasoning capabilities on Earth imagery (satellite, aerial, and drone views) remain underdeveloped. Existing benchmarks for Earth imagery suffer from three critical limitations:

• Lack of Quantitative Reasoning: They focus on coarse spatial relations (e.g., "left of," "near") rather than precise quantitative metrics like Euclidean distance and azimuth angles.
• Limited Topological Complexity: They rarely evaluate systematic topological relations (e.g., intersection, containment) required for complex geospatial analysis.
• Simplified Geometry: They rely almost exclusively on 2D bounding boxes (bboxes), ignoring complex geometric shapes like polylines (roads, rivers) and polygons (parks, airports) essential for real-world GIS tasks.

This gap hinders the deployment of MLLMs in critical societal applications such as disaster response, urban planning, and precision agriculture, where precise interaction with georeferenced data is required.

2. Methodology: EarthSpatialBench

The authors propose EarthSpatialBench, a comprehensive benchmark designed to evaluate MLLMs on Earth imagery through a multi-dimensional framework.

A. Dataset Construction

• Source: Derived from SatlasPretrain, a large-scale remote sensing dataset.
• Scale: Contains 325,000 question-answer (QA) pairs across 21,000 high-resolution (1m ground resolution) 512x512 RGB images.
• Geometric Diversity: Unlike previous datasets, it incorporates three distinct geometry types:
  • Bounding Boxes: For small objects (e.g., buildings, parking lots).
  • Polylines: For linear features (e.g., roads, rivers, railways).
  • Polygons: For areal regions (e.g., parks, stadiums, agricultural fields).
• Ground Truth Generation: Spatial relations are computed using standard GIS procedures:
  • Distance: Euclidean metrics and buffer queries.
  • Direction: Azimuth angles discretized into 8-direction bins.
  • Topology: Computed using the DE-9IM (Dimensionally Extended nine-Intersection Model) standard for topological relations (e.g., "within," "intersect").

B. Evaluation Dimensions

The benchmark evaluates models across four orthogonal dimensions:

1. Spatial Relations: Distance, Direction, and Topology.
2. Geometry Types: Combinations of BBox, Polyline, and Polygon.
3. Object Representations:
   • Textual: Descriptions based on semantics or spatial context (e.g., "the northernmost building").
   • Visual Overlays: Objects highlighted directly on the image.
   • Explicit Coordinates: Precise geometric inputs (e.g., specific bbox or polygon coordinates).
4. Question Formats:
   • Choice-based: Categorical answers (Yes/No, Cardinal directions).
   • Quantitative: Numerical outputs (exact distance, angle, counts).
   • Localization: Identifying specific object coordinates satisfying constraints.

C. Quality Control

A multi-stage pipeline ensures data reliability, including automated checks for geometric validity (non-zero area/length), noise filtering, and selective manual inspection of complex scenes (e.g., high object density or boundary intersections).

3. Key Contributions

1. First Comprehensive Benchmark: EarthSpatialBench is the first dataset to systematically integrate quantitative distance/direction reasoning, topological relations, and complex geometries (polylines/polygons) for Earth imagery.
2. Rich Geometric Modeling: It moves beyond simple bounding boxes to model real-world geospatial entities, enabling evaluation of both object-level and region-level reasoning.
3. Systematic Evaluation Framework: The benchmark provides a structured way to test MLLMs on grounding (locating objects) versus reasoning (calculating relations), revealing specific failure modes.

4. Experimental Results

The authors evaluated state-of-the-art open-source models (Qwen3-VL, InternVL3.5) and proprietary models (GPT-5, Gemini 2.5 Pro, Claude 3.5 Sonnet).

• Performance Gaps: Significant disparities exist across reasoning types. Models generally perform better on classification (choice-based) tasks than on quantitative (distance/angle) or localization tasks.
• Grounding vs. Reasoning: There is a clear decoupling between high-level reasoning and low-level visual grounding. Models like GPT-5 and Claude achieve high accuracy in classification but fail significantly in localization metrics (IoU), indicating they cannot reliably map abstract reasoning to precise image regions.
• Impact of Representation:
  • Visual Overlays: Generally improve localization performance for open-source models (Qwen) but have mixed effects on proprietary models.
  • Coordinate Input: Providing explicit coordinates significantly reduces error in angle estimation compared to textual descriptions, highlighting the difficulty of grounding linguistic references to spatial locations.
• Geometry Sensitivity: Performance varies by geometry type. Polyline-Polyline queries (e.g., road-to-river) often yield lower errors than BBox-BBox queries, suggesting linear structures provide clearer cues. However, Polygon-Polyline interactions remain the most challenging.
• Chain-of-Thought (CoT): CoT prompting improves classification and regression tasks but often yields limited or negative gains for localization tasks, suggesting that explicit reasoning traces do not automatically translate to better spatial grounding.

5. Significance and Future Work

• Societal Impact: The benchmark addresses the need for AI systems capable of assisting in critical scenarios like disaster response (e.g., locating damaged houses within specific distance ranges of roads) and urban planning.
• Research Direction: The results highlight that current MLLMs lack the ability to jointly reason over precise geometry, numerical relations, and visual evidence. Future work should focus on:
  • Integrating explicit geometric reasoning mechanisms.
  • Improving the alignment between visual grounding and symbolic reasoning.
  • Extending benchmarks to multi-temporal and multi-modal data (e.g., SAR, LiDAR, DEM).

In conclusion, EarthSpatialBench establishes a rigorous standard for measuring spatial intelligence in geospatial contexts, revealing that while MLLMs are improving, they remain far from robust enough for autonomous, precise geospatial analysis.