SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition

This paper introduces SpatialBench, a large-scale benchmark and hierarchical framework that evaluates multimodal large language models across five levels of spatial cognition, revealing their strong perceptual grounding but significant limitations in symbolic reasoning, causal inference, and planning compared to human capabilities.

The Gist

Imagine you are teaching a robot to navigate the world. You don't just want it to recognize that a "car" is a car; you want it to understand where the car is, how it moves, why it might crash, and what route to take to get home.

This paper, titled SpatialBench, is essentially a report card for the newest generation of "super-smart" AI models (called Multimodal Large Language Models, or MLLMs) on how well they understand space.

Here is the breakdown in simple terms, using some everyday analogies.

1. The Problem: The "Flat" Map

Previously, when we tested AI on spatial skills, it was like testing a student's geography knowledge with a single question: "Can you point to Paris on this map?"

If the AI got it right, we said, "Great, it knows geography!" But this ignored everything else: Can it read the traffic signs? Can it predict if a car will turn left? Can it plan a detour if there's a roadblock?

The authors argue that current AI benchmarks are too simple. They treat spatial intelligence as a flat list of tasks rather than a hierarchy of skills, like climbing a ladder.

2. The Solution: The "Spatial Ladder"

The researchers built a new framework called SpatialBench. Imagine a 5-story building where each floor represents a deeper level of understanding:

• Level 1: The Eyes (Observation)

  • What it is: "I see a red car and a blue box."
  • Analogy: A baby looking at a toy and naming it.
  • AI Status: Excellent. AI is great at spotting objects.

• Level 2: The Map (Topology & Relation)

  • What it is: "The red car is next to the blue box, and the box is inside the garage."
  • Analogy: Arranging furniture in a room. You know where things sit relative to each other.
  • AI Status: Good. AI can usually figure out where things are in relation to one another.

• Level 3: The Translator (Symbolic Reasoning)

  • What it is: "That arrow sign means 'One Way,' so I can't drive the other way."
  • Analogy: Reading a rulebook. You aren't just seeing a shape; you understand the meaning behind the symbol.
  • AI Status: Getting there, but shaky. AI sometimes misses the "rules" of the road.

• Level 4: The Detective (Causality)

  • What it is: "If that truck accelerates, it will hit the wall in 3 seconds."
  • Analogy: Predicting the future based on physics. "If I drop this glass, it will break."
  • AI Status: Struggling. AI often fails to predict how objects interact or move over time.

• Level 5: The Captain (Planning)

  • What it is: "To get out of this parking lot, I need to reverse, turn left, then drive straight to the exit."
  • Analogy: Being the captain of a ship. You combine all the previous skills to make a complex plan to reach a goal.
  • AI Status: Very Weak. This is the hardest part. AI often gets lost in the details and forgets the ultimate goal.

3. The Test: A Real-World Driving Simulator

To test these levels, the team didn't use fake computer graphics. They went out into the real world with a camera and a laser scanner (LiDAR).

• They recorded 50 real videos of people walking through offices, forests, and parking lots.
• They created 1,347 questions based on these videos.
• Example Question: "If the white car turns right and goes straight, which parking spot will it pass?"

They tested dozens of the world's most famous AI models (like GPT-5, Gemini, and Claude) against these questions.

4. The Results: The "Smart but Clueless" Robot

The results were surprising and humbling:

• The Good: The AI models are like super-photographers. They can count objects, measure distances, and describe a scene perfectly.
• The Bad: When asked to think ahead or plan a route, they often fail.
  • The "Hallucination" Problem: When humans look at a scene, we focus on the goal (e.g., "I need to get to the exit"). The AI, however, tends to look at everything equally. It gets distracted by a bird, a sign, or a shadow, and loses track of the main path.
  • The "Perspective" Problem: Humans understand that "left" depends on which way I am facing. The AI often gets confused between "what the camera sees" and "what the robot should do."

The Human Gap:
When humans took the test, they scored nearly 100%. They could easily predict cause-and-effect and plan routes. The best AI models scored around 70-75% on the easy stuff, but dropped to 20-30% on the hard planning stuff.

5. The Takeaway

The paper concludes that while AI has learned to see the world very well, it hasn't yet learned to understand the world like a human does.

• Current AI: "I see a car, a tree, and a sign. Here is a description of them."
• Human Intelligence: "The car is moving fast toward the tree. If I don't turn left now, I will crash. So, I will turn left."

The Bottom Line: We have built AI that is a brilliant observer, but we still need to teach it how to be a strategic thinker. SpatialBench is the new ruler we will use to measure how close we get to that goal.

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in visual understanding, existing benchmarks for spatial cognition suffer from critical limitations:

• Oversimplification: Current benchmarks often reduce spatial intelligence to single-dimensional metrics (e.g., object detection or simple distance estimation), failing to capture the hierarchical and interdependent nature of spatial abilities.
• Fragmentation: Evaluations are typically task-oriented (e.g., "find the object") rather than ability-oriented, lacking a unified framework to assess the progression from perception to reasoning.
• Data Limitations: Many datasets rely on synthetic environments or narrow indoor scenes, lacking the visual diversity, dynamic complexity, and real-world variability necessary to test genuine spatial reasoning.
• Evaluation Gap: There is a lack of systematic tools to diagnose why models fail at high-level spatial tasks (e.g., planning or causal inference) versus low-level perception.

2. Methodology

A. Hierarchical Spatial Cognition Framework

Inspired by Cognitive Map Theory from neuroscience, the authors propose a five-level hierarchical taxonomy to decompose spatial intelligence. This shifts evaluation from isolated tasks to a progressive cognitive ladder:

1. Observation (L1): Basic perception of objects, attributes (size, color), and absolute metrics (counting, distance).
2. Topology and Relation (L2): Understanding spatial relationships (adjacency, containment, relative direction, ordering) among entities.
3. Symbolic Reasoning (L3): Interpreting visual symbols (arrows, signs) and applying rule-based reasoning to infer constraints or intents.
4. Causality (L4): Predicting outcomes of actions or environmental changes based on spatiotemporal dependencies and physical intuition.
5. Planning (L5): Integrating perception, relations, and causality to formulate coherent, goal-oriented navigation strategies or action sequences.

B. SpatialBench Dataset Construction

To support this framework, the authors constructed SpatialBench, a large-scale, fine-grained benchmark:

• Data Source: 50 first-person (egocentric) videos captured using a custom platform with synchronized RGB cameras and 3D LiDAR sensors.
• Environment: Diverse real-world settings (indoor/outdoor, static/dynamic), including city streets, forests, and underground areas.
• Scale: 1,347 question-answer pairs mapped to 15 distinct task types aligned with the 5 cognitive levels.
• Annotation Process:
  • Hybrid Generation: Human annotators generate questions; AI models generate answers for non-metric tasks; LiDAR point clouds are used to compute ground-truth ground for metric tasks (size, distance) to ensure geometric precision.
  • Verification: A rigorous multi-step protocol involving consistency checks across multiple models and mandatory human verification for high-complexity tasks (L3–L5).

C. Evaluation Metrics

• Task-Specific Metrics: Accuracy (ACC) for multiple-choice questions; Mean Relative Accuracy (MRA) for numerical answers.
• Overall Score: A novel capability-oriented metric that aggregates performance across the five levels. It uses an optimization-based weighting scheme to assign adaptive weights to each level, ensuring higher weights for more complex cognitive tasks while maintaining monotonicity (higher levels are weighted more heavily).

3. Key Contributions

1. First Hierarchical Framework: Established the first systematic, cognitively grounded framework for evaluating MLLM spatial intelligence, moving beyond task-specific metrics to a structured ability hierarchy.
2. SpatialBench Dataset: Released a large-scale, real-world multimodal dataset with 15 task types and precise geometric ground truth (via LiDAR), addressing the lack of diversity in existing benchmarks.
3. Unified Evaluation Metric: Introduced a complexity-aware overall score that balances performance across different cognitive levels, allowing for fair comparison of models with varying architectures.
4. Human-Model Comparison: Conducted controlled human benchmarks to quantify the gap between artificial and human spatial intelligence, revealing specific failure modes in MLLMs.

4. Experimental Results

The authors evaluated a wide range of proprietary (e.g., Gemini-2.5-pro, GPT-5) and open-source models (e.g., Qwen, GLM, ERNIE) on SpatialBench.

• Performance Stratification:
  • Strong Perception: Models perform well on L1 (Observation) and L2 (Topology), showing high accuracy in object counting, size estimation, and simple relational queries.
  • High-Level Deficiencies: Performance drops sharply at L3 (Symbolic Reasoning), L4 (Causality), and L5 (Planning). Models struggle to convert visual inputs into symbolic rules, infer causal consequences, or generate multi-step plans.
  • Scale vs. Architecture: While larger models generally perform better, architectural design and the integration of visual-linguistic information play a crucial role. "Thinking" modes in some models showed mixed results compared to standard instruction-tuned versions.
• One-Shot Learning:
  • Providing a single example (one-shot) significantly boosted performance for some models (e.g., GPT-5, Qwen3-VL), suggesting they can leverage linguistic reasoning to compensate for spatial weaknesses.
  • However, top-tier models (Gemini-2.5-pro) performed best without prompting, indicating strong intrinsic spatial logic.
• Human Benchmark:
  • Humans achieved near-perfect scores (~96.40 overall), particularly excelling in L3–L5 tasks (100% accuracy in symbolic/causal/planning tasks).
  • Key Gap: Humans exhibit selective, goal-directed abstraction, focusing on relevant spatial cues. MLLMs tend to over-attend to surface details, lacking coherent spatial intent and failing to filter irrelevant information during reasoning.

5. Significance and Future Directions

• Diagnostic Tool: SpatialBench provides a granular lens to diagnose where and why MLLMs fail in spatial tasks, moving beyond "black box" accuracy scores.
• Research Direction: The results highlight that future MLLMs need to improve in symbolic abstraction, causal inference, and egocentric perspective tracking (distinguishing between camera view and agent view).
• Foundation for Embodied AI: By establishing a rigorous standard for spatial cognition, this work lays the groundwork for developing truly spatially intelligent systems capable of real-world navigation and interaction.
• Open Resource: The code and dataset are publicly available, encouraging community efforts to bridge the gap between current MLLMs and human-level spatial intelligence.

Conclusion: The paper demonstrates that while MLLMs have mastered basic visual perception, they currently lack the deep, hierarchical reasoning required for complex spatial planning and causal inference. SpatialBench serves as a critical stepping stone toward achieving robust, human-like spatial intelligence in AI.