Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models

This paper introduces Spatial-DISE, a unified benchmark and large-scale dataset grounded in a four-quadrant taxonomy of spatial reasoning, which reveals significant gaps between current Vision-Language Models and human competence while providing a scalable framework for advancing human-like spatial intelligence.

The Gist

Imagine you are trying to teach a robot how to navigate a messy living room, fold a map, or assemble IKEA furniture without instructions. You'd need it to not just see the objects, but to understand how they fit together, how they move, and how they change shape.

This paper, Spatial-DISE, is a report card for the current generation of "Vision-Language Models" (AI that can see and talk). The authors from the University of Liverpool are saying: "These AI models are great at describing what they see, but they are terrible at figuring out how things work in 3D space."

Here is the breakdown using simple analogies:

1. The Problem: The "Flat-Earth" AI

Current AI models are like tourists who have only ever looked at postcards. They can tell you, "That's a red car next to a blue tree." But if you ask them, "If I fold this paper in half and poke a hole, what will the pattern look like when I unfold it?" they get confused.

Existing tests for these AIs were too easy. They mostly asked static questions like, "Is the cup to the left of the mug?" The paper argues this is like testing a pilot only on a calm, flat runway and never asking them to handle turbulence or a storm. We need to test their ability to mentally rotate, fold, and combine objects.

2. The Solution: The "Spatial-DISE" Gym

The authors built a new, tougher gym called Spatial-DISE. They created a massive set of puzzles (over 12,000 of them!) to test four specific types of "spatial muscles":

• Intrinsic-Static (The Puzzle Piece): Looking at a single object and understanding its fixed parts. Example: "Which face is on the back of this cube?"
• Intrinsic-Dynamic (The Mental Gymnast): Taking a single object and mentally twisting or folding it. Example: "If I rotate this block 90 degrees, what does it look like?"
• Extrinsic-Static (The Photographer): Understanding how objects sit next to each other in a fixed scene. Example: "Is the cat under the table?"
• Extrinsic-Dynamic (The Choreographer): Understanding how multiple objects move and interact. Example: "If I stack these blocks in this order, will the tower fall?"

The Cool Part: They didn't just hire humans to draw these puzzles. They built a robot factory (using a 3D software called Blender) that automatically generates thousands of unique, mathematically perfect puzzles. This ensures the answers are 100% correct and the AI can't cheat by memorizing previous answers.

3. The Results: The AI vs. Human Showdown

They tested 32 of the smartest AI models in the world (including GPT-4o, Gemini, and Claude) against this new gym.

• The Score: The average AI scored about 28%. Since there are usually 4 options, random guessing would get you 25%. So, the AIs are barely doing better than a monkey throwing darts.
• The Human Score: Humans scored around 77%.
• The Gap: There is a massive canyon between human intelligence and current AI when it comes to "spatial reasoning."

The Surprise: Interestingly, some AIs got better at the hardest, most dynamic puzzles (like combining 3D shapes) than the static ones. It turns out, some AIs are like calculators: they are bad at "feeling" the shape, but if you force them to calculate the edges and angles mathematically, they can sometimes outperform humans who rely on intuition.

4. Why Do They Fail? (The Autopsy)

The authors looked at where the AI went wrong. It wasn't because the AI couldn't "see" the picture (it wasn't a vision problem). The problem was thinking.

• Rule Breakers: The AI didn't know the basic laws of geometry. For example, it thought two opposite sides of a cube could be touching.
• Bad Memory: In "Fold and Punch" tasks, the AI lost track of how many layers of paper existed after folding. It's like trying to juggle while forgetting how many balls you started with.
• Superficial Glances: The AI would look at a shape and say, "It looks similar," without checking the tiny details that actually mattered.

5. The Takeaway

This paper is a wake-up call. If we want AI to drive cars, build robots, or help in surgery, they need to stop just "looking" and start "thinking" in 3D.

The authors have released their "gym" (the dataset and code) to the public. They hope that by training AI on these 12,000+ tough puzzles, we can finally teach machines to understand the physical world the way humans do: not just as a flat image, but as a dynamic, manipulatable space.

In short: We taught AI to read the menu, but we haven't taught it how to cook the meal yet. Spatial-DISE is the new cooking class.

Technical Summary

1. Problem Statement

Current Vision-Language Models (VLMs) excel at tasks like object detection and scene captioning but struggle significantly with sophisticated dynamic spatial reasoning, a critical capability for robotics, augmented reality, and autonomous navigation. Existing benchmarks suffer from three major limitations:

• Lack of Cognitive Framework: They lack a systematic taxonomy, leading to fragmented tasks that focus on basic skills rather than deep cognitive abilities.
• Static Bias: Most benchmarks test static spatial relationships (e.g., "is the cup to the left of the mug") rather than dynamic mental transformations (e.g., mental rotation, folding).
• Data Scarcity: Benchmarks addressing dynamic tasks are often small in scale, insufficient for robust evaluation or training, and lack verifiable ground truth.

2. Methodology

A. Theoretical Framework: The DISE Taxonomy

The authors propose a unified 2x2 cognitive taxonomy based on cognitive science (Maier, 1996; Uttal et al., 2013) to categorize spatial reasoning into four quadrants:

1. Intrinsic-Static (I-S): Analyzing fixed internal properties of a single object (e.g., 2D/3D Shape Finding).
2. Intrinsic-Dynamic (I-D): Mentally simulating transformations on a single object (e.g., 2D/3D Rotation, Folding, Fold-and-Punch).
3. Extrinsic-Static (E-S): Understanding fixed spatial relations between multiple objects (e.g., 3D Projection).
4. Extrinsic-Dynamic (E-D): Reasoning about changing relationships between multiple objects (e.g., 2D/3D Combination/Assembly).

B. Dataset Construction

To address data scarcity and ensure verifiability, the authors developed a scalable, automated synthetic data generation pipeline using Blender:

• Initialization & Seeding: Uses reproducible random seeds for every instance.
• Core Asset Generation: Creates complex 3D shapes, textures, and nets programmatically.
• Question & Answer Rendering: Renders optimal camera perspectives for questions and correct answers.
• Systematic Distractor Generation: Implements tiered strategies to create "near-miss" distractors, including geometric variations, pattern/orientation errors, incorrect views, and component replacements.
• Quality Control: A three-stage pipeline involving real-world data collection (from psychometric tests), synthetic generation, and rigorous human verification (checking for solution uniqueness, clarity, and redundancy).

The resulting datasets are:

• Spatial-DISE Bench: 559 VQA pairs covering all 10 task types and 4 DISE quadrants for evaluation.
• Spatial-DISE-12K: Over 12,000 verified VQA pairs focused on 3D tasks, intended for training and fine-tuning.

C. Evaluation Protocol

• Models: Evaluated 32 state-of-the-art VLMs, including proprietary models (GPT-4o, Claude 3.7, Gemini), open-source models (InternVL, Qwen, Llama), reasoning models (LLaVA-CoT, VLM-R1), and spatial-specialized models.
• Baselines: Compared against Random Guessing and a Human Baseline (54 participants, 1,679 responses, validated via Classical Test Theory and Item Response Theory).
• Fine-tuning: Conducted Supervised Fine-Tuning (SFT) with LoRA on Qwen2.5-VL-7B and SpaceOm using the Spatial-DISE-12K dataset to test transferability.

3. Key Contributions

1. Cognitively Grounded Taxonomy: Introduced the DISE framework, providing a unified structure to classify and diagnose specific weaknesses in spatial reasoning (e.g., distinguishing between static perception and dynamic simulation).
2. Scalable & Verifiable Pipeline: Developed an automated Blender-based pipeline that generates complex, verifiable 3D spatial reasoning tasks with controlled distractors, solving the data scarcity issue for dynamic reasoning.
3. Unified Benchmark & Dataset: Released Spatial-DISE Bench and Spatial-DISE-12K, the first large-scale, balanced benchmark covering all four cognitive quadrants of spatial reasoning.
4. Comprehensive Analysis: Defined the current performance ceiling of VLMs, revealing a universal gap between AI and human spatial intelligence, particularly in multi-step mental simulation.

4. Key Results

A. Performance Gap

• Universal Weakness: The average accuracy of 32 VLMs on the benchmark was 28.4%, barely above random chance (25%) and drastically lower than the human baseline (76.8%).
• Best Performer: The reasoning-enhanced Doubao-1.5-VL-thinking achieved the highest score (42.0%), yet still fell short of human performance.
• Dynamic vs. Static: Contrary to intuition, proficiency in static reasoning does not guarantee dynamic reasoning skills. Some models performed better on dynamic tasks than static ones, suggesting they rely on fragmented pattern recognition rather than a coherent spatial model.

B. Specific Failure Modes

• Multi-Step Transformations: Models fail catastrophically on tasks requiring sequential mental steps (e.g., Fold and Punch). Even the best model scored only 30.8% on this task, indicating a lack of "spatial working memory" to track state changes.
• Error Analysis: Analysis of 200 incorrect responses revealed that 72.5% were Reasoning Errors, not perceptual errors.
  • Failure in Rule Application (44.8%): Models ignore basic geometric axioms (e.g., adjacent vs. opposite faces on a cube).
  • Failure in Mental Simulation (40.0%): Inability to track object states through transformations.
  • Failure in Holistic-Local Processing (15.2%): Models get misled by superficial similarities while missing critical local details.

C. Fine-Tuning & Transferability

• Training Gains: Fine-tuning on Spatial-DISE-12K significantly improved performance (e.g., Qwen2.5-VL-7B rose from 26.1% to 47.0%).
• Quadrant Specialization: Training on one DISE quadrant did not generalize well to others, suggesting that spatial competencies are not factorized but rather distinct cohorts.
• Cross-Domain Transfer: 3D dynamic training showed positive transfer to 2D dynamic tasks but negative or neutral transfer to static tasks.
• Human Gap Persists: Even after fine-tuning, the best model (47.0%) remained far below the human baseline (76.8%), highlighting that current architectures lack the fundamental mechanisms for human-like spatial cognition.

5. Significance and Future Directions

• Diagnostic Tool: Spatial-DISE provides a robust framework to diagnose why models fail (e.g., lack of rule application vs. lack of simulation), moving beyond simple accuracy metrics.
• Research Direction: The paper argues that future VLM development must shift from passive visual perception to active, human-like reasoning. Key directions include:
  • Bridging the sim-to-real gap by transferring abstract cognitive concepts from synthetic data.
  • Moving from isolated puzzles to interactive, real-world tasks (navigation, manipulation).
  • Adopting process-oriented assessments that require textual justifications to verify true mental simulation.
• Resource: The release of the benchmark, dataset, and code provides the community with the necessary tools to systematically advance spatial intelligence in AI.

In conclusion, Spatial-DISE exposes a fundamental deficit in current VLMs: they can "see" but cannot "think" spatially. The benchmark establishes a new standard for evaluating and developing models capable of genuine spatial cognition.