Technical Summary: MMSI-Bench

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly deployed in physical environments, requiring robust spatial intelligence—the ability to understand object locations, motions, and relationships. However, existing benchmarks primarily evaluate single-image spatial understanding or rely on templated questions generated from metadata. These limitations fail to capture the complexity of real-world spatial reasoning, where models must integrate information across multiple viewpoints to track ego-motion, reconstruct scenes, and reason about entities that never co-occur in a single frame. Consequently, there is a lack of rigorous, diverse, and challenging evaluation standards for multi-image spatial reasoning.

Methodology

The authors introduce MMSI-Bench, a comprehensive benchmark designed to evaluate multi-image spatial intelligence. The construction and evaluation methodology involves:

1. Benchmark Construction

• Data Curation: Six 3D-vision researchers manually curated the dataset from over 120,000 images sourced from eight diverse real-world datasets, including indoor 3D scans (ScanNet, Matterport3D), autonomous driving footage (nuScenes, Waymo), robotics data (AgiBot-World), and egocentric videos (Ego4D).
• Annotation Process: Researchers selected image sets exhibiting non-trivial spatial relations and crafted 1,000 unique, unambiguous multiple-choice questions. No templates were used; all questions were created in free form to ensure diversity.
• Taxonomy: The benchmark categorizes tasks based on three spatial elements: Camera (observer), Object, and Region. It defines 11 task categories:
  • Positional Relationships: Camera-Camera, Camera-Object, Camera-Region, Object-Object, Object-Region, Region-Region.
  • Attributes: Measurement (geometric properties) and Appearance (visual characteristics).
  • Motion: Camera movement and Object movement.
  • Multi-Step Reasoning: Complex tasks chaining multiple reasoning steps (e.g., navigation paths).
• Reasoning Annotations: Each question is paired with a detailed, step-by-step human-annotated reasoning process to facilitate correctness verification and automated error analysis.
• Quality Control: Three independent reviewers audited all samples to remove ambiguities, incorrect answers, or questions solvable by single-image cues or common sense.

2. Evaluation Setup

• Models Evaluated: 37 MLLMs were tested, including proprietary models (e.g., GPT-5, o3, GPT-4.5, Gemini-2.5-Pro, Claude-3.7-Sonnet) and open-source models (e.g., Qwen2.5-VL, InternVL3, LLaVA-OneVision, NVILA).
• Baselines: Performance was compared against Random Guessing, Human Performance (5 independent evaluators), and a "Blind" GPT-4o baseline (text-only input).
• Metrics: Accuracy was measured via exact match between the model's extracted answer and the ground truth.
• Prompting Analysis: The study investigated the efficacy of linguistic prompting (Zero-Shot Chain-of-Thought) and visual prompting (using PATS to highlight image overlaps).

3. Automated Error Analysis

Leveraging the human-annotated reasoning chains, the authors developed an automated pipeline to diagnose model failures. Errors were categorized into four types:

1. Grounding Errors: Failure to correctly identify or localize objects/details.
2. Overlap-Matching and Scene-Reconstruction Errors: Failure to match corresponding points across images or reconstruct the underlying scene layout.
3. Situation-Transformation Reasoning Errors: Mistakes in converting between relative and absolute directions or reasoning from non-observer perspectives.
4. Spatial-Logic Errors: Hallucinations, incorrect transitivity application, or misuse of reference objects in motion reasoning.

Key Results

• Significant Performance Gap: There is a substantial gap between current MLLMs and human-level spatial intelligence. Humans achieved 97.2% accuracy, whereas the strongest proprietary model (GPT-5) reached only 41.9%, and the best open-source model (Qwen2.5-VL-72B) achieved 30.7%.
• Limitations of Scaling: Increasing model parameters yielded limited gains. For instance, Qwen2.5-VL-72B outperformed its 32B variant by only ~3%, and InternVL3-78B outperformed the 1B variant by just 1.5%.
• Prompting Ineffectiveness: Standard prompting techniques (CoT and visual overlap cues) provided minimal to no improvement for most models, suggesting that current architectures lack the fundamental spatial reasoning capabilities required to benefit from such cues.
• Fine-Tuning Limitations: Models fine-tuned on existing spatial-reasoning datasets showed only marginal improvements over their base counterparts, indicating that current training data (often template-based) fails to generalize to the diverse scenarios in MMSI-Bench.
• Error Distribution: The most prevalent failure mode across all models was Overlap-Matching and Scene-Reconstruction errors, indicating a fundamental struggle in linking visual information across different viewpoints.

Key Contributions

1. MMSI-Bench Dataset: A fully human-curated, template-free benchmark of 1,000 challenging multi-image spatial reasoning questions with detailed reasoning annotations.
2. Comprehensive Evaluation: A large-scale assessment of 37 state-of-the-art MLLMs, revealing a wide performance gap between current models and human capabilities.
3. Automated Error Analysis Pipeline: A novel method using annotated reasoning chains to systematically diagnose and categorize spatial reasoning failures, identifying specific bottlenecks like scene reconstruction.
4. Empirical Insights: Evidence suggesting that current progress in spatial intelligence is bottlenecked by data quality and diversity rather than model scale, and that standard prompting strategies are insufficient for this domain.

Significance

The paper positions MMSI-Bench as a critical resource for the community to drive progress toward Embodied Artificial General Intelligence (AGI). By highlighting the substantial gap between current MLLMs and human spatial intelligence, the benchmark underscores the need for new research directions in data curation, model architecture, and training paradigms. The authors argue that without benchmarks demanding genuine multi-image reasoning, reliable measurement and improvement of spatial competence in physical-world deployments remain impossible. The release of the dataset and the error analysis framework aims to provide actionable insights for developing more robust and spatially capable multimodal AI systems.