Technical Summary: From Where Things Are to What They Are For (SFI-Bench)

Problem Statement

Current Multimodal Large Language Models (MLLMs) have demonstrated significant progress in low-level geometric perception and factual recall. However, existing benchmarks largely fail to evaluate the higher-order cognitive abilities required for grounded, agentic intelligence. Specifically, there is a gap in assessing whether models can move beyond recognizing "where things are" (spatial layout) to understanding "what they are for" (functional affordances and context-dependent utility). While prior works like VSI-Bench test geometric perception, they do not sufficiently probe the construction of coherent cognitive maps, the inference of object affordances, or the integration of external knowledge for goal-directed reasoning.

Methodology

The SFI-Bench Benchmark

The authors introduce the Spatial–Functional Intelligence Benchmark (SFI-Bench), a video-based evaluation suite designed to systematically measure advanced reasoning in MLLMs.

• Dataset Composition: SFI-Bench comprises 1,555 expert-annotated multiple-choice questions derived from 134 real-world egocentric indoor video scans (sourced from ARKitScenes and ScanNet++). These videos cover diverse residential, professional, and industrial environments.
• Task Taxonomy: The benchmark evaluates two complementary cognitive dimensions across six core tasks:
  1. Structured Spatial Reasoning:
     • Global & Conditional Counting: Requires logical inference over attributes and relations (e.g., counting the maximum number of same-brand bottles) rather than simple enumeration.
     • Cross-View Multi-hop Path Reasoning: Tests the ability to integrate spatial evidence across time and viewpoints to infer relationships not visible in a single frame.
     • Layout Inference: Evaluates the construction of a global scene layout and reasoning about occlusion relationships where referenced objects never appear together.
  2. Functional Reasoning:
     • Functional Association: Tests the inference of affordance relationships between objects (e.g., linking a remote to a specific TV) based on design or context.
     • Operation Planning: Requires determining how to operate a device, often necessitating the retrieval of external knowledge (e.g., user manuals).
     • Causal Hypothesis & Troubleshooting: Assesses the ability to diagnose problems by combining scene understanding with external knowledge sources.

Curation Pipeline

The dataset was constructed via a three-stage pipeline:

1. Automatic Generation: Using Gemini-2.5-Pro to extract fine-grained metadata (objects, attributes, spatial relations, functional roles) from videos, which are then converted into candidate questions using task-specific templates.
2. Human Verification: Expert annotators (trained ML engineers) verify questions, validate answers against visual evidence, and refine options to ensure cognitive complexity.
3. Quality Filtering: Automated and manual checks remove questions solvable without visual grounding. For knowledge-grounded tasks, answers are derived from expert-retrieved manuals.

Evaluation Setup

The authors evaluated a wide range of proprietary (e.g., GPT-5, Gemini-3, o4-mini) and open-source models (e.g., Qwen3-VL, InternVL, LLaVA-Video).

• Settings: Evaluations were conducted in a zero-shot setting.
• Tool Use: For functional tasks requiring external knowledge (Operation Planning, Troubleshooting), models with web search capabilities were allowed to retrieve information, while others were evaluated offline.
• Metrics: Performance was measured by answer accuracy on the multiple-choice questions.

Key Results

Model Performance

• Proprietary vs. Open-Source: Proprietary models generally outperform open-source counterparts. Among proprietary systems, reasoning-enabled variants (e.g., Gemini-3.1-Pro, GPT-5.4-High) show substantial gains, with Gemini-3.1-Pro achieving the highest average accuracy (73.8%).
• Spatial vs. Functional: While leading models exhibit strong capabilities in spatial cognitive map construction (e.g., Layout Inference), they struggle significantly with functional reasoning tasks.
• The Bottleneck: Global Conditional Counting emerged as a primary bottleneck across all models, revealing persistent limitations in compositional and logical reasoning.
• Tool Use Dependency: For knowledge-grounded tasks, models equipped with web search (e.g., GPT-5) significantly outperformed their offline counterparts. However, the authors note that tool use introduces noise; weaker reasoning models often performed worse with search tools than without them, suggesting that strong reasoning is a prerequisite for effective tool use.

Analysis of Reasoning Mechanisms

• Reasoning Length vs. Quality: The study found that longer reasoning chains do not correlate with higher accuracy. In fact, models often produce unnecessarily long chains that introduce semantic drift. "Overthinking" leads to errors, particularly in functional association tasks.
• Model Scale: Larger models (e.g., 235B parameters) tend to generate shorter, more effective reasoning chains compared to smaller models (e.g., 8B), which rely on verbose but shallow reasoning.
• Temporal Dependence: Shuffling video frame orders had minimal impact on model performance, indicating that current MLLMs rely on aggregated visual evidence rather than temporally coherent dynamics to form spatial representations.
• Visual Grounding: Performance dropped significantly when models were provided with structured textual captions instead of raw video, highlighting that direct visual grounding is critical for cognitive map construction.

Key Contributions

1. SFI-Bench: The introduction of a comprehensive benchmark that shifts the evaluation focus from perceptual recognition to structured spatial and functional reasoning.
2. Diagnostic Insights: The identification of specific failure modes in state-of-the-art MLLMs, including visual perception errors, spatial inconsistency (e.g., "teleportation" of objects), and affordance overgeneralization.
3. Reasoning Dynamics: The discovery that effective reasoning is defined by alignment with perceptual evidence rather than the length of the reasoning chain, and that current open-source reasoning models (RLVR-trained) fail to transfer mathematical reasoning capabilities to spatial-functional domains.
4. Knowledge Integration: Highlighting the critical role of external knowledge acquisition for agentic tasks, demonstrating that parametric memory alone is insufficient for operational and troubleshooting scenarios.

Significance

The paper argues that achieving truly grounded, agentic intelligence requires moving beyond perception-oriented models toward systems capable of integrated spatial–functional cognition. The authors posit that SFI-Bench serves as a diagnostic tool to measure progress in this direction. The findings suggest that future advancements must focus on:

• Building and maintaining coherent cognitive maps that are temporally consistent.
• Developing robust compositional reasoning that avoids semantic drift.
• Seamlessly integrating visual perception with dynamic, external knowledge sources to support purposeful action in real-world environments.

The work underscores that while current MLLMs excel at local perception, they remain brittle in maintaining global spatial memory and grounding affordances, representing a critical frontier for multimodal AI research.