Technical Summary: Towards Spatial Supersensing in the Wild (VSI-SUPER-WILD)

Problem Statement

Current Multimodal Large Language Models (MLLMs) have made significant strides in static image understanding and short-video comprehension. However, their ability to perform spatial supersensing—the capacity to construct, update, and predict an implicit 3D world model from continuous sensory streams—remains underexplored, particularly in real-world, unconstrained settings.

Existing benchmarks, such as VSI-SUPER, suffer from three primary limitations:

1. Limited Scene Diversity: They predominantly rely on household scenes, lacking the complexity of "in-the-wild" environments.
2. Artificial Data Construction: Long videos are often synthesized by concatenating random short clips with in-frame editing, failing to represent natural, continuous sensory streams.
3. Narrow World-State Coverage: Evaluation is heavily object-centric, neglecting the agent (observer) and environment (global layout and places), which are critical components of a unified world model inspired by cognitive science.

Consequently, it remains unclear whether current MLLMs can maintain coherent, recallable world states over long temporal horizons in diverse real-world scenarios.

Methodology: VSI-SUPER-WILD Benchmark

To address these gaps, the authors introduce VSI-SUPER-WILD, a large-scale benchmark designed to evaluate spatial supersensing across three anchors: the agent, objects, and the environment.

1. Dataset Construction

The benchmark is constructed from 442 real-world, long-form panoramic videos (totaling ~285 hours) sourced from YouTube, spanning 8 scene categories (e.g., street views, transportation hubs, commercial retail, medical facilities).

• Data Integrity: Unlike prior work, these videos are raw, unedited, and naturally continuous, with no generative editing or clip concatenation.
• Preprocessing: Panoramic videos are projected into four orthogonal perspective views (Front, Back, Left, Right).
• Metadata Generation: A semi-automatic pipeline with human-in-the-loop verification generates:
  • Temporal Metadata: Object appearance timestamps via human/MLLM proposals, YOLO-World filtering, and SAM3 instance segmentation.
  • Spatial Metadata: Time-varying camera poses estimated using Depth-Anything-3 (DA3).

2. Task Suite

The benchmark defines four cognitively grounded tasks to probe different aspects of world state:

• VMR (Motion Orientation Recall): Probes the agent. Models infer the camera's motion orientation relative to its viewing direction at a specific queried frame, requiring latent world state recall rather than frame-level matching.
• VPO (Place Temporal Ordering): Probes the environment. Models determine the temporal order of four query frames sampled from different places, where each frame is yaw-rotated to a different viewing direction to prevent trivial matching.
• VOO (Object Temporal Ordering): Probes objects. Models output the temporal order of specific objects based on their first or last appearance, requiring self-updating world modeling to distinguish occurrence types in a continuous stream.
• VOC (Continuous Object Counting): Integrates all anchors. Models count the unique instances of a queried object category throughout the video, testing persistent tracking and state updates.

3. Evaluation Protocol

The authors evaluated 13 mainstream MLLMs (including proprietary models like GPT-5.4 and Gemini-3.1-Pro, and open-source models like Cambrian-S and Qwen-VL series).

• Input: 128 frames uniformly sampled at 1080P (or streaming inputs for specific models).
• Metrics: Accuracy (%) for multiple-choice tasks (VMR, VPO, VOO) and Mean Relative Accuracy (MRA) for the counting task (VOC).

Key Results

The evaluation reveals a fundamental disconnect between current MLLM capabilities and the requirements of spatial supersensing in the wild:

1. Overall Performance: Models perform poorly, often near random chance. The best-performing model, Gemini-3.1-Pro, achieved an overall score of 44.36%, while GPT-5.4 scored 34.52%. Open-source models generally scored lower, with the best (Cambrian-S-7B) at 34.22%.
2. Task Disparities: Performance varies significantly by world-state anchor. Models are relatively more reliable on object-centric queries (VOO) but degrade substantially on agent-centric (VMR) and environment-centric (VPO) tasks, which require stronger 3D spatial cognition.
3. Temporal Horizon Sensitivity: Performance degrades as video duration increases. The average score for open-source models dropped from 35.0% in the 0–10 minute bin to 26.3% in the 120+ minute bin, indicating an inability to sustain consistent world-state updates over extended streams.
4. Robustness: Under the "Shuffle24" evaluation (testing stability against answer option permutations), VSI-SUPER-WILD showed higher option-order bias and lower Top-K/Pass@K scores compared to VSI-SUPER, confirming its increased difficulty and the instability of current models.

Failure Modes

Qualitative and quantitative analyses identified four recurring failure modes:

1. Spatial Collapse: Models fail to maintain a coherent 3D world state across viewpoint changes. Performance on VPO drops significantly when frames are rotated, suggesting reliance on view-specific 2D matching rather than implicit 3D modeling.
2. Semantic Shortcuts: Models rely on frame-level semantic cues (e.g., recognizing a "walking street" scene) to infer motion, rather than analyzing spatiotemporal displacement. This leads to plausible but incorrect motion orientation predictions.
3. Insufficient Update: Models struggle to update world states as new evidence arrives. Accuracy on "last-occurrence" ordering (VOO) is significantly lower than "first-occurrence," indicating a failure to integrate later temporal evidence into the world model.
4. Instance Confusion: Under real-world visual noise (motion blur, partial occlusion), models fail to track object identity robustly, leading to incorrect temporal ordering and counting.

Significance and Claims

The paper claims that VSI-SUPER-WILD establishes a rigorous standard for evaluating spatial supersensing by moving beyond synthetic, object-centric, and short-horizon evaluations to real-world, diverse, and long-horizon scenarios.

The authors conclude that current MLLMs lack the mechanisms to bind agents, objects, and environments into a unified spatial world model. The observed failures suggest that progress in this field hinges on developing representations that move beyond frame-level semantics to support stable, queryable, and coherent spatiotemporal world modeling capable of handling long-horizon, continually updated streams. The benchmark serves as a diagnostic tool to identify these specific gaps and guide future research toward true world modeling capabilities.