Technical Summary: SSMNBench

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated significant progress in single-image perception, their ability to reason about complex, cross-view human-centric scenes remains largely unverified. Existing multi-view benchmarks typically evaluate models using a fixed "bag of frames" and report a single end-to-end accuracy metric. This methodology conflates two distinct cognitive capabilities:

1. Robustness to Visual Distraction: The ability to identify a single informative view when redundant views are present.
2. Genuine Cross-View Fusion: The ability to integrate fragmented, complementary evidence across multiple views when no single view suffices.

Consequently, current benchmarks fail to diagnose whether model failures stem from distraction by redundant information or an inability to synthesize fragmented geometric cues. Furthermore, standard evaluations often reward models that rely on semantic priors or single unobstructed frames rather than performing true 3D spatial integration.

Methodology: SSMNBench

To address these gaps, the authors introduce SSMNBench, a diagnostic benchmark comprising 3,300 manually curated question-answer (QA) pairs across 11 diverse tasks. The benchmark is constructed through a rigorous pipeline involving data collection from dense, occlusion-heavy multi-view datasets (e.g., Core4D, M3GYM, Harmony4D), expert annotation, and structured distractor generation.

Core Evaluation Taxonomy

The central innovation of SSMNBench is the categorization of tasks into two distinct types based on the epistemic relationship between visual input and semantic understanding:

1. Single-View Sufficiency (SVS): Tasks where at least one view (the "Golden View") contains all necessary evidence to answer the question. These tasks test whether a model can identify and utilize an informative view without being distracted by additional redundant viewpoints.
   • Tasks: Fine-grained Action Recognition, Human-Human/Object Contact, Relative Self-Pose Comparison, Anatomical Joint Articulation.
2. Multi-View Necessity (MVN): Tasks where no single view contains sufficient evidence; the model must fuse complementary cues across views to resolve occlusions, depth ambiguities, or viewpoint-dependent appearances.
   • Tasks: Distinct Human Counting (with/without attributes), Relative Cross-Person Pose, Relative Distance Estimation, Global Orientation Identification, Global Depth Ordering.

Experimental Protocol

The benchmark employs a systematic view perturbation strategy to probe model robustness:

• SVS Settings: Normal (Golden View only), +1, +2, +3 (adding redundant views).
• MVN Settings: Normal (Minimal necessary subset), -1 (removing a critical view), +1, +2 (adding extra views).

Evaluation Metric: Distraction Decay ($\delta_{dis}$)

Beyond standard accuracy, the authors propose Distraction Decay ($\delta_{dis}$) to quantify a model's vulnerability to context saturation. This metric measures the average performance drop when additional (redundant) views are introduced. A high $\delta_{dis}$ indicates that the model's cross-attention is hijacked by irrelevant visual noise, while a low value suggests effective selective attention.

Key Results

The authors evaluated 17 state-of-the-art MLLMs (including proprietary models like GPT-5.2, Gemini-2.5, and open-source models like Qwen and InternVL). The findings reveal significant limitations in current architectures:

1. Performance Gap: A substantial gap exists between MLLMs and human perception. Human evaluators achieve >87% accuracy, while top proprietary models peak at only ~47.5% (SVS) and ~42.4% (MVN).
2. Universal Distraction Degradation: Contrary to the assumption that more views aid reasoning, adding redundant views consistently degrades performance in SVS tasks. Models struggle to isolate the "Golden View" from distractors.
3. Capacity Paradox: Higher-capacity models often exhibit greater vulnerability to visual distraction (higher $\delta_{dis}$) compared to smaller models. This suggests that parameter-rich models aggressively attempt to extract patterns from all visual tokens, making them more susceptible to irrelevant noise.
4. Failure to Fuse: In MVN tasks, models fail to integrate fragmented geometric evidence. Interestingly, in the "-1" setting (removing a view), some models perform as well as or better than in the full-context setting. This indicates an over-reliance on monocular priors from 2D pre-training; the introduction of conflicting multi-view data triggers representational conflicts that the models cannot resolve via geometric triangulation.
5. Resolution Sensitivity: High input resolution (1080p) is critical for fine-grained perception and distraction robustness. Compressing frames to maintain a fixed total token budget significantly increases distraction decay.

Failure Analysis

Manual error analysis on 660 instances categorizes failures into:

• Image-Level Errors (33.1% Spatial Analysis Failure): Models fail to infer 3D relationships, depth, or contact boundaries within single frames.
• View-Level Errors (67.2% Conflict Fusion Failure): Models detect conflicting cues across views but fail to synthesize them, often anchoring on a single "preferred" view and ignoring others, or hallucinating interactions due to poor boundary localization.

Significance and Contributions

The paper claims the following contributions to the field of multimodal AI:

• Novel Evaluation Taxonomy: SSMNBench is the first diagnostic framework to explicitly decouple a model's robustness to visual distraction (SVS) from its capacity for cross-view fusion (MVN).
• Comprehensive Benchmark: It provides a rigorous, expert-annotated dataset of 3,300 QA pairs focused on dense, occlusion-heavy, human-centric understanding, filling the gap between low-level geometric datasets and high-level semantic reasoning.
• Systematic Diagnostic Findings: By introducing controlled view perturbations and the $\delta_{dis}$ metric, the work reveals that current MLLMs rely on "multiple single-image semantic averaging" and view preferences rather than genuine 3D cross-view synthesis.

The authors conclude that SSMNBench serves as a rigorous diagnostic platform to expose fundamental vulnerabilities in current architectures, aiming to drive the development of future multimodal systems capable of true cross-view understanding in complex, real-world environments.