ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

1. Problem Statement

The paper identifies a critical validity gap in current benchmarks for evaluating Visual Spatial Intelligence (VSI) in Vision-Language Models (VLMs). Specifically, widely used benchmarks like VSI-Bench suffer from two systemic flaws that render their evaluations unreliable under modern VLM settings:

1. Annotation-to-Video Ground-Truth Drift: Existing benchmarks repurpose point-cloud-based 3D annotations (originally designed for static 3D perception) for video-based evaluation. These annotations often contain errors due to imperfect 3D reconstructions, such as missing objects visible in raw video, mislabeled identities, or corrupted geometry (e.g., incorrect room sizes). Consequently, many Question-Answer (QA) pairs are factually incorrect or ambiguous when viewed against the actual video evidence.
2. Scene-Observability Mismatch: Traditional benchmarks often assume "full-scene" access (thousands of frames), whereas practical VLMs operate on sparsely sampled frames (e.g., 16–64 frames). This mismatch means that key objects queried in questions may be entirely absent from the model's input frames, making questions unanswerable or rendering the ground truth invalid for the specific input budget.

These issues lead to unreliable conclusions, where models may appear to perform well by exploiting dataset biases or hallucinating answers based on priors rather than visual evidence.

2. Methodology: ReVSI Benchmark Construction

The authors propose ReVSI, a rebuilt benchmark and evaluation protocol designed to ensure that every QA pair is answerable and correct under the model's actual visual inputs. The methodology involves three core pillars:

A. Video-Aligned Re-Annotation

• Scope: Re-annotated 381 scenes from 5 datasets (ScanNetv2, ScanNet++, ARKitScenes, 3RScan, MultiScan).
• Process: Used professional 3D annotation tools to manually verify and refine object labels, 3D bounding boxes, and room geometry directly against raw video frames.
• Improvements:
  • Open-Vocabulary: Expanded from 65 closed-set labels to 504 open-vocabulary labels (e.g., specific brands like "Sony PlayStation").
  • Geometry Correction: Manually corrected fragmented 3D meshes and missing objects.
  • Room Size: Replaced algorithmic Alpha Shape calculations with manual polygon annotation of floor boundaries to ensure accurate area measurements.

B. QA Regeneration with Bias Mitigation

• Template Refinement: Redesigned question templates to eliminate ambiguity (e.g., distinguishing between "room" and "scene" size, handling multi-room videos).
• Bias Reduction:
  • Counting: Reintroduced single-object counting (previously excluded) and added cumulative counting to prevent models from guessing "2" as a default.
  • Distance: Removed trivial short-range questions (<1m) that can be solved via 2D cues, focusing on long-range 3D reasoning.
  • Verification: Applied rigorous human verification to ensure every question is supported by the specific video evidence.

C. Frame-Aware Evaluation Protocols

• Adaptive Ground Truth: Unlike VSI-Bench, which uses a single ground truth for all inputs, ReVSI generates frame-adaptive QA pairs. For each sampling budget (16, 32, 64, or All frames), the system computes which objects are visible.
• Visibility Metadata: Provides fine-grained metadata on object visibility per frame, enabling controlled diagnostic experiments.
• Dummy Videos: Constructed "stress test" videos where frames containing the queried object are removed (Query-Drop), replaced with black frames, or repeated. This creates scenarios where the ground truth is definitively zero (for counting) or unknown, testing if models rely on visual evidence or memorized priors.

3. Key Contributions

1. ReVSI Benchmark: A high-fidelity, manually verified dataset with 381 scenes and 5,365 objects, featuring open-vocabulary labels and consistent video-ground-truth alignment.
2. Frame-Budgeted Evaluation: A protocol that aligns the evaluation ground truth with the specific number of frames provided to the model, eliminating the "unanswerable question" problem.
3. Diagnostic Tools: Introduction of dummy-video stress tests to quantify hallucination rates and distinguish between genuine spatial reasoning and reliance on scene priors.
4. Comprehensive Analysis: A systematic dissection of why existing benchmarks fail and how they distort the perceived capabilities of both proprietary and open-source models.

4. Key Results

Evaluations on ReVSI reveal significant discrepancies compared to VSI-Bench:

• Proprietary vs. Open-Source Models:
  • VSI-Bench: Suggested open-source models were competitive or superior to proprietary models.
  • ReVSI: Reveals that proprietary models (GPT-5.2, Gemini 3) consistently outperform open-source models. Open-source models suffer accuracy drops of up to 40% on tasks like object counting and relative direction, exposing their reliance on dataset biases rather than robust reasoning.
• Fine-Tuning Effectiveness:
  • Models fine-tuned on 3D spatial data (e.g., Spatial-MLLM, SpaceR) showed marginal or negative gains on ReVSI compared to their base models.
  • In some cases, fine-tuning led to performance degradation, suggesting that models learned incorrect patterns from the noisy annotations of previous benchmarks.
• Hallucination & Priors (Stress Tests):
  • Dummy Video Results: When queried objects were removed from the video (Ground Truth = 0), fine-tuned specialized models failed catastrophically, predicting non-zero counts (e.g., predicting 2 pillows when none were visible).
  • Zero-Shot Models: Proprietary models and some strong zero-shot open-source models (like Qwen3-VL) correctly predicted "0" in these scenarios, demonstrating better grounding in visual evidence.
  • InternVL3.5: Showed high hallucination rates, often predicting standard scene priors (e.g., "2") even on black videos, indicating severe overfitting to training data distributions.

5. Significance

• Correcting Evaluation Bias: The paper demonstrates that previous benchmarks were systematically under-assessing proprietary models and over-assessing open-source models due to annotation errors and frame-sampling mismatches.
• Grounding in Evidence: ReVSI proves that high benchmark scores do not guarantee true spatial reasoning; models can achieve high accuracy by memorizing scene priors. The new "dummy video" protocol is essential for detecting this behavior.
• Future Direction: The work highlights that data quality and supervision fidelity are more critical than data quantity. It calls for a shift in VLM development toward rigorous, frame-consistent evaluation to build truly robust 3D reasoning capabilities for real-world applications (e.g., robotics, AR/VR).

In summary, ReVSI provides a necessary "reality check" for the field, establishing a new standard for evaluating visual spatial intelligence that prioritizes input-consistency and ground-truth fidelity over convenience.