🤖 AI

From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

This paper introduces CRISP, a novel diagnostic evaluation paradigm that utilizes metric 3D Scene Graphs and oracle interventions to decouple perception from reasoning, revealing that while proprietary models suffer from inaccurate metric estimation despite robust latent reasoning, open-source models are fundamentally limited by a lack of multi-hop compositional reasoning.

Original authors: Zhixing Li, Yinan Yu

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhixing Li, Yinan Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a robot's ability to navigate a room. You ask it, "Is the cup on the left or right of the book?" The robot answers, "Left," and gets it right. You might think, "Great! It understands space!"

But according to this paper, that robot might actually be a cheater. It didn't look at the room to figure out the answer; it just guessed based on how humans usually talk about cups and books. It's like a student who memorized the answer key without ever opening the textbook.

The authors of this paper, Li and Yu, built a new test called CRISP to catch these cheaters and see if robots truly "see" the 3D world or just "guess" the words.

The Problem: The "Illusion" of Intelligence

Current AI models (called Vision-Language Models) are great at recognizing objects. They can point to a "dog" or a "chair." But when it comes to spatial intelligence—understanding exactly where things are, how far apart they are, and how they fit together in 3D space—they often fail.

The paper argues that these models are suffering from hallucinations. They produce the correct words, but their internal mental map of the room is completely wrong. They are "guessing correctly" using language tricks rather than "seeing correctly" using their eyes.

The Solution: The "CRISP" Test

To fix this, the researchers created a two-part test called CRISP (Consistency of Reasoning In Spatial Perception). Think of it as a "lie detector" for AI.

Instead of just asking the AI a question, CRISP forces the AI to do two things at once:

Answer the Question: (e.g., "How far is the chair from the wall?")
Draw the Map: The AI must build a 3D Scene Graph. This is like a blueprint or a skeleton of the room, where it has to write down the exact size of every object and the precise distance between them.

The Magic Trick: The researchers then compare the two.

If the AI says "The chair is 2 meters away" in its answer, but draws a map showing the chair is 10 meters away, it failed.
This proves the AI wasn't actually using the image to answer; it was just guessing the answer based on language patterns.

What They Discovered

When they ran this test on the smartest AI models available (both expensive "proprietary" ones and free "open-source" ones), they found three main types of failures:

The "Semantic Shortcut" Cheaters (Mostly Open-Source Models):
These models give the right answer to the question but draw a terrible, nonsensical map. They are like a person who knows the answer to a riddle because they've heard it before, but they can't actually visualize the scene. They rely on "language priors" (guessing what humans usually say) rather than looking at the picture.
The "Disconnected" Brains (Mostly Proprietary Models):
These are the most surprising findings. The expensive models (like Gemini and GPT-5) are actually very good at drawing the map! They can estimate distances and sizes pretty well. However, when they answer the question, they ignore their own map. It's like a brilliant architect who draws a perfect blueprint but then, when asked where the door is, gives a random guess because they forgot to look at their own drawing. They have a powerful "reasoning engine" that isn't connected to their "vision."
The "Consistent" Winners:
A few models managed to get both the map and the answer right. These are the only ones showing true spatial intelligence.

The "Oracle" Experiment: Do They Have the Brains?

To prove that the "Disconnected" models actually have the reasoning power, the researchers did a special experiment. They took the "cheating" models and gave them a perfect, pre-drawn map (like giving a student a cheat sheet with the correct blueprint) along with the picture.

The Result: The models' performance skyrocketed. Suddenly, they could answer complex questions perfectly.

The Conclusion: This proved that the AI models already have the logic and reasoning skills to solve these problems. The problem isn't that they are "stupid" or lack logic; the problem is that they can't connect their eyes (perception) to their brain (reasoning). They are failing to trust their own visual data.

Summary

The paper concludes that we have been tricked by AI models that look smart because they guess the right words. To build robots that can truly interact with the physical world (like a robot arm picking up a glass), we need to stop just asking questions and start checking if the AI's internal "mental map" matches its answers.

The path forward isn't just making the AI bigger; it's forcing the AI to be consistent—to make sure its reasoning is always grounded in what it actually sees, not just what it thinks it should say.

Technical Summary: From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

Problem Statement

Current evaluations of Vision-Language Models (VLMs) regarding visual spatial intelligence suffer from a critical conflation of language priors with genuine spatial reasoning. Traditional Question Answering (QA) benchmarks often allow models to exploit "semantic shortcuts"—relying on statistical co-occurrences in training data rather than perceiving the physical scene. Consequently, models may produce correct linguistic outputs while harboring flawed implicit 3D models, creating an "illusion of spatial intelligence."

Existing diagnostic tools, such as textual rationales or 2D cognitive maps, fail to provide the metric precision required for embodied AI. Furthermore, standard benchmarks do not decouple perceptual bottlenecks (the ability to see and measure) from reasoning deficits (the ability to logically manipulate spatial concepts). This disconnect prevents a granular understanding of whether a model's failure stems from inaccurate perception or an inability to utilize its own structural representations.

Methodology: The CRISP Benchmark

To address these limitations, the authors introduce CRISP (Consistency of Reasoning In Spatial Perception), a structural-diagnostic evaluation paradigm. CRISP shifts the focus from result-oriented correctness to process-oriented grounding by coupling discriminative QA with a generative 3D Scene Graph (SGC) task.

1. Dataset Construction

Source: The benchmark curates 1,162 high-quality static scenes (indoor and outdoor) from nuScenes and ScanNet++.
Ground Truth: A rigorous automated pipeline projects 3D annotations onto 2D images, filtering for visibility and resolution. Crucially, it constructs a Master 3D Scene Graph using unique numerical IDs rather than textual descriptions to prevent semantic leakage.
Scale: The dataset contains 9,839 questions covering seven spatial competencies: distance estimation, size estimation, directional perception, counting, spatial ranking, view transformation, and logical deduction.

2. Dual-Task Paradigm

CRISP evaluates models on two distinct but linked tasks:

Spatial QA: Models answer questions (Multiple Choice and Numerical) regarding the scene. This tests explicit reasoning.
3D Scene Graph Construction (SGC): Models are given a list of object IDs and must generate a star-topology subgraph. This requires predicting:
- Nodes: 3D dimensions and distance to the camera (metric estimation).
- Edges: Semantic directional predicates and metric Euclidean distances between objects.
- Note: The SGC task forces the model to externalize its implicit 3D modeling into a structured, metric-aware format, serving as a strict informational bottleneck.

3. Evaluation Metrics

QA Score: Accuracy for MCQs and Mean Relative Accuracy (MRA) for numerical questions.
SGC Score: A composite metric evaluating Metric Estimation ( $S_{est}$ ) via ratio-based IoU for size and relative accuracy for distance, and Relation Score ( $S_{rel}$ ) for semantic consistency of edge predicates.
Self-Consistency Score: A novel metric that verifies alignment between the two tasks. It uses a deterministic solver to derive a QA answer from the generated SGC ( $A_{derived}$ ) and compares it against the model's direct QA response ( $A_{QA}$ ). This distinguishes between "genuine understanding" and "consistent hallucination" (where a model is wrong in both tasks but consistently so).

4. Intervention Protocols

To disentangle perception from reasoning, the authors employ:

Blind Tests: Evaluating models with text-only inputs (numerical IDs only) to establish a baseline of language priors.
Structural Intervention: Providing models with either Ground-Truth (GT) 3D SGs or their own Predicted (Pred) SGs alongside the image. This tests the "ceiling" of latent reasoning and the model's robustness to modality conflict.

Key Results

1. The Perception-Reasoning Disconnect

Analysis of 13 state-of-the-art VLMs (including proprietary models like Gemini and GPT-5, and open-source models like Qwen and InternVL) reveals a systematic disconnect:

Proprietary Models: Possess robust latent reasoning engines but suffer from inaccurate metric estimation and a failure to leverage their own implicit structural representations. They often build reasonable structural models (moderate SGC scores) but fail to utilize them during end-to-end QA generation.
Open-Source Models: Often exhibit semantic shortcuts. They achieve higher QA scores by relying on language priors while failing to generate accurate metric structures (low SGC scores).
Consistency: A significant portion of models exhibit "consistent hallucination" (10-12% of instances), where they reason coherently based on fundamentally flawed geometric grounding.

2. The Illusion of Spatial Intelligence

Semantic Shortcuts: In blind text-only tests, models maintained near-perfect SGC format compliance (>99%) but relied on distributional priors. When visual input was added, models like Qwen2.5-VL showed significant QA gains but negligible SGC gains, indicating the vision encoder identified objects but failed to translate this into spatial location.
Specialized Models: Counter-intuitively, specialized spatial models (e.g., VG LLM, Cambrian-S) underperformed generalist baselines in consistency, suggesting that spatial fine-tuning may activate local cues without ensuring the reasoning engine utilizes them.

3. Unlocking Latent Reasoning via Intervention

Oracle Gain: When provided with Ground-Truth 3D SGs, models showed dramatic performance surges (e.g., +30.3% for Gemini 2.5 Flash). This confirms that robust latent reasoning capabilities already exist within current VLMs but are bottlenecked by perception.
Modality Conflict: When provided with their own Predicted SGs, most models suffered performance collapses, prioritizing their erroneous text-based externalizations over raw visual evidence. This highlights a "trust bias" and a lack of robustness in resolving modality conflict.
Reasoning Depth: Proprietary models showed significant gains in compositional logic (multi-hop reasoning) when given GT SGs, whereas open-source models primarily improved on atomic tasks, indicating a fundamental deficiency in their reasoning depth.

Significance and Claims

The paper claims that CRISP offers a rigorous roadmap for multi-modal alignment beyond end-to-end post-training. Its primary contributions are:

Diagnosis over Guessing: CRISP moves beyond "guessing correctly" via language priors to genuinely "perceiving, verifying, and reasoning." It provides a granular diagnosis that traditional black-box QA cannot afford.
Redefining Bottlenecks: The findings redefine the development roadmap for embodied AI. The primary bottleneck is identified not as an intrinsic logical deficiency in VLMs, but as a structural alignment failure between perception and reasoning.
Actionable Insights: The framework identifies three distinct failure modes for future development:
- Semantic Shortcuts: Requiring better vision encoders for metric perception.
- Perception-Reasoning Disconnect: Requiring a redesign of multimodal alignment strategies to bridge the internal utilization gap.
- Modality Trust Bias: Requiring upgrades to LLM backbones to improve conflict resolution and compositional reasoning.

Ultimately, the authors argue that scaling end-to-end training alone is insufficient. True visual spatial intelligence requires explicitly enforcing structural consistency, ensuring that logical deductions are strictly grounded in explicitly perceived 3D geometry.