💻 computer science

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

This paper introduces IDEAL-Bench, a novel evaluation suite based on a procedurally generated dataset of photorealistic indoor scenes that assesses Vision-Language Models' ability to perform holistic 3D layout inference, revealing that current models struggle with geometric reasoning despite strong object recognition and highlighting the need for benchmarks that distinguish genuine spatial understanding from linguistic approximation.

Original authors: Yuening Cai, Junwei Zhou, Youran Qu, Yu-Wing Tai

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuening Cai, Junwei Zhou, Youran Qu, Yu-Wing Tai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photo of a messy living room. You ask a smart AI, "How many chairs are there?" or "What's near the door?" The AI answers correctly: "Two chairs, and a lamp near the door." It sounds smart, right?

But here is the catch: The AI might be guessing the answers without actually "seeing" the room in 3D. It might know that chairs usually come in pairs, or that lamps often go near doors, without truly understanding where those objects are, how big they are, or how they are tilted.

This is the problem the paper IDEAL-Bench tries to solve.

The Problem: "Describing" vs. "Measuring"

The authors compare current AI testing to a game of "Guess the Answer."

Old Way (QA Benchmarks): You ask, "Is the chair to the left of the table?" The AI says "Yes." It gets the point, but it might just be guessing based on language patterns. It's like a student who memorized the answer key but doesn't understand the math.
New Way (IDEAL-Bench): Instead of asking questions, the researchers ask the AI to draw a blueprint of the entire room from just one photo. The AI must output a list of every object with its exact coordinates (where it is), dimensions (how big it is), and rotation (which way it's facing).

The Test: The "Architect's Exam"

To test this, the researchers built a special playground called IDEAL-Scenes.

Think of this as a digital Lego factory. They used a computer program to build 1,000 perfect, realistic rooms (bedrooms, offices, kitchens, etc.).
Because they built these rooms on a computer, they know the exact truth. They know the bed is exactly 2 meters long and sits at coordinate (0, 5, 0).
They showed a single photo of these rooms to 15 different AI models (like GPT-4o, Gemini, Claude, etc.) and asked them to produce the "blueprint."

How They Graded the AI

The researchers used two ways to grade the AI's blueprints:

The Math Check (Numerical Metrics): They compared the AI's numbers against the perfect computer truth.
- Did the AI say the bed was 2 meters long when it was actually 1 meter?
- Did it place the chair inside the wall?
- Did it get the rotation wrong?
The "Render-and-Compare" Check (Perceptual Metrics): This is the clever part. The researchers took the AI's blueprint and used it to re-build the room on the computer. Then, they showed the original photo side-by-side with the AI's re-built version.
- If the AI got the layout wrong, the re-built room would look weird (floating furniture, chairs inside walls, or the whole room shifted).
- They even used another AI (a "Judge") to look at the two images and say, "Do these look like the same room?"

The Shocking Results

The paper found that even the smartest AI models are terrible at this specific task.

The "Eye" vs. The "Ruler": The AIs are great at recognizing objects (they can tell a chair from a table). But they are terrible at measuring them. It's like having a painter who can perfectly copy the colors of a scene but gets the perspective and size completely wrong.
The Score: The best model (Gemini 2.5 Pro) only scored 62.1 out of 100. That means even the "smartest" AI is failing to truly understand the 3D layout of a room.
The "Grid" Illusion: In rooms with repeating patterns (like a classroom with many desks), the AIs got high scores. But the researchers found they were just copying the pattern (e.g., "desks are in rows") without actually knowing where the desks were in the room. They were faking the structure.

The Big Takeaway

The paper concludes that current AI models are "describing" scenes, not "measuring" them.

They can tell you what is in a room because they have read millions of books about rooms. But they cannot tell you exactly where things are or how big they are because they lack a true, internal 3D map of the world. IDEAL-Bench is a new tool designed to expose this weakness, showing us that before AI can truly navigate our physical world (like driving a car or helping in a hospital), it needs to learn how to stop guessing and start measuring.

Technical Summary: IDEAL-Bench

Problem Statement

Current evaluation paradigms for Vision-Language Models (VLMs) rely heavily on Spatial Question Answering (QA) and object grounding. While these methods effectively test relational reasoning and directional understanding, they fail to evaluate holistic 3D layout inference. QA benchmarks allow models to answer correctly by leveraging local visual cues or statistical priors without constructing a coherent, globally consistent 3D representation of a scene. Consequently, a critical axis of spatial competence—predicting the precise 3D pose (position, orientation, and scale) of every visible object from a single image—remains under-evaluated. This gap is significant for applications requiring physical interaction, such as augmented reality anchoring and robotic navigation, where "describing" a scene is insufficient without "measuring" it.

Methodology

1. IDEAL-Scenes Dataset

To enable rigorous evaluation, the authors introduce IDEAL-Scenes, a synthetic dataset of 1,000 indoor scenes across 10 room types (e.g., bedrooms, offices, classrooms).

Generation: Built upon a modified version of the InfiniGen engine, ensuring single-rectangular room geometries, standardized lighting, and physically plausible furniture placement.
Ground Truth: Each scene includes rendered RGB images, instance segmentation maps, and, crucially, re-renderable Blender source files with programmatically extracted ground-truth layouts. This allows for exact numerical comparison and perceptual "render-and-compare" evaluation.
Diversity: The dataset covers both irregular "living space" layouts and highly structured "grid" layouts (e.g., classrooms with repetitive furniture) to test generalization.

2. Task Formulation

The task requires a VLM to ingest a single RGB image, a list of candidate object categories, and candidate room types, then output a structured JSON prediction ( $\hat{S}$ ) containing:

Predicted room type and dimensions ( $W, D, H$ ).
For each visible object: category, 3D center coordinates ( $x, y, z$ ), bounding box dimensions ( $w, d, h$ ), and yaw rotation (discretized to cardinal directions for most objects).
Constraints: Models receive no depth maps, camera intrinsics, or extrinsics; they must reason purely from monocular appearance.

3. Evaluation Protocol (IDEAL-Bench)

The benchmark employs a dual-pathway evaluation strategy to ensure both numerical precision and perceptual validity:

Numerical Metrics: Predicted layouts are aligned to the ground truth and scored across five dimensions:
- D1: Scene Validity: Parsing success and room type accuracy.
- D2: Physical Plausibility: Non-overlap rate and in-bounds rate.
- D3: Geometric Accuracy: Position accuracy, Intersection-over-Union (IoU) precision, and rotation accuracy.
- D4: Object Recognition: Recognition rate and non-hallucination rate.
- D5: Grid Layout: Specific metrics for grid-structured scenes (Grid Coverage Rate and Grid Self-consistency Rate) to avoid ill-posed per-instance matching.
Perceptual Evaluation: Predicted layouts are used to re-render the original scene assets from the original camera viewpoint. A "Judge VLM" (Claude Opus 4.7) compares these reconstructions against the ground-truth images, assigning a Likert score based on spatial similarity. This protocol is validated against human judgments, showing a strong correlation ( $\rho = 0.79$ ).

Key Results

The authors evaluated 15 prominent VLMs (ranging from 7B to 235B parameters, including proprietary and open-source models) on the benchmark.

Task Difficulty: The task remains substantially unsolved. The best-performing model, Gemini 2.5 Pro, achieved an overall score of only 62.1/100.
The "Describing vs. Measuring" Gap: A sharp asymmetry exists across all models. While Object Recognition (D4) and Scene Validity (D1) are high (often >80-90%), Geometric Accuracy (D3) is extremely low. For instance, even the leader achieved only 25.5% precision on IoU@0.15 and 10.8% on Position Accuracy (@0.3m). This indicates models can identify objects but cannot accurately measure their 3D extent or location.
Scaling and Architecture: Performance does not scale monotonically with parameter count or release date.
- Smaller models (e.g., Gemma-4-31B) outperformed larger proprietary models (e.g., GPT-5.4, Claude Sonnet 4.6) in some metrics.
- Within the Qwen3-VL family, the 8B model outperformed the 30B and 235B variants.
- Some models failed to parse the output format entirely, while others succeeded in formatting but failed in geometric reasoning.
Grid vs. Non-Grid: Models performed differently on grid-structured scenes (offices, classrooms). While many models could generate internal regularity (high Grid Self-consistency), they failed to anchor these patterns to the correct world coordinates (low Grid Coverage), suggesting they learn structural priors rather than performing genuine geometric inference.
Divergence from Existing Benchmarks: Model rankings on IDEAL-Bench diverge significantly from existing QA-based benchmarks. Models that excel at spatial QA often perform poorly on holistic layout inference, confirming that QA success does not imply global 3D understanding.

Significance and Claims

The paper positions IDEAL-Bench as a necessary diagnostic complement to existing spatial evaluation suites. Its primary contributions are:

New Evaluation Paradigm: It shifts the focus from relational queries to holistic 3D layout inference, requiring structured, geometrically verifiable estimations of object pose and extent.
Diagnostic Capability: The benchmark surfaces "hidden geometric deficiencies" that linguistic spatial reasoning (QA) masks. It demonstrates that current VLMs possess "describing eyes" but lack "measuring eyes."
Validation of Perceptual Metrics: By combining numerical metrics with a render-and-compare protocol validated by human correlation, the paper establishes a more rigorous standard for evaluating spatial intelligence than QA alone.
Call to Action: The findings suggest that bridging the gap between discrete token generation and continuous spatial signals is a critical open problem for next-generation VLMs, and that simply scaling parameters is not a sufficient solution for 3D spatial reasoning.

The authors conclude that IDEAL-Bench provides a falsifiable, structured interface for probing 3D understanding, revealing that the field of spatial intelligence in VLMs is still in its early stages regarding metric accuracy and global coherence.