SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis

This paper introduces SceneEval, a comprehensive evaluation framework and benchmark (SceneEval-500) designed to assess text-conditioned 3D indoor scene synthesis by measuring both explicit user requirements and implicit plausibility expectations, revealing significant gaps in current generation methods.

The Gist

Imagine you are a homeowner who wants to hire a digital interior designer. You don't want to draw blueprints or pick 3D models yourself; you just want to say, "I want a cozy bedroom with a big blue bed, two nightstands, and a wardrobe tucked into the corner."

You type this into a computer, and it spits out a 3D room. But how do you know if the computer actually listened?

This is the problem SceneEval solves.

The Problem: The "Vibe Check" vs. The "Rule Book"

Until now, checking if a computer-generated room was good was like judging a painting by how much it looked like other famous paintings.

• The Old Way: Researchers would compare the new room to a database of "perfect" real rooms. If the new room looked statistically similar to the "perfect" ones, they gave it a high score.
• The Flaw: This doesn't care if you asked for a blue bed and got a red one. It also doesn't care if the bed is floating in mid-air or if the nightstands are inside the wall. It just checks if the room looks like a room.

The Solution: SceneEval (The Ultimate Inspector)

The authors created SceneEval, a new way to grade these 3D rooms. Think of it as a strict building inspector who checks two very different things:

1. The "Did You Listen?" Test (Fidelity)

This checks if the computer followed your specific instructions.

• The Count: You asked for two nightstands. Did it give you two, or three?
• The Details: You asked for a "blue" bed. Is it actually blue, or is it beige?
• The Placement: You said the wardrobe should be in the "corner." Is it actually in the corner, or is it blocking the door?

2. The "Does It Make Sense?" Test (Plausibility)

This checks the "unspoken rules" of how the world works. Humans know these rules without being told, but computers often forget them.

• No Ghosts: Are any objects floating in the air? (They shouldn't be).
• No Crashes: Are the chairs crashing into the table? (They shouldn't be).
• Walkability: If you were a human, could you walk through the room without getting stuck behind a sofa?
• Usability: Is the front of the TV facing the sofa so you can watch it? Or is it facing the wall?

The Tool: The "Recipe Book" (SceneEval-500)

To test these computers fairly, you need a standard test. The authors created SceneEval-500.

• Imagine a cookbook with 500 different recipes (text descriptions) for rooms.
• Some recipes are simple ("A room with a bed").
• Some are complex ("A messy teenager's room with a twin bed, a desk with a specific monitor, a beanbag, and a poster on the wall").
• Crucially, every recipe comes with a Answer Key. The authors wrote down exactly what the room should look like (e.g., "Bed count: 1," "Bed color: Blue," "Bed on floor: Yes").

When a computer generates a room, SceneEval compares the result against this Answer Key to give it a grade.

The Results: The Computers Are Still Learning

The authors tested six of the smartest AI room generators using this new system. Here is what they found:

• The Good: The computers are getting better at making rooms that look like rooms. They rarely make rooms that look like abstract art.
• The Bad: They are terrible at following specific instructions.
  • If you ask for a "red sofa," they might give you a "blue sofa."
  • If you ask for "two chairs," they might give you "three."
  • They struggle with relationships. If you say "put the lamp next to the bed," they might put the lamp on the ceiling or inside the bed.
• The Ugly: Some computers try to "cheat." One method put all the furniture outside the room to avoid collisions. It technically had no crashes, but it wasn't a usable room! SceneEval caught this because it checks if objects are actually inside the room.

The Big Picture

Think of 3D scene generation like teaching a child to cook.

• Old Metrics just asked, "Does this dish look like a photo of a burger?"
• SceneEval asks, "Did you use the right ingredients? Did you follow the recipe? And did you remember to turn on the stove so it's actually cooked?"

The paper concludes that while AI is amazing at creating "vibes," it still needs a lot of help to follow specific instructions and understand the physical rules of the world. SceneEval is the new ruler we need to measure how far we have to go before we can truly trust AI to design our dream homes.

Technical Summary

Here is a detailed technical summary of the paper "SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis."

1. Problem Statement

Despite rapid advancements in text-conditioned 3D indoor scene generation, there is a critical lack of systematic evaluation frameworks. Existing metrics suffer from two main limitations:

• Lack of Text Fidelity: Current metrics (e.g., FID, CLIP scores) primarily measure distributional similarity to ground-truth datasets or coarse text-image alignment. They fail to verify if specific constraints in the input text (e.g., "two red chairs," "sofa against the wall") are actually satisfied.
• Incomplete Plausibility Assessment: Existing evaluations often focus on isolated physical checks (e.g., collision detection) but ignore broader implicit expectations such as object support (gravity), navigability, and functional accessibility.
• Absence of Standardized Benchmarks: Prior work relies on ad-hoc text descriptions and qualitative user studies without fine-grained ground-truth annotations, making reproducible, quantitative comparison impossible.

2. Methodology

The authors propose SceneEval, a comprehensive evaluation framework, and SceneEval-500, a new benchmark dataset.

A. The Benchmark: SceneEval-500

• Composition: A dataset of 500 text descriptions of indoor scenes, categorized into three difficulty levels (Easy, Medium, Hard) based on object count and complexity.
• Annotation Schema: Each description is annotated with machine-checkable ground-truth properties:
  • Object Count: Exact or relative quantities (e.g., "exactly 2 beds").
  • Object Attributes: Specific properties (e.g., "wooden," "red").
  • Object-Object Relationships: Spatial relations between objects (e.g., "next to," "facing").
  • Object-Architecture Relationships: Relations between objects and structural elements (e.g., "against wall," "hanging from ceiling").
• Creation Process: A hybrid approach combining 100 manually curated descriptions with 400 semi-automatically generated descriptions (using an LLM with human-in-the-loop validation) to ensure diversity and scale.

B. The Evaluation Framework: SceneEval

The framework evaluates generated scenes against the annotations using two primary dimensions: Fidelity (Explicit Requirements) and Plausibility (Implicit Expectations).

1. Object Matching:
Before evaluation, the system renders front-view images of objects in the generated scene and uses a Vision-Language Model (VLM) to map them to the annotated categories, handling cases where metadata is missing or unreliable.

2. Fidelity Metrics (Explicit Constraints):

• Object Count (CNT): Verifies if the number of generated objects matches the text.
• Object Attribute (ATR): Uses VLMs to check if objects possess specified attributes (color, material, style) by analyzing rendered images with scale references.
• Object-Object Relationship (OOR): Uses geometric techniques (ray casting, point sampling) to verify spatial relationships (e.g., "left of," "surrounding") between matched objects.
• Object-Architecture Relationship (OAR): Verifies relationships between objects and architectural elements (walls, floors, corners).

3. Plausibility Metrics (Implicit Constraints):

• Collision (COL): Mesh-based intersection tests to ensure objects do not overlap.
• Support (SUP): Determines if objects are stably supported (ground, wall, ceiling, or other objects) using ray casting and VLM-assisted support type classification.
• Navigability (NAV): Calculates the ratio of the largest connected free space to total free space to ensure the room is traversable.
• Accessibility (ACC): Checks if the "functional sides" of objects (e.g., the front of a sofa or the door of a wardrobe) are unobstructed.
• Out-of-Bounds (OOB): Ensures objects remain within the defined floor plan boundaries.

3. Key Contributions

1. SceneEval Framework: A modular, interpretable evaluation system that disentangles text fidelity from physical plausibility, offering granular diagnostics rather than a single scalar score.
2. SceneEval-500 Dataset: The first large-scale benchmark with fine-grained, structured annotations for text-conditioned 3D scene generation, enabling reproducible research.
3. Comprehensive Metric Suite: The introduction of metrics for functional accessibility and navigability, which are often overlooked but critical for practical usability.
4. Open Release: The authors commit to releasing the code, dataset, and evaluation pipeline.

4. Experimental Results

The authors evaluated six state-of-the-art scene generation methods (ATISS, DiffuScene, InstructScene, LayoutGPT, LayoutVLM, Holodeck) using SceneEval.

• Overall Performance: Holodeck achieved the highest overall fidelity, particularly in object-architecture relationships. LayoutVLM performed best on object counts.
• Fidelity Gaps: All methods struggle significantly with fine-grained constraints. Even the best methods satisfied fewer than 30% of attribute requirements and less than 20% of object-object relationship constraints. This indicates a major gap in translating rich textual specifications into 3D structure.
• Plausibility Insights:
  • LayoutGPT showed high scores in collision and navigability but failed dramatically on Out-of-Bounds (OOB) and Support metrics. This revealed that the method often placed objects outside the room or floating to trivially satisfy other metrics, a failure mode hidden by traditional metrics.
  • Holodeck demonstrated better physical consistency but still struggled with complex spatial arrangements.
• Correlation: Traditional metrics (FID, CLIPsim) showed poor correlation with the specific constraints evaluated by SceneEval, confirming the need for the proposed framework.
• Human Alignment: The framework's automated metrics showed high agreement (Cohen's kappa > 0.5) with human judgments in user studies.

5. Significance and Future Directions

• Diagnostic Value: SceneEval moves beyond "black box" scoring, providing specific feedback on why a method failed (e.g., "objects are colliding" vs. "attributes are wrong").
• Research Direction: The results highlight that current models, often trained on limited datasets like 3D-FRONT, lack the generalization to handle complex, open-vocabulary constraints. Future work must focus on improving retrieval mechanisms for specific attributes and better spatial reasoning.
• Practical Impact: By identifying gaps in navigability and accessibility, the framework pushes the field toward generating scenes that are not just visually plausible but functionally usable for robotics, simulation, and gaming.

In conclusion, SceneEval establishes a new standard for evaluating 3D scene synthesis, shifting the focus from generic realism to semantic coherence and physical plausibility, thereby guiding the development of more controllable and practical scene generation systems.