💻 computer science

RoomPilot: Controllable Indoor Scene Synthesis via Multimodal Semantic Parsing

RoomPilot is a unified framework that enables controllable indoor scene synthesis from multi-modal inputs like text and CAD plans by mapping them to a structured Indoor Domain-Specific Language (IDSL) and employing a hierarchical pipeline to generate physically consistent and semantically coherent 3D environments.

Original authors: Wentang Chen, Shougao Zhang, Yiman Zhang, Tianhao Zhou, Ruihui Li

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wentang Chen, Shougao Zhang, Yiman Zhang, Tianhao Zhou, Ruihui Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Digital Architect"

Imagine you want to build a house, but you don't want to hire a team of architects, engineers, and interior designers. You just want to say, "I want a cozy living room with a sofa facing a TV," or hand over a rough sketch of a floor plan, and have a computer instantly build a perfect, realistic 3D version of it.

That is what RoomPilot does. It is a new computer system that turns text descriptions or CAD (computer-aided design) floor plans into fully furnished, 3D indoor scenes.

The Problem: Why is this hard?

Previous attempts at this were like trying to build a house with a blindfold on:

Old "Rule" systems were like rigid robots. They could build houses, but they couldn't understand if you asked for a "cozy" vibe or a "modern" look. They just followed a strict script.
Old "AI" systems were like creative artists who had no sense of physics. They could make a room look pretty, but they might put a sofa floating in mid-air or a bed inside a wall.
Old "LLM" systems (using Large Language Models) were like great storytellers who couldn't draw. They could describe a room perfectly in words, but when they tried to build it, the furniture didn't fit together, and the rooms didn't connect logically.

The Solution: The "Universal Translator" (IDSL)

The core magic of RoomPilot is something they call IDSL (Indoor Domain-Specific Language).

Think of IDSL as a universal translator or a blueprint language that sits between your messy human instructions and the computer's rigid building rules.

You speak: "I want a big sofa in the living room."
The Translator (IDSL) converts it to: "Object: Sofa. Location: Living Room. Relation: On the floor. Size: Large."
The Builder: Takes this clean, structured list and builds the scene.

This is crucial because it allows the computer to understand exactly what you want without getting confused by vague words.

How It Works: The Three-Step Assembly Line

RoomPilot builds a house in three distinct layers, like a construction crew working from the outside in:

The Skeleton (Building Level):
First, it looks at the big picture. If you gave it a text description, it figures out: "Okay, this is a 3-bedroom apartment. The kitchen is next to the living room, and there's a door leading outside." If you gave it a CAD drawing, it reads the lines to get the exact same info. It builds the walls and the floor plan first.
The Brain (Room Level):
Next, it fills in the details for each room. It asks, "What usually goes in a bedroom? A bed, a nightstand, a closet." It places these items conceptually, ensuring they fit the room's shape.
The Muscle (Object Level):
Finally, it picks the actual 3D furniture. It doesn't just guess; it has a massive library of real furniture models. It finds a sofa that matches your description, checks if it fits the space, and places it.

The Secret Sauce: The "Self-Regulating" Optimizer

Here is the most clever part. Sometimes, the computer makes a mistake. Maybe it puts a table in the middle of a door, or a chair is floating.

RoomPilot uses a Self-Regulating Optimization process. Imagine a very strict, invisible interior designer who keeps walking through the room and fixing things:

The "Structural" Check: "Is this table crashing into the wall? Is the lamp floating? Fix it!" (This ensures physics are real).
The "Semantic" Check: "Is the TV facing the sofa? Is the bed near the window? Fix it!" (This ensures the room makes sense).

The system uses a technique called Adaptive Annealing. Think of this like cooling down hot metal.

At first (Hot): The system is wild and tries big, crazy changes to find a good layout.
As it cools down: It gets more careful, making tiny, precise adjustments to perfect the details until everything is locked in place.

The Results: Why is this better?

The paper tested RoomPilot against other methods and found:

It listens better: If you ask for 5 chairs, it puts exactly 5 chairs there. Other methods often guess wrong.
It builds better: The rooms are physically possible. No floating furniture, no walls passing through tables.
It handles complexity: It can build entire multi-room apartments where the kitchen connects to the dining room, and the bedrooms are in the right spots, all based on a single sentence or a floor plan.

In a Nutshell

RoomPilot is like having a super-smart, super-fast architect who speaks your language, understands physics, and has an endless supply of furniture. It translates your vague ideas or rough sketches into a perfectly organized, realistic 3D home, fixing its own mistakes along the way to ensure everything looks and feels right.

Technical Summary: RoomPilot: Controllable Indoor Scene Synthesis via Multimodal Semantic Parsing

Problem Statement
Generating controllable indoor scenes is essential for applications in game development, architectural visualization, and embodied AI. However, existing approaches struggle to simultaneously satisfy the requirements of multi-room support, fine-grained controllability, explicit spatial relations, and physical plausibility.

Model-based methods (e.g., ATISS, DiffuScene) often lack diversity and generalize poorly to novel scenes due to data scarcity.
Rule-based and procedural systems (e.g., Infinigen Indoors, ProcTHOR) ensure physical validity but offer limited controllability over semantic details and user intent.
Vision-based approaches struggle to precisely control scene layouts.
LLM-based methods enhance open-vocabulary understanding but frequently lack precise control over spatial relations between objects.
A primary obstacle is the lack of a unified scene representation that bridges heterogeneous inputs (text or CAD) with interpretable, controllable 3D synthesis.

Methodology
The authors propose RoomPilot, a unified framework that synthesizes controllable indoor scenes from detailed textual descriptions or CAD floor plans. The framework operates through three tightly integrated modules:

Cross-Modal Semantic Parsing:
- Inputs (text or CAD) are transformed into an initial Indoor Domain-Specific Language (IDSL) configuration ( $S_0$ ).
- For text, a hierarchical LLM parsing strategy (using GPT-4) extracts global structure (building level), furniture plans (room level), and precise spatial arrangements (object level).
- For CAD, a geometric parsing pipeline extracts room contours, doors, and windows from vectorized representations (e.g., SVG), mapping them to IDSL. Object-level content for CAD is inferred by the LLM based on room types.
Self-Regulating Scene Optimization:
- The initial IDSL is iteratively refined into an optimized configuration ( $S^*$ ) using a self-regulating, energy-based process.
- Dual-Channel Energy: The objective function decomposes into a Structural Energy (collisions, boundaries, stability) and a Semantic Energy (functional proximity, directional alignment).
- Adaptive Annealing: Unlike fixed schedules, the optimization weights and temperature are adaptively modulated by a rule-alignment score ( $\rho(t)$ ). As structural constraints are satisfied, the focus shifts from geometric feasibility to semantic organization.
- Progressive Rule Activation: Rules are activated in stages (structural $\to$ functional $\to$ stylistic) to prevent premature coupling of heterogeneous objectives.
Hierarchical Indoor Scene Generation:
- The optimized IDSL is instantiated into high-fidelity 3D scenes.
- Building Level: Procedural generation creates watertight wall meshes based on room polygons.
- Room Level: Boolean operations instantiate doors and windows.
- Object Level: A hybrid ecosystem retrieves assets from a curated dataset (9,271 annotated assets) or public datasets (Objaverse, 3D-FUTURE). If retrieval confidence is low, category-specific procedural generators (PCG) synthesize geometry.
- Relationship-Aware Refinement: A post-placement optimization step enforces physical stability (e.g., objects resting on surfaces) and eliminates floating or intersecting artifacts.

Key Contributions

RoomPilot Framework: A unified system for controllable indoor scene synthesis supporting both textual descriptions and CAD floor plans as inputs.
Indoor Domain-Specific Language (IDSL): A structured, hierarchical intermediate representation (Building, Room, Object levels) that translates heterogeneous inputs into interpretable semantic and geometric specifications, serving as a controllable bridge between user intent and 3D synthesis.
Self-Regulating Optimization: An energy-based algorithm that adaptively balances structural stability and semantic alignment without fixed cooling schedules, enabling coherent multi-room layouts.
High-Fidelity Dataset: A curated asset dataset with rich semantic annotations to support realistic scene synthesis.

Experimental Results
The authors evaluate RoomPilot against six state-of-the-art methods (e.g., Holodeck, I-Design, LayoutGPT) across open-vocabulary text generation and CAD-conditioned generation.

Controllability: RoomPilot significantly outperforms baselines in Layout Fidelity (LF), Constraint Satisfaction Rate on object count (CSR $_{count}$ ), and relational constraints (CSR $_{rel}$ ).
Physical Plausibility: The method achieves near-zero out-of-boundary and collision cases, with superior scores in physical realism and stability compared to diffusion-based or retrieval-based approaches.
CAD Fidelity: When conditioned on CAD floor plans, RoomPilot preserves room shapes, dimensions, and opening positions with high geometric accuracy (Wall Accuracy: 0.87 vs. 0.31 for Holodeck).
User Study: Human evaluators rated RoomPilot highest in visual quality, layout reasonableness, and physical plausibility.
Ablation Studies: Removing IDSL or the self-regulating optimization module causes the most significant performance drops, confirming their necessity for high-quality, controllable synthesis.

Significance
The paper claims that RoomPilot represents a significant step toward controllable 3D indoor scene synthesis by addressing the fragmentation in existing paradigms. By introducing IDSL and a self-regulating optimization process, the framework achieves a balance between structural coherence, semantic alignment, and physical plausibility that prior methods could not simultaneously satisfy. This enables the generation of structurally valid, multi-room environments that faithfully respond to complex user instructions, facilitating practical deployment in embodied AI and architectural visualization. The authors note that while the system is robust, failure cases (e.g., missing objects in complex relational constraints) can occur due to the inherent behavior of simulated annealing when optimizing discrete composition and continuous relations jointly.