🤖 AI

HetScene: Heterogeneity-Aware Diffusion for Dense Indoor Scene Generation

HetScene is a heterogeneity-aware, two-stage diffusion framework that improves dense indoor scene generation by decomposing objects into primary and secondary categories to first establish a stable global structural skeleton and then synthesize detailed contextual layouts, thereby overcoming the limitations of existing methods in modeling complex spatial dependencies and physical plausibility.

Original authors: Zini Chen, Junming Huang, Rong Zhang, Jiamin Xu, Cheng Peng, Chi Wang, Weiwei Xu

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zini Chen, Junming Huang, Rong Zhang, Jiamin Xu, Cheng Peng, Chi Wang, Weiwei Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to design a busy, realistic living room for a video game or a robot training simulator. The challenge isn't just picking furniture; it's figuring out how to fit a massive sofa, a dining table, and a bookshelf together, and then filling in the tiny details like throw pillows, books, and lamps without everything floating in mid-air or crashing into each other.

The paper "HetScene" argues that most current AI tools try to do this all at once, like trying to paint the entire room in a single brushstroke. This works okay for empty rooms, but when the room gets crowded with hundreds of items, the AI gets confused. It struggles to balance the "big picture" (where the walls and big furniture go) with the "small details" (where the small items go).

The Core Idea: The "Skeleton and Flesh" Approach

The authors propose a new method called HetScene that splits the job into two distinct stages, treating big objects and small objects differently. They call this "heterogeneity-aware," which is just a fancy way of saying, "We know these things are different, so let's handle them differently."

Think of it like building a house:

Stage 1: The Skeleton (Structural Layout Generation)
First, the AI builds the "skeleton" of the room. It focuses only on the Primary Objects: the big, heavy hitters like beds, sofas, dining tables, and cabinets.
- The Analogy: Imagine you are arranging the main pieces of a puzzle. You place the large corner pieces and the big central shapes first. You make sure the table is against the wall and the sofa faces the TV. At this stage, the AI ignores the tiny stuff. It just ensures the room has a solid, logical structure.
Stage 2: The Flesh and Details (Contextual Layout Generation)
Once the big furniture is locked in place, the AI moves to the Secondary Objects: the small, numerous items like books, lamps, pillows, and decorations.
- The Analogy: Now that the furniture is set, you start decorating. You know the lamp goes on the table, and the pillows go on the sofa. The AI uses the "skeleton" from Stage 1 as a guide. It doesn't guess where the lamp goes; it looks at the table it just placed and says, "Ah, the table is here, so the lamp should be right next to it."

Why This Works Better

The paper claims that previous AI models tried to do both steps simultaneously. It's like trying to arrange the furniture and place the tiny trinkets at the exact same time. The AI gets overwhelmed, often forgetting the small items or placing them in impossible spots (like a lamp floating in the air).

By separating the tasks:

Stage 1 solves the hard math of "Where does the big stuff go?" without getting distracted by the noise of small items.
Stage 2 solves the logic of "Where does the small stuff go?" by using the big stuff as a map.

The Secret Sauce: A "Learnable Map"

To make this work, the AI uses a special "relationship graph." Think of this as a digital instruction manual that tells the AI how things relate.

If the text prompt says "a dining room," the AI knows a table and chairs are the "Primary" skeleton.
The "Secondary" items (like a centerpiece or napkins) are then generated specifically around that table.

The authors also invented a special "modulation" trick. In computer terms, the numbers representing the size of a table are huge compared to the numbers for a tiny book. If you mix them directly, the big numbers drown out the small ones. The new method acts like a volume knob, turning the "volume" of the big furniture down slightly so the AI can hear the small furniture clearly, ensuring everything converges into a perfect design.

The Results

When they tested this on a massive dataset of 3D rooms (M3DLayout), HetScene created rooms that were:

More Realistic: Fewer floating objects or furniture stuck inside walls.
Denser: It could handle rooms with dozens or even hundreds of objects without breaking.
Better Aligned: The rooms matched the text descriptions (e.g., "dining area") much more accurately than previous methods.

In short, HetScene stops trying to do everything at once. It builds the frame first, then fills in the details, resulting in indoor scenes that look like real, lived-in spaces rather than messy, floating collections of objects.

Technical Summary: HetScene

Problem Statement

The generation of controllable and physically plausible 3D indoor scenes is a critical prerequisite for constructing high-fidelity simulation environments for embodied AI. While recent data-driven approaches utilizing auto-regressive and diffusion models have improved the diversity of generated layouts, they face significant limitations when scaling to realistic, dense indoor environments.

Existing methods typically treat all objects as homogeneous instances within a unified generation process. This "homogeneous formulation" forces the model to simultaneously handle global room-level planning (e.g., wall alignment, functional zones) and local, support-aware placement of small objects. Consequently, these models struggle with:

Scalability: They often fail to generate scenes with high object counts, leading to small-object omission or capacity limitations.
Physical Plausibility: The entanglement of global and local dependencies results in artifacts such as floating objects, structural overlaps, and weak local coherence.
Modeling Mismatch: Real-world scenes exhibit structural heterogeneity, where objects play distinct roles. Primary objects (e.g., beds, tables) are sparse and governed by global constraints, while secondary objects (e.g., lamps, books) are dense and locally conditioned on primary objects. Current pipelines fail to explicitly model this distinction.

Methodology

To address these challenges, the authors propose HetScene, a heterogeneity-aware, two-stage diffusion framework that decomposes indoor scene synthesis into Structural Layout Generation (SLG) and Contextual Layout Generation (CLG).

1. Hierarchical Decomposition

The framework explicitly separates objects into two disjoint sets based on semantic roles and geometric scales:

Primary Objects ( $S_L$ ): Large, core furniture items that form the room's skeleton. These are sparse and globally constrained.
Secondary Objects ( $S_S$ ): Smaller, numerous items that depend on primary objects for physical support and semantic co-occurrence.

2. Two-Stage Generation Pipeline

Stage 1: Structural Layout Generation (SLG)
- Goal: Generate globally coherent layouts containing only primary objects.
- Conditioning: The model is conditioned on text descriptions, top-down binary room masks, and a Learnable Scene Graph (LSG).
- Mechanism: The LSG explicitly encodes spatial relationships and topological constraints as a conditional signal, guiding the global organization of the room.
Stage 2: Contextual Layout Generation (CLG)
- Goal: Synthesize dense secondary objects around the fixed primary layout.
- Mechanism: The generated primary layout ( $\hat{X}_L$ ) serves as a fixed geometric anchor (without diffusion noise). Secondary objects ( $X_S$ ) are denoised in a sequence concatenated with the primary tokens.
- Dependency Modeling: Through multi-head self-attention, secondary object slots aggregate information from both primary and other secondary tokens, modeling cross-scale long-range dependencies and local support relations.

3. Key Architectural Innovations

Transformer-Based Denoising Backbone: Replacing the 1D-UNet used in prior work (e.g., DiffuScene) with a Point·E-style Transformer to eliminate task-irrelevant sequential order bias. Spatial geometric cues are injected via spatial encodings derived from bounding box positions rather than standard positional encodings.
Learnable Spatial-Semantic Modulation: To prevent numerically large geometric signals from overwhelming discrete semantic attributes during training (a common issue in dense scenes), the authors introduce a learnable scalar $\gamma_{pos}$ . This scalar adaptively weights the injection of spatial embeddings, ensuring balanced optimization and convergence.
Learnable Scene Graph (LSG) Condition: Instead of relying solely on implicit self-attention, the LSG explicitly encodes a relationship graph $G=(V, E)$ where vertices represent object instances and edges represent directed spatial constraints. These are processed via a lightweight Transformer encoder and aggregated into a global graph token for cross-attention guidance.

Experimental Results

The method was evaluated on the M3DLayout benchmark, a large-scale multi-source dataset integrating 3D-FRONT, Matterport3D, and Inf3DLayout. The dataset was filtered to include scenes with up to 20 primary and 100 secondary objects.

Quantitative Performance: HetScene achieved state-of-the-art performance across all evaluation metrics (Fréchet Inception Distance [FID], Kernel Inception Distance [KID], and CLIP scores) on the full test set and across individual data sources (Inf3DLayout, MP3D, 3D-FRONT).
- On the XZ plane (floor plan), HetScene reduced the FID score by approximately 11.7% compared to the second-best method (DiffuScene).
- It demonstrated consistent advantages in side-view metrics (XY and YZ), indicating better handling of vertical structure and object stacking.
Ablation Studies:
- Replacing the 1D-UNet with the Transformer backbone improved global dependency modeling.
- The heterogeneity-aware pipeline (decoupling primary and secondary generation) significantly mitigated optimization conflicts caused by scale variances.
- The inclusion of the LSG condition enhanced spatial logic and complexity without degrading text-semantic consistency (CLIP scores remained stable).
Qualitative Results: Visual comparisons showed that HetScene produces physically plausible layouts with high complexity, avoiding common artifacts like floating objects or structural overlaps seen in baseline models.

Significance and Contributions

The paper claims that HetScene represents a shift from homogeneous to heterogeneity-aware generation, offering the following contributions:

Framework Design: A two-stage framework that decouples global room organization from local support-aware placement, enabling the scalable generation of realistic scenes with dozens to hundreds of objects.
Modularity: The Contextual Layout Generation (CLG) stage is designed as a plug-and-play component compatible with arbitrary upstream coarse layouts.
Training Stability: The introduction of the learnable spatial-semantic modulation mechanism is identified as empirically necessary for the convergence of secondary-object generation in high-density scenarios.
Physical Plausibility: By explicitly modeling the distinct statistical and geometric distributions of primary and secondary objects, the method improves the physical plausibility of generated scenes, addressing the limitations of existing data-driven approaches in dense environments.

The authors conclude that the principle of semantic-level decomposition adopted in this work is broadly applicable and can be extended to other layout and structured generation tasks involving hierarchical organization.