💻 computer science

Repurposing 3D Generative Model for Autoregressive Layout Generation

LaviGen is a novel framework that repurposes 3D generative models into an autoregressive system for creating physically plausible and geometrically coherent 3D layouts directly in native 3D space, achieving state-of-the-art performance with significantly improved efficiency and physical plausibility on the LayoutVLM benchmark.

Original authors: Haoran Feng, Yifan Niu, Zehuan Huang, Yang-Tian Sun, Chunchao Guo, Yuxin Peng, Lu Sheng

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoran Feng, Yifan Niu, Zehuan Huang, Yang-Tian Sun, Chunchao Guo, Yuxin Peng, Lu Sheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an interior designer, but instead of working with real furniture, you are working with a magical, invisible 3D grid. Your goal is to take a list of instructions (like "put a bed here and a chair there") and arrange the furniture so it looks good, doesn't crash into walls, and doesn't float in mid-air.

For a long time, computers were bad at this. They were like novel writers who could write a beautiful story about a room but couldn't actually build the room. They would say, "Here is a bed," but they might accidentally place the bed inside the wall or make it float three feet off the floor because they didn't truly understand the physics of 3D space.

Enter LaviGen. Think of LaviGen not as a writer, but as a master sculptor who works directly in the clay.

Here is how it works, broken down into simple concepts:

1. The Old Way: "The JSON Writer"

Previous methods (like LayoutGPT) treated room design like writing a computer code list (JSON). They would say: "Bed: x=1, y=2, z=3."

The Problem: The computer is great at following the grammar of the sentence, but it has no "feeling" for the room. It's like asking someone who has never seen a chair to describe where to put it. They might put the chair on the ceiling because they didn't understand gravity.
The Result: Rooms with floating chairs, chairs inside tables, or beds stuck in the wall.

2. The LaviGen Way: "The Native Sculptor"

LaviGen changes the game. Instead of writing a list, it builds the room directly in 3D space, just like a human sculptor.

The Analogy: Imagine you have a box of Lego bricks. Instead of writing a manual on how to stack them, you just pick them up and snap them together. You can feel if a piece fits. If a piece is too big, you know it won't fit.
How it works: LaviGen uses a "3D brain" (a 3D generative model) that already knows how rooms usually look. It knows that chairs go around tables, not inside them. It places objects one by one, checking the physical space as it goes.

3. The "Autoregressive" Process: "The Domino Effect"

LaviGen doesn't try to build the whole room at once (which is chaotic). It builds it step-by-step, like a line of falling dominoes.

It places the bed.
It looks at the bed and the empty space, then places the nightstand next to it.
It looks at the bed and nightstand, then places the rug.

The Benefit: Because it builds step-by-step, it can adjust to what it just placed. If the bed is big, the nightstand automatically knows to be smaller or further away.

4. The "Teacher-Student" Trick: "The Rehearsal"

Here is the tricky part. When you build a long line of dominoes, if you make a tiny mistake with the first one, the whole line might fall over later. This is called Exposure Bias. The computer gets good at following instructions, but bad at fixing its own mistakes.

To fix this, the researchers used a clever training trick called Dual-Guidance Self-Rollout:

The Student: The AI that is learning to build the room.
The Teacher (Holistic): A "Grand Architect" that looks at the entire finished room and says, "Does this whole room look good?"
The Teacher (Step-Wise): A "Foreman" who watches every single step and says, "Hey, you put that chair too close to the wall, fix it right now."
The Magic: The Student is forced to practice building the room using its own imperfect previous steps (not the perfect ones from the textbook). It makes mistakes, gets corrected by the Teachers, and learns how to recover. It's like a musician practicing a song; they don't just play the notes perfectly once; they practice recovering when they hit a wrong note so they don't crash the whole performance.

5. The Results: "No More Floating Chairs"

Because LaviGen works directly in the 3D world and practices fixing its own mistakes:

Physics: Objects stay on the floor. They don't float. They don't crash into each other.
Speed: It is 65% faster than the old methods because it doesn't need to run expensive simulations to fix errors later.
Editing: You can tell it, "Move the bed to the window," and it instantly rearranges the whole room to make it fit, just like a human would.

Summary

LaviGen is like upgrading from a computer that writes a recipe for a cake (which might result in a burnt mess) to a robot chef that actually knows how to mix, bake, and frost the cake in real-time, adjusting the heat if it gets too hot. It understands the physics of the room, not just the words describing it.

1. Problem Statement

Generating coherent 3D scene layouts is critical for VR/AR, robotics, and simulation. Existing methods face significant limitations:

Text-to-Layout (LLM-based): Methods like LayoutGPT treat layout as a language task (JSON output). While semantically coherent, they lack physical modeling, leading to object collisions, floating artifacts, and boundary violations.
Vision-based Optimization: Methods like LayoutVLM use 2D rendered images to supervise layout generation. While this improves physical plausibility, it is computationally expensive, relies on 2D supervision which is not holistic for 3D interactions, and lacks fundamental understanding of 3D spatial structures.
Core Challenge: How to leverage the rich 3D geometric priors of generative models to perform layout generation directly in native 3D space, ensuring both semantic consistency and physical plausibility without iterative 2D refinement.

2. Methodology: LaviGen Framework

LaviGen repurposes a pretrained 3D generative model to perform autoregressive layout generation directly in 3D space. Instead of predicting coordinates from text, it places objects sequentially into a scene state.

A. Core Architecture: Adapted 3D Diffusion Model

The framework is built upon a structured 3D latent diffusion model (inspired by TRELLIS).

Native 3D Space: The model operates on sparse voxel occupancies rather than 2D images or JSON text.
Autoregressive Process: Given a current scene state $S_i$ , a target object $O_i$ , and a text instruction $c$ , the model generates an updated scene state $S_{i+1}$ .
Identity-Aware Embedding: To distinguish between the existing scene context and the newly added object, the model uses an extended Rotary Position Embedding (RoPE). It assigns a specific identity flag ( $f=0$ for scene/noise, $f=1$ for the new object) to tokens, allowing the model to preserve spatial alignment while disentangling semantic roles.
Pipeline:
1. Encode scene state $S_i$ and object $O_i$ into latent representations.
2. Concatenate with noisy latents and text embeddings.
3. Denoise via a Multimodal Diffusion Transformer (MMDiT).
4. Decode the updated voxel occupancy to extract surface points.
5. Fit the actual 3D mesh to these points using Iterative Closest Point (ICP) to determine final position, rotation, and scale.

B. Training Strategy: Dual-Guidance Self-Rollout Distillation

Autoregressive generation suffers from exposure bias (training on ground truth but inferring on imperfect model outputs), leading to error accumulation. LaviGen addresses this via a post-training distillation strategy:

Self-Rollout: During training, the student model generates sequences based on its own previous predictions (rather than ground truth), forcing it to learn to recover from its own errors.
Dual-Guidance Supervision:
1. Holistic Guidance ( $L_{holistic}$ ): A frozen bidirectional teacher model supervises the final scene state to ensure global scene quality and semantic coherence.
2. Step-Wise Guidance ( $L_{step}$ ): A causal autoregressive teacher model supervises each step of the generation to ensure individual object placements are accurate and physically valid.
Objective: The student minimizes the reverse KL divergence (via Distribution Matching Distillation) against both teachers, balancing global scene integrity with local object accuracy.

3. Key Contributions

Native 3D Autoregressive Framework: Proposes LaviGen, the first framework to treat layout generation as a direct 3D generative process, eliminating the need for 2D supervision or JSON-based coordinate regression.
Adapted 3D Diffusion & Identity Embedding: Introduces a modified diffusion model with identity-aware positional embeddings to handle the complex interaction between scene context and new object insertion.
Dual-Guidance Self-Rollout: Develops a novel distillation mechanism that mitigates exposure bias in long-sequence generation by combining global and step-wise supervision, significantly improving training stability and inference speed.
Versatility: The framework naturally supports layout completion (filling missing objects) and layout editing (insertion/removal) without retraining, as it operates on the scene state directly.

4. Experimental Results

Evaluated on the LayoutVLM benchmark (8–10 objects per scene):

Physical Plausibility: LaviGen achieves a 19% improvement in physical plausibility (measured by Collision-Free and In-Boundary scores) over the state-of-the-art LayoutVLM. It effectively eliminates floating objects and collisions common in LLM-based methods.
Efficiency: LaviGen is 65% faster in computation time compared to LayoutVLM, which relies on expensive iterative 2D optimization.
Semantic Alignment: While slightly behind LayoutVLM in pure semantic metrics (due to the difficulty of 3D text grounding), LaviGen maintains high semantic consistency while vastly outperforming others in physical feasibility.
User Study: In a study with 43 participants, LaviGen was rated highest for Physical Plausibility (52.1%) and Overall Quality (55.6%), significantly outperforming LayoutGPT and LayoutVLM.
Ablation: Removing the identity embedding or the dual-guidance distillation leads to severe collisions, inversion errors, and slow inference, validating the necessity of each component.

5. Significance

LaviGen represents a paradigm shift in 3D layout generation:

From "Language" to "Geometry": It moves away from treating 3D space as a text generation problem, instead leveraging the geometric priors inherent in 3D generative models.
Scalability: By operating natively in 3D, it avoids the computational bottlenecks of rendering-based optimization, making it suitable for real-time applications and complex scenes with many objects.
Foundation for Interaction: Its ability to perform editing and completion directly in 3D space makes it a robust foundation for interactive design tools, robotic path planning, and autonomous environment generation where physical constraints are non-negotiable.

In summary, LaviGen successfully bridges the gap between semantic intent and physical reality in 3D scene generation by repurposing generative models for autoregressive, geometry-aware layout synthesis.