💻 computer science

GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation

The paper introduces GaussianGPT, a transformer-based autoregressive model that generates 3D scenes by predicting discrete tokens derived from compressed Gaussian primitives, offering a scalable and controllable alternative to diffusion-based approaches for step-by-step 3D scene construction.

Original authors: Nicolas von Lützow, Barbara Rössle, Katharina Schmid, Matthias Nießner

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicolas von Lützow, Barbara Rössle, Katharina Schmid, Matthias Nießner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a massive, intricate LEGO city.

The Old Way (Diffusion Models):
Most current 3D AI models work like a sculptor starting with a giant, blurry block of clay. They slowly chip away the excess, refining the whole shape at once until a city emerges. It's great for getting a good overall look, but if you want to add a specific window to a specific building in the middle of the process, or if you want to extend the city infinitely to the left, it's very hard. You have to start over or try to "fix" the whole block, which often leads to weird glitches.

The New Way (GaussianGPT):
The paper introduces GaussianGPT, which works more like a master storyteller or a very clever LEGO builder who constructs a scene one brick at a time, from start to finish.

Here is how it works, broken down into simple concepts:

1. The "Magic Bricks" (3D Gaussians)

Instead of using solid blocks, this AI uses "3D Gaussians." Think of these as fuzzy, glowing marbles that can be stretched, squished, and colored. When you put thousands of these marbles together, they look like a solid chair, a tree, or a whole room. They are the building blocks of the scene.

2. The "Translator" (Compression)

A room full of these marbles is too messy for a computer to read directly. So, the first step is a Translator.

It takes the messy pile of marbles.
It organizes them into a neat grid (like a 3D checkerboard).
It turns each spot on the grid into a simple code (a "token"), like turning a complex sentence into a single word.
Analogy: Imagine taking a whole library of books and turning every book into a single barcode. Now the computer just needs to read a long string of barcodes.

3. The "Storyteller" (The Autoregressive Model)

This is the heart of GaussianGPT. It uses a Transformer (the same kind of brain behind chatbots like me) to read and write these barcodes.

How it thinks: It doesn't look at the whole city at once. It looks at the first few bricks, guesses what the next brick should be, places it, then looks at the first three to guess the fourth, and so on.
The "Next Token" Prediction: Just like when you type "The cat sat on the..." and your phone suggests "mat," this model predicts the next 3D brick based on the ones already placed.
The Twist: It predicts two things at once:
1. Where to put the next brick (Position).
2. What that brick looks like (Color/Texture).

4. Why This is a Game-Changer

Because it builds the scene step-by-step, it has superpowers that the "clay sculptor" models don't have:

Infinite Expansion (Outpainting): Since it builds brick-by-brick, you can tell it, "Keep going!" and it will just keep adding bricks to the side, creating a city that is larger than the training data ever saw. It's like a story that never has to end.
Fixing Broken Parts (Completion): If you give it a half-built room (missing a wall), it can look at the existing wall and say, "Okay, based on this, the missing wall should look like this," and fill it in perfectly. It understands the context of what's already there.
Control: You can control the "creativity" of the builder. If you want a very safe, standard room, you tell it to be strict. If you want a wild, crazy room, you tell it to be more random.

5. The Secret Sauce: "3D RoPE"

One of the hardest parts of building a 3D world in a 1D line (like a sentence) is keeping track of space. If you list bricks in a line, the 10th brick might be right next to the 1st brick in 3D space, even though they are far apart in the list.

GaussianGPT uses a special trick called 3D Rotary Positional Embedding (RoPE).

Analogy: Imagine the bricks have little GPS chips inside them. Even though they are lined up in a single file, the AI knows exactly where they are in 3D space relative to each other. This prevents the AI from getting confused and putting a ceiling where a floor should be.

Summary

GaussianGPT is like teaching an AI to build 3D worlds the way humans actually build them: incrementally. Instead of magic-waving a whole city into existence, it lays one brick, then the next, understanding the context of what came before. This makes it perfect for creating huge, endless worlds, fixing broken scenes, and giving creators total control over how their 3D environments are built.

1. Problem Statement

Current state-of-the-art 3D generative models predominantly rely on diffusion or flow-matching formulations. While these methods achieve high visual fidelity, they typically treat 3D scene generation as a holistic denoising process. This approach presents several limitations:

Lack of Incremental Control: Diffusion models struggle with incremental editing, scene completion, or "outpainting" (extending a scene beyond its original boundaries) because they refine the entire scene simultaneously rather than building it step-by-step.
Representation Challenges: 3D data lacks a natural sequential order (unlike text or 2D images), making it difficult to apply autoregressive (next-token prediction) paradigms directly.
Scalability: Generating large-scale, coherent indoor scenes requires managing long-range spatial dependencies and compositional flexibility (object placement, layout) which is difficult for global denoising models to handle efficiently.

The authors propose GaussianGPT, a purely autoregressive framework that treats 3D scene generation as a sequential prediction task, enabling step-by-step construction, completion, and infinite outpainting.

2. Methodology

GaussianGPT operates on 3D Gaussian Splatting primitives but transforms them into a discrete token sequence suitable for a Transformer architecture. The pipeline consists of two main stages:

A. Scene Compression & Tokenization (Autoencoder)

To enable autoregressive modeling, continuous 3D Gaussian scenes must be converted into discrete tokens.

Sparse 3D Feature Grid: Input Gaussian primitives (position, opacity, size, rotation, color) are projected onto a sparse 3D voxel grid. Absolute positions are replaced with relative offsets from voxel centers.
Sparse 3D CNN Encoder-Decoder: A sparse 3D convolutional autoencoder compresses the feature grid into a compact latent representation.
Vector Quantization (LFQ): The latent features are discretized using Lookup-Free Quantization (LFQ). Instead of a standard codebook lookup, the encoder output is discretized based on sign (0 or 1) to directly correspond to codebook indices. This improves codebook utilization and reconstruction quality.
Loss Function: The autoencoder is trained using a combination of re-rendering losses (L1 color + VGG perceptual), occupancy loss (binary cross-entropy), and a codebook entropy loss to ensure diverse token usage.

B. Autoregressive Modeling (Transformer)

The quantized latent grid is serialized into a 1D token sequence and modeled by a causal Transformer.

Serialization: The 3D grid is linearized using a fixed XYZ traversal order (iterating through columns). While this does not preserve strict 3D locality in the 1D sequence, it provides a simple, interpretable ordering.
Token Interleaving: The sequence alternates between Position Tokens ( $p_i$ $p_{i}$ ) and Feature Tokens ( $f_i$ $f_{i}$ ).
- Position Head: Predicts the index of the next occupied voxel.
- Feature Head: Predicts the latent feature vector for that voxel.
- This separation decouples geometric structure from appearance, preventing competition between spatial and semantic features.
3D Rotary Positional Embeddings (RoPE): Standard 1D positional embeddings are insufficient for 3D data where sequence proximity $\neq$ spatial proximity. The authors inject 3D RoPE into the attention mechanism. This encodes the actual relative $(x, y, z)$ coordinates between tokens, allowing the model to reason about spatial locality regardless of the serialization order. An additional dimension in the RoPE distinguishes between position and feature token types.
Architecture: Based on a GPT-2 style decoder-only transformer, utilizing Muon optimizer, query-key normalization, and per-layer residual scaling.

3. Key Contributions

Structured Tokenization of 3D Gaussians: A novel method to compress vector-quantized 3D Gaussian primitives into a discrete latent grid, enabling autoregressive modeling of complex indoor environments.
Autoregressive Transformer Framework: A unified model capable of unconditional generation, scene completion, and large-scale outpainting using a single architecture.
3D-Aware Positional Encoding: The integration of 3D Rotary Positional Embeddings allows the model to maintain spatial coherence even when the 3D grid is serialized into a 1D sequence.
Compositional Inductive Biases: By generating scenes incrementally (next-token prediction), the model naturally supports controllable sampling (via temperature), causal reasoning over partial geometry, and flexible generation horizons.

4. Results & Evaluation

The authors evaluated GaussianGPT on three datasets: PhotoShape (objects), 3D-FRONT (indoor scenes), and Aria Synthetic Environments (ASE) (large-scale scenes).

Unconditional Shape Generation (PhotoShape):
- Outperformed state-of-the-art methods (DiffRF, L3DG, EG3D) in FID (5.68 vs 8.49 for L3DG), KID, and Coverage (COV).
- Produced cleaner Gaussian allocations with sharper structures and fewer noisy outliers compared to diffusion-based baselines.
Scene Generation & Outpainting:
- Successfully generated large-scale scenes (e.g., $12m \times 12m$ ) by iteratively appending latent grid columns.
- Maintained geometric coherence (floor alignment, wall continuity) and stylistic consistency across chunk boundaries, demonstrating the ability to scale beyond fixed training horizons.
Scene Completion:
- Given partial scene inputs (1/4 or 1/2 of a chunk), the model generated semantically plausible and diverse completions.
- Unlike diffusion models that require specialized conditioning, GaussianGPT handles completion naturally by treating the partial scene as a prefix prompt.
Ablation Studies:
- Serialization Order: Surprisingly, the simple XYZ traversal outperformed complex space-filling curves (Z-order, Hilbert). The authors attribute this to the 3D RoPE, which handles spatial locality internally, making the serialization order less critical while simplifying prediction.

5. Significance and Future Work

Significance:

Paradigm Shift: GaussianGPT challenges the dominance of diffusion models in 3D generation, proving that autoregressive transformers are a viable, complementary paradigm for structured 3D synthesis.
Controllability: The step-by-step generation process offers unprecedented control for interactive design, allowing users to extend scenes, fill in missing parts, or sample diverse variations of a layout without retraining or complex conditioning mechanisms.
Efficiency: By operating on explicit 3D Gaussians, the generated scenes are immediately compatible with modern neural rendering pipelines for real-time visualization.

Limitations & Future Work:

Real-World Fidelity: While effective on synthetic data, the autoencoder struggles with high-frequency details in real-world scans (e.g., ScanNet++), leading to noisier geometry.
Uncertainty Modeling: Future work aims to leverage the autoregressive nature to explicitly model uncertainty in partially observed real-world environments, potentially masking uncertain regions while generating plausible completions.
Long-Horizon Stability: Extending the generation context beyond fixed spatial chunks to support truly infinite world generation remains an open challenge.

In conclusion, GaussianGPT establishes a new baseline for compositional 3D generation, demonstrating that treating 3D scenes as structured token sequences enables scalable, controllable, and high-fidelity scene synthesis.