💻 computer science

PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenes

PrITTI is a latent diffusion model that utilizes a hybrid representation of vectorized object primitives and rasterized ground surfaces to generate controllable, editable, and memory-efficient 3D semantic urban scenes, outperforming traditional voxel-based methods in quality, speed, and flexibility.

Original authors: Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas Geiger

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas Geiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a massive, realistic 3D city for a video game or a self-driving car simulator. Traditionally, builders have used a method like Lego bricks (called "voxels" in the paper). You stack millions of tiny, identical cubes to form buildings, trees, and roads.

The problem with this "Lego" approach is that it's heavy, rigid, and hard to fix. If you want to move a car, you have to find every single brick that makes up that car, move them all, and then fill in the empty space where the car used to be with new bricks. If you want a taller building, you might run out of memory because you need so many bricks.

PrITTI proposes a completely different way of building: The "Smart Blueprint" Approach.

Instead of millions of tiny bricks, PrITTI builds the city using smart, pre-made shapes (called "primitives").

A car isn't a million cubes; it's just one smart "box" with a label saying "I am a car, I am here, and I am this size."
A tree is a single "egg shape" (ellipsoid) labeled "I am a tree."
The road is a flat, flexible sheet that can be stretched and shaped like a piece of clay.

Here is how the paper explains this system, broken down into simple steps:

1. The "Smart Box" System (Primitive Representation)

Think of PrITTI as a set of digital LEGO instructions rather than the bricks themselves.

Objects: Every car, pedestrian, or pole is represented by a simple mathematical shape (a box or an egg). The system remembers exactly where it is, how big it is, and which way it's facing.
The Ground: The road and sidewalks are treated like a topographic map. Instead of building the road out of blocks, the system draws a 2D map that says "Here is the road, and here is how high it is."
The Benefit: This is incredibly lightweight. Storing a city of millions of Lego bricks takes up a huge hard drive. Storing a city of "smart boxes" and a "map" takes up very little space.

2. The "Magic Painter" (The Diffusion Model)

The paper uses a type of AI called a Latent Diffusion Model. You can think of this as a magic painter that learns to draw city blueprints.

Training: The AI looks at thousands of real city maps (from the KITTI-360 dataset). It learns the rules: "Cars usually sit on roads," "Trees are often on sidewalks," and "Buildings are tall."
Creation: When you ask it to "paint a new city," it doesn't just guess randomly. It generates a clean, organized blueprint of where every object should go.
Control: You can tell the painter, "Make this city very green" (high vegetation) or "Make it a busy downtown" (lots of cars). The AI adjusts the blueprint accordingly.

3. The "Edit Button" (Editing and Inpainting)

This is where PrITTI shines compared to the old "Lego" methods.

Moving Things: Because every car is just one "smart box," if you want to move a car, you just tell the box to move. You don't have to rebuild the whole street. It's like moving a sticker on a piece of paper rather than chipping away at a stone wall.
Fixing Holes (Inpainting): Imagine you have a city blueprint, but the left half is missing. PrITTI can look at the right half and "paint" in the missing left side, making sure the new buildings line up perfectly with the old ones.
Expanding the City (Outpainting): If you want a bigger city, PrITTI can take your existing blueprint and slide it over, painting a new section that connects seamlessly to the old one, like extending a comic strip.

4. The "Realistic Filter" (Photo-Realistic Synthesis)

The paper notes that PrITTI creates the skeleton of the city (the layout and shapes). To make it look like a real photo, you can take this skeleton and feed it into a separate tool (ControlNet) that paints realistic textures on top.

Think of PrITTI as the architect drawing the floor plan and placing the furniture.
The "Realistic Filter" is the interior designer who paints the walls, adds the wood grain, and puts the sunlight in the windows.
Even though the "furniture" (primitives) is simple shapes, the final photo looks surprisingly real because the layout is so accurate.

Why This Matters (According to the Paper)

Speed: It generates cities much faster than the old Lego methods.
Memory: It uses a fraction of the computer memory.
Flexibility: You can edit specific objects (like rotating a truck) without breaking the whole scene.
Quality: The cities it builds are more organized and less "glitchy" than previous methods, with clearer boundaries between roads and buildings.

In short, PrITTI swaps the heavy, rigid "stack of bricks" method for a lightweight, editable, and smart blueprint system, making it much easier to create, control, and fix 3D cities for simulations.

Technical Summary: PrITTI

Problem Statement

The generation of large-scale, semantically structured 3D urban scenes is critical for digital world modeling and autonomous driving simulation. However, existing methods face significant hurdles. Most current approaches rely on voxel-based representations (e.g., voxel grids or hierarchical structures), which suffer from three fundamental limitations:

Memory Inefficiency: Due to cubic scaling with resolution, dense voxel representations are prohibitively expensive for large environments.
Limited Detail: Fixed spatial resolution often results in truncated or distorted tall elements (e.g., buildings, trees).
Poor Editability: Voxel grids lack explicit object instances. Simple edits, such as translating a vehicle, require identifying and updating all related voxels and filling vacated spaces, a process complicated by ambiguous object-background boundaries.

While some recent works employ abstract semantic layouts, these are often hand-crafted or pre-defined inputs rather than generated outputs. There is a lack of frameworks capable of automatically producing diverse, realistic, and editable 3D semantic urban scenes using a representation that is both memory-efficient and compositional.

Methodology

The authors introduce PrITTI (Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenes), a framework that utilizes a hybrid primitive-based representation combined with a Latent Diffusion Model (LDM).

1. Hybrid Representation

Instead of volumetric voxels, PrITTI represents scenes using two distinct components:

Ground Surfaces: Modeled as upwards-extruded 2D polygons (e.g., roads, sidewalks). Due to the arbitrary geometry of these polygons, they are rasterized into Bird's Eye View (BEV) height maps and binary occupancy masks.
Object Primitives: Objects (e.g., vehicles, vegetation, poles) are modeled as parameterized 3D primitives. Specifically, most objects are represented as 3D cuboids, while vegetation is represented as 3D ellipsoids.
- Each primitive is parameterized by a 9D feature vector: a normalized 3D center location and 6D Cholesky parameters.
- The Cholesky decomposition of a positive-definite scatter matrix is used to jointly encode orientation and size. This approach eliminates orientation ambiguities (common in quaternion encodings) and improves training stability.

2. Two-Stage Training Pipeline

The method employs a two-stage training process:

Stage 1: Layout Variational Autoencoder (LVAE)
A joint VAE compresses the 3D scene layout into a compact, structured 2D latent space.

Encoding: The architecture uses separate encoders for ground and object modalities.
- The Ground Encoder (convolutional) processes raster maps into a latent code.
- The Object Encoder (Transformer-based) processes fixed-size sets of object primitives. To handle variable object counts, scenes are padded to a constant number of primitives per category. The resulting embeddings are mapped onto a 2D latent grid via a scatter-mean operation, preserving spatial structure and permutation invariance.
- The ground and object latents are channel-wise concatenated to form a disentangled joint latent representation ( $\mathbf{z}_{\mathcal{L}}$ ).
Decoding: The decoder reconstructs the scene. The ground decoder uses a convolutional architecture, while the object decoder (based on DETR) uses learnable queries to predict 3D centers, Cholesky parameters, and existence probabilities for each primitive.
Loss: The training objective combines reconstruction losses (L1 for height maps, BCE for masks, and matching losses for primitives via the Hungarian algorithm) with a KL divergence term.

Stage 2: Latent Diffusion Model (LDM)
A Diffusion Transformer (DiT) is trained on the joint latent space generated by the LVAE.

Conditioning: The model is trained to generate scenes conditioned on discrete scene labels (e.g., vegetation density: low, medium, high).
Architecture: The DiT backbone incorporates conditioning via adaLN-Zero blocks.
Applications via Latent Manipulation: Without fine-tuning, the pre-trained LDM supports downstream tasks using a RePaint-inspired latent manipulation strategy:
- Inpainting: Localized edits are performed by synchronizing the denoising of unknown regions with the noised version of known regions in the latent space. The disentangled structure allows for ground-conditioned object generation (editing objects while keeping the ground fixed).
- Outpainting: Scenes are extended beyond their original spatial extent using a sliding-window approach with 50% overlap to ensure spatial consistency.

Key Contributions

First Primitive-Based Framework: PrITTI is the first framework to use coarse 3D bounding primitives (cuboids and ellipsoids) for generating controllable and editable 3D semantic urban scenes at an abstract, semantic level.
Disentangled Latent Representation: The proposed hybrid representation enables a disentangled latent space where ground and object elements are encoded in separate channel groups. This facilitates intuitive, localized manipulation (e.g., ground-conditioned generation) without extra supervision.
Stable Parameterization: The introduction of a Cholesky-based parameterization for 3D object orientation and size offers greater numerical stability and eliminates orientation ambiguities compared to quaternion or eigen-based encodings.
Efficiency and Editability: The primitive-based approach achieves state-of-the-art generation quality with significantly lower memory requirements and faster inference than voxel-based baselines, while supporting direct instance-level editing (translation, rotation, scaling) that is difficult or impossible with voxel grids.

Experimental Results

Experiments were conducted on the KITTI-360 dataset, which provides dense primitive annotations for entire urban environments.

Reconstruction (Stage 1): PrITTI achieves competitive reconstruction quality (IoU and mIoU) compared to voxel-based baselines (SemCity, PDD, XCube) despite being natively primitive-based. Crucially, it requires ~2.52 MB per scene, significantly less than the 8.00 MB required by SemCity and 3.05 MB by XCube at comparable resolutions.
Generation (Stage 2): PrITTI significantly outperforms all voxel-based baselines in generative metrics (Precision, Recall, FID, IS).
- Quality: PrITTI produces diverse, coherent scenes with well-shaped objects and clear boundaries. Baselines often exhibit irregular shapes, indistinct boundaries, and semantic inconsistencies.
- Speed: PrITTI achieves the fastest generation time (0.58s per scene) compared to hierarchical baselines which require multi-stage processing (e.g., PDD at 12.84s).
- Controllability: The model successfully generates scenes conditioned on vegetation density, demonstrating effective control over scene composition.
Downstream Applications: The framework successfully performs scene inpainting, outpainting, and photo-realistic street-view synthesis (using ControlNet) without additional fine-tuning.

Significance and Claims

The paper claims that PrITTI addresses the critical trade-off between memory efficiency, generation quality, and editability in 3D urban scene synthesis. By shifting from voxel grids to a primitive-based paradigm, the method unlocks the full capabilities of diffusion transformers for large-scale scene generation.

The authors emphasize that their approach enables direct instance-level control, allowing for intuitive editing operations (e.g., moving a car) that are computationally expensive and structurally ambiguous in voxel-based methods. Furthermore, the method supports the generation of large-scale, semantically coherent urban layouts through iterative outpainting, a capability that existing voxel-based methods struggle to maintain due to global consistency issues.

While the current framework is limited to static scenes and coarse geometric primitives, the authors posit that this work establishes a foundation for future research into more expressive primitives, dynamic environments, and task-specific rendering pipelines for autonomous driving simulation.