SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generation

1. Problem Statement

Generating large-scale, photorealistic 3D driving scenes is critical for autonomous vehicle simulation, data synthesis, and scene editing. However, existing approaches face significant trade-offs:

• Image/Video-based Diffusion: Methods that distill 2D generative models into 3D (e.g., via 3D Gaussian Splatting) often suffer from poor geometric coherence, lack of multi-view consistency, and limited editability. They are often tied to specific camera trajectories.
• Voxel-based 3D Generation: Direct 3D generation using dense voxel grids scales poorly. Computational cost and memory usage grow rapidly with scene size, making high-resolution, large-extent generation impractical.
• Implicit Representations: Large-scale world models often rely on highly compressed or implicit representations that are difficult to render efficiently or manipulate structurally.

The Core Challenge: How to achieve a framework that simultaneously ensures 3D consistency, scalability to large urban environments, and photorealistic rendering without requiring per-scene optimization.

2. Methodology

The authors propose SEM-ROVER, a framework that generates scenes directly in 3D space using a novel discrete representation and a semantic-conditioned diffusion model.

A. The Representation: $\Sigma$-Voxfield

Instead of dense voxels or continuous fields, the authors introduce the $\Sigma$-Voxfield grid:

• Definition: A discrete representation where each occupied voxel contains a fixed number ($n$) of colorized surface samples (points).
• Structure: Each voxel $v_\Sigma$ is defined as a set of $n$ points with 3D coordinates $(x, y, z)$ relative to the voxel center and RGB colors.
• Advantages: This representation decouples geometry and appearance into a fixed-cardinality token, making it ideal for transformer-based architectures. It captures local geometry and photometry jointly while remaining sparse and aligned with world coordinates.
• Rendering: To render 2D images, points are converted into 2D Gaussians aligned with local surface normals (via PCA), ensuring completeness without increasing the token count.

B. Generative Model: Semantic-Conditioned Diffusion

The core generation engine is a 1D Diffusion Transformer operating on local neighborhoods of $\Sigma$-Voxfield tokens.

• Input: A local set of adjacent voxels ($\mathcal{X}_\xi$) containing up to $N_\xi$ tokens.
• Conditioning: The model is conditioned on:
  1. Semantic Labels: Each voxel is associated with a class (e.g., road, building, vegetation).
  2. 3D Positional Encoding: Sinusoidal encodings of voxel centers are added to expose the 3D spatial structure to the transformer.
• Process: The model learns to denoise the stacked surface samples ($\psi(v_\Sigma)$) to predict the clean geometry and appearance, minimizing an $\ell_2$ loss.

C. Scalability: Progressive Spatial Outpainting

To generate scenes spanning tens of thousands of square meters without exploding memory:

• Strategy: The model operates on fixed-size local neighborhoods. Large scenes are synthesized via iterative spatial outpainting.
• Mechanism: Using the Repaint diffusion scheduler, the system generates new regions conditioned on previously generated overlapping neighborhoods.
• Benefit: This decouples generation cost from total scene size. Memory and compute remain constant per step, allowing linear scaling to arbitrarily large extents.

D. Deferred Rendering

The raw $\Sigma$-Voxfield grid is a coarse approximation. To achieve photorealism:

• Pipeline: The grid is first rendered into a 2D buffer ($I_\Sigma$) using 2D Gaussian Splatting.
• Refinement: A Deferred Rendering Module (based on Stable Diffusion) takes $I_\Sigma$ as a 3D buffer input and generates the final high-fidelity image ($I_{DR}$).
• Consistency: To ensure temporal coherence in video sequences, the authors employ either Autoregressive Stable Diffusion (ASD) (conditioning on the previous frame) or Video Stable Diffusion (VSD) (generating 12 frames simultaneously).
• Sky/Background: A visibility mask is used to prevent the model from hallucinating geometry in areas not covered by the 3D buffer (e.g., sky, distant background).

3. Key Contributions

1. $\Sigma$-Voxfield Representation: A novel, fixed-cardinality discrete surface approximation that jointly encodes local geometry and photometry, tailored for 3D generative modeling.
2. Semantic-Conditioned 3D Diffusion: A transformer-based diffusion model that operates directly on 3D surface tokens with semantic and positional conditioning, avoiding the need for 2D-to-3D distillation.
3. Scalable Outpainting: A progressive generation strategy that enables the synthesis of massive urban scenes (e.g., $\approx 100,000 \text{ m}^2$) with constant per-step computational cost.
4. Per-Scene Optimization-Free Rendering: A coupled deferred rendering pipeline that produces photorealistic, multi-view consistent images without the heavy optimization costs associated with NeRF or 3DGS training per scene.

4. Experimental Results

The method was evaluated on the Waymo Open Dataset (WOD) and PandaSet.

• Qualitative Performance:
  • Generated scenes adhere strictly to semantic priors (road topology, building layouts).
  • Demonstrates strong multi-view consistency across camera trajectories, outperforming competitors like InfiniCube and GEN3C, especially in side-camera views where other methods often degrade.
  • Supports semantic editing (e.g., removing/adding cars) and inpainting (filling masked regions) while maintaining global coherence.
• Quantitative Performance:
  • Image Quality: Achieves competitive FID and KID scores on seen views and significantly outperforms baselines on novel views (shifted camera poses), proving superior geometric consistency.
  • Efficiency:
    • VRAM Usage: Requires only 8 GB VRAM, compared to 43 GB (GEN3C) and 75 GB (InfiniCube).
    • Runtime: Generates a large scene in $\approx 20$ minutes, comparable to competitors but with a fraction of the memory footprint.
• Ablation Studies: Confirmed that both semantic conditioning and point ordering within voxels are critical for performance; removing them degrades geometric fidelity (measured by Chamfer Distance and MMD).

5. Significance and Impact

SEM-ROVER represents a paradigm shift in 3D scene generation by moving away from "distilling 2D models into 3D" toward native 3D generation.

• Scalability: It solves the memory bottleneck of large-scale 3D generation, enabling the creation of entire city blocks rather than just small patches or object-centric scenes.
• Editability: The explicit, structured nature of the $\Sigma$-Voxfield grid allows for precise semantic editing and scene manipulation, which is difficult with implicit or purely image-based methods.
• Application: The framework is highly relevant for autonomous driving simulation, providing a cost-effective way to generate diverse, photorealistic, and geometrically consistent training data for perception and planning algorithms.

Limitations: The current method focuses on static scenes and does not yet model dynamic elements (moving vehicles, pedestrians) or temporal evolution explicitly. Additionally, control over fine-grained appearance attributes (texture style, lighting) is limited to the semantic conditioning.