Technical Summary: Scaling Up Occupancy-centric Driving Scene Generation

1. Problem Statement

Driving scene generation is essential for autonomous driving (AD) development, particularly for perception and planning evaluation, as it reduces reliance on costly real-world data collection. However, existing generative models face three primary limitations:

1. Data Scarcity: High-performance occupancy-centric methods rely on annotated semantic occupancy data, which remains scarce and limited in scale (e.g., previous works relied on the Nuscenes dataset).
2. Modality Fragmentation: Current approaches typically generate data in a single format (e.g., RGB video) or rely on coarse geometric conditions (BEV maps, 3D bounding boxes) that fail to capture the intricate distributions of real-world scenes, leading to suboptimal fidelity.
3. Scalability Constraints: Prior unified frameworks (e.g., UniScene) are constrained by limited scenario diversity and dataset scale, hindering their utility for large-scale downstream tasks.

2. Methodology: UniScenev2

The paper proposes UniScenev2, a unified, occupancy-centric framework designed for large-scale 4D dynamic scene generation. The system synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds through a hierarchical, spatio-temporal disentangled architecture.

A. Dataset: Nuplan-Occ

To overcome data scarcity, the authors curated Nuplan-Occ, the largest semantic occupancy dataset to date.

• Scale: It contains approximately 19× more annotated scenes and 18× more frames than Nuscenes-Occupancy, totaling ~3.6 million frames with high-resolution voxel annotations (400×400×32).
• Annotation Strategy: The dataset employs a Foreground-Background Separate Aggregation (FBSA) pipeline:
  1. Separation: Point clouds are separated into background (static) and foreground (dynamic objects) based on bounding boxes.
  2. Aggregation & Registration: Background points undergo statistical filtering and iterative registration (Kiss-ICP) to reduce noise and align geometry. Foreground points are aggregated in local coordinates.
  3. Reconstruction: Neural Kernel Surface Reconstruction (NKSR) is used to generate dense meshes for both background and objects.
  4. Labeling: A hybrid labeling approach combines bounding box annotations for foreground objects and BEV map annotations for background regions to assign precise semantic labels.

B. 4D Occupancy Generation

The framework decomposes 4D scene synthesis into two complementary tasks using a Spatio-temporal Disentangled Architecture:

1. Spatial Expansion: A spatial occupancy Diffusion Transformer (DiT) generates expanded global semantic occupancy grids from noise or optional BEV layouts.
2. Temporal Forecasting: A temporal occupancy DiT predicts future occupancy sequences based on static ego-motion and dynamic object motion.

• Data Filtering: To train these distinct components, the dataset is filtered: Spatial data includes scenes with ego-vehicle motion, while Temporal data includes scenes where the ego is stationary but surrounding traffic moves.
• Architecture: The system utilizes a 4D Occupancy VAE (with 3D CNN and axial attention) for compression and a DiT backbone for denoising.

C. Multi-View Video Generation

To bridge the gap between 3D occupancy and 2D video, the authors introduce a Gaussian Splatting-based Sparse Point Map Rendering strategy:

• Sparse Rendering: Occupancy grids are converted into 3D Gaussian primitives and rendered into sparse semantic and depth point maps. A small Gaussian scale (0.01) is used to ensure geometric precision.
• Robust Calibration: To address sensor misalignment and lens distortion in large-scale data (Nuplan), an Unscented Transform (UT) is integrated into the projection pipeline. This approximates the distribution of 3D Gaussians under non-linear camera models, ensuring precise alignment with multi-view images.
• Generation: These rendered maps serve as robust conditional guidance for a Video Diffusion Transformer (DiT), which synthesizes multi-view videos.

D. LiDAR Point Cloud Generation

To simulate realistic LiDAR data, the framework employs a Sensor-Specific Embedding strategy:

• Sensor Awareness: The model explicitly encodes LiDAR sensor rig data (position and ray geometry) using Fourier encoding and Plücker coordinates.
• Flexible Simulation: A resampling network generates a unified LiDAR rig embedding, allowing the model to flexibly simulate different LiDAR configurations and patterns.
• Ray Smoothness Regularization: A novel loss term encourages depth smoothness between rays with similar histogram features, mimicking the continuous nature of LiDAR scanlines on flat surfaces (e.g., roads).
• Output: The model predicts depth, reflection intensity, and ray-drop probability for each ray.

3. Key Contributions

1. Scalable Unified Framework: UniScenev2 overcomes previous scale limitations by jointly scaling model architecture and training data, achieving high-fidelity generation of occupancy, video, and LiDAR.
2. Spatio-temporal Disentanglement: The introduction of a decoupled architecture and dedicated data filtering strategy enables robust spatial expansion and temporal forecasting for 4D dynamic scenes.
3. Modality-Bridging Techniques:
   • A Gaussian Splatting-based sparse rendering method with Unscented Transform calibration for noise-robust video conditioning.
   • A Sensor-specific embedding scheme for flexible and realistic multi-LiDAR simulation.
4. Nuplan-Occ Dataset: The release of a massive, high-resolution semantic occupancy benchmark (3.6M frames) with a novel FBSA annotation pipeline.

4. Experimental Results

Extensive experiments on the Nuplan-Occ mini and full validation sets demonstrate superior performance:

• Occupancy: Achieves 98.5% mIoU in reconstruction and 33.41% mIoU in generation (full set), outperforming UniScene and OccSora significantly.
• Video: Achieves 7.59 FID and 61.42 FVD on the full validation set, surpassing state-of-the-art methods like Vista and MagicDrive.
• LiDAR: Achieves an MMD of 0.575 × 10⁻⁵ on the full set, a 54.25% improvement over UniScene.
• Downstream Utility: Synthetic data generated by UniScenev2 improves downstream planning metrics (No-At-Fault Collisions and Drivable Area Compliance) on the NAVSIM/Nuplan test set. Scaling the generated data volume from 10% to 100% of Nuplan-Occ yields compelling improvements in planning performance.
• Generalization: The model demonstrates strong generalization capabilities on in-house datasets with different sensor configurations (6 fisheye cameras + front LiDAR).

5. Significance and Claims

The paper claims that UniScenev2 represents a significant advancement in driving scene generation by addressing the critical bottlenecks of data scarcity and modality fragmentation. By leveraging the massive Nuplan-Occ dataset and a unified, occupancy-centric approach, the method enables the synthesis of diverse, high-fidelity 4D dynamic scenes. The authors assert that this framework not only achieves state-of-the-art generation fidelity but also provides practical value for autonomous driving research by enhancing the performance of downstream perception and planning tasks through scalable, high-quality synthetic data.