DriveSplat: Unified Neural Gaussian Reconstruction for Dynamic Driving Scenes

1. Problem Statement

Reconstructing large-scale dynamic driving scenes is a significant challenge in autonomous driving simulation and 3D vision. Existing methods face several critical limitations:

• Scale and Depth Variation: Driving scenes span vast spatial extents with extreme depth variations (from close-range road markings to distant buildings). Existing Gaussian Splatting (3D-GS) methods often use single-scale densification, leading to redundant primitives in distant regions and poor geometric consistency under novel viewpoints.
• Dynamic Complexity: Scenes contain both rigid objects (vehicles) and non-rigid actors (pedestrians, cyclists) with complex motions. Current approaches often struggle to decouple static backgrounds from dynamic foregrounds while maintaining temporal stability.
• Geometric Fidelity: Many methods produce visually plausible images but lack accurate surface geometry (normals) and depth consistency, especially in distant or occluded areas. LiDAR supervision is often sparse and uneven, insufficient for supervising large-scale structures.

2. Methodology: DriveSplat

The authors propose DriveSplat, a unified neural Gaussian framework that reconstructs static backgrounds and dynamic actors within a single optimization loop. The pipeline consists of four core components:

A. Scene-Aware Multi-Scale Background Representation

To handle the extreme depth variation in driving scenes, the authors introduce a Learnable Level-of-Detail (LOD) strategy:

• Geometry-Guided Partitioning: The background is decomposed into three depth-ordered regions (Near, Mid, Far) based on the principal direction of the scene (aligned with the ego-vehicle's forward view).
• Multi-Scale Gaussian Representation: Instead of a uniform grid, the method uses a unified, learnable hash table where each spatial anchor is associated with level-specific latent features.
• View-Adaptive LOD Allocation: Rather than fixed heuristics, the effective LOD for each anchor is dynamically selected based on the current camera viewpoint. Anchors are rendered only if their assigned resolution level matches their visual relevance (distance to camera). This ensures fine-grained modeling for near objects and coherent global geometry for distant structures, reducing redundancy.
• Learnable Boundaries: The thresholds separating Near/Mid/Far regions are treated as learnable parameters, allowing the model to refine the partitioning during training to better align with view-dependent visibility.

B. Dynamic Actor Modeling

The framework models dynamic elements using an object-centric formulation that separates global motion from local deformation:

• Global Rigid Transformation: For each actor (rigid or non-rigid), a time-varying rigid transformation (rotation and translation) aligns the actor's canonical coordinate system to the world frame.
• Anchor-Centric Non-Rigid Deformation (AnchorFlow): For non-rigid actors (e.g., pedestrians), the method avoids dense 4D deformation fields. Instead, it uses a sparse set of anchors. An MLP (AnchorFlow) predicts the motion of these anchors over time.
• Deformation Propagation: The motion of the anchors is propagated to the associated neural Gaussians via a multi-head update scheme. A geometry head synchronizes spatial attributes (position, scale, rotation) with the anchor, while appearance heads update opacity and color independently. This ensures coherent motion while reducing the number of learnable parameters.

C. Geometry-Enhanced Optimization

To improve geometric accuracy and surface consistency, the method incorporates auxiliary supervision from pre-trained monocular models:

• Priors: It utilizes dense depth (from DepthAnything-V2 and ZoeDepth) and surface normal priors (from Omnidata).
• Loss Functions: The optimization includes a depth loss (correlation loss for relative depth, L1 for absolute) and a normal loss (L1 + cosine similarity) to enforce geometric consistency across scales and viewpoints.

D. Initialization

The system supports flexible initialization using SfM, LiDAR, or dense DUSt3R point clouds. In the absence of LiDAR, it uses COLMAP for static backgrounds and random initialization within bounding boxes for dynamic actors.

3. Key Contributions

1. Unified Representation: A single Gaussian-based framework that simultaneously models large-scale static backgrounds and diverse dynamic actors (rigid and non-rigid), avoiding ad-hoc decoupling strategies.
2. Scene-Aware LOD: A novel, learnable multi-scale modeling strategy that explicitly accounts for the near/mid/far structure of driving environments, enabling adaptive Gaussian allocation and eliminating redundant primitives in distant regions.
3. Anchor-Centric Deformation: A lightweight, efficient mechanism for modeling non-rigid dynamics by propagating motion from sparse anchors to neural Gaussians, ensuring temporal stability without the computational cost of dense 4D fields.
4. Geometry Priors: The integration of dense depth and normal priors significantly enhances geometric fidelity, producing reconstructions that are not only visually realistic but also geometrically consistent.

4. Experimental Results

The method was evaluated on the Waymo and KITTI datasets, outperforming state-of-the-art (SOTA) methods in both reconstruction and novel-view synthesis (NVS).

• Quantitative Performance (Waymo):
  • Achieved 33.83 dB PSNR and 0.923 SSIM in Novel View Synthesis, surpassing competitors like StreetGS (30.24 dB), OmniRe (31.19 dB), and AD-GS (33.08 dB).
  • Demonstrated superior performance in LPIPS (0.103), indicating better perceptual quality and fewer artifacts.
  • In sequences with non-rigid actors (pedestrians), DriveSplat achieved 29.46 dB PSNR* (dynamic regions), outperforming OmniRe (28.79 dB) and Deform-GS.
• Quantitative Performance (KITTI):
  • Achieved 24.53 dB PSNR in NVS, outperforming all baselines including OmniRe (20.86 dB) and Deform-GS (22.01 dB).
• Robustness to Viewpoint Shifts:
  • Under challenging viewpoint shifts (1m, 2m, 5m), DriveSplat maintained high-quality rendering with significantly lower FID and FVD scores compared to StreetGS and OmniRe, demonstrating superior generalization.
• Efficiency:
  • Training time: 69.6 minutes (significantly faster than Desire-GS at 182.9 mins).
  • Rendering speed: 73.5 FPS, enabling real-time applications.

5. Significance and Impact

DriveSplat addresses a critical gap in 3D reconstruction for autonomous driving by providing a scalable, unified solution that handles the unique challenges of driving scenes: extreme depth ranges and complex dynamic interactions.

• Simulation & Training: The high-fidelity, geometrically consistent reconstructions enable the creation of realistic training environments for perception and planning models.
• Efficiency: By optimizing the number of Gaussians through LOD and using anchor-centric deformation, it achieves SOTA quality with real-time rendering speeds, making it practical for large-scale deployment.
• Geometric Accuracy: The incorporation of depth and normal priors moves beyond purely photometric optimization, resulting in reconstructions that are structurally sound, which is vital for safety-critical applications.

In summary, DriveSplat sets a new benchmark for dynamic driving scene reconstruction, balancing high visual fidelity, geometric accuracy, and computational efficiency.