GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generation

1. Problem Statement

Developing photorealistic, editable, and free-viewpoint driving simulators is critical for training and evaluating end-to-end autonomous driving systems. Existing approaches face significant limitations:

• Reconstruction-based methods (e.g., PVG, OmniRe, S3Gaussian) rely on recorded trajectories. They struggle to render high-quality views from novel, off-trajectory viewpoints due to limited observations, often resulting in geometric distortions (e.g., "floaters") and artifacts.
• Generative/World Models (e.g., DriveDreamer-2, MaskGWM) can generate videos but lack explicit 3D scene representations. This leads to poor 3D consistency, inability to guarantee camera pose control, and a lack of scene editing capabilities.
• Hybrid methods attempt to combine generation and reconstruction but often suffer from blurred details, inconsistent 3D structures, and suboptimal editability.
• Data Scarcity: Training diffusion models for novel view synthesis typically requires datasets with multiple simultaneously recorded trajectories (to use one as condition and another as ground truth), which are unavailable in public driving datasets (usually single-trajectory per scene).

2. Methodology: GA-Drive Framework

GA-Drive proposes a novel framework that decouples scene geometry from appearance, leveraging a Geometry-Appearance Decoupling strategy and a Diffusion-Based Generation pipeline.

A. 4D Geometry Reconstruction (The Geometry Module)

The system first reconstructs the 4D scene geometry using a Gaussian Scene Graph (based on OmniRe). To address geometric ambiguities common in limited-viewpoint driving scenes, the authors introduce two key improvements:

1. Dense Depth Supervision: Since LiDAR data is sparse (often only covering the bottom half), they use Marigold to estimate relative depth maps and align them with absolute LiDAR scales. This provides dense geometric constraints across the entire image.
2. Depth Distortion Regularization: They incorporate a loss function (from Mip-NeRF 360) to encourage Gaussian weights to concentrate along rays, preventing the formation of floating artifacts ("floaters").

B. Geometry-Guided Pseudo-View Synthesis

Once the 4D geometry is established, the system generates "pseudo-views" for novel camera trajectories:

1. Ray Casting: Rays are cast from the target novel pose into the reconstructed 3D space using the rendered depth map to generate a 3D point cloud.
2. Projection & Sampling: These 3D points are projected onto the original recorded source views to sample RGB colors.
3. Visibility Handling: A visibility check ensures that points occluded by closer surfaces in the source view are masked (kept black) rather than incorrectly colored.
   Result: These pseudo-views contain correct 3D spatial relationships and occlusion patterns but may have holes or artifacts. They serve as strong geometric priors.

C. Segment-wise Video Diffusion Model

A trained video diffusion model transforms the geometrically correct but imperfect pseudo-views into photorealistic frames.

• Architecture: Based on CogVideoX (I2V), using a 3D VAE to encode conditioning frames.
• Segment-wise Generation: To generate long trajectories, the model processes the video in segments. The first frame of a new segment is conditioned on the last generated frame of the previous segment to ensure temporal consistency, while the remaining frames are conditioned on the corresponding pseudo-views.

D. Pseudo-View Simulation Pipeline (Training Strategy)

To train the diffusion model without multi-trajectory datasets, the authors designed a simulation pipeline that mimics the characteristics of novel pseudo-views using only a single trajectory:

1. Pattern Simulation: The pipeline artificially introduces:
   • Random Masks: Simulating projection errors.
   • Local Blending: Simulating depth estimation inaccuracies.
   • Occluded Regions: Masking areas where depth changes sharply.
2. Training Loop: The model is trained to take these "simulated" pseudo-views (as conditions) and the original recorded frames (as ground truth) to learn how to fill in missing details and correct artifacts while preserving 3D consistency.

3. Key Contributions

1. Geometry-Appearance Decoupling: A framework that separates 3D structure reconstruction from appearance synthesis. This allows for flexible appearance editing (e.g., changing weather, car colors) via state-of-the-art video editing tools without affecting the underlying geometry, ensuring consistency across all generated trajectories.
2. Pseudo-View Simulation Pipeline: A novel training strategy that enables the use of single-trajectory datasets (like Waymo) to train video diffusion models for novel view synthesis, eliminating the need for expensive multi-camera simultaneous recordings.
3. Enhanced 4D Reconstruction: The introduction of dense depth supervision and distortion regularization significantly improves geometric accuracy in LiDAR-sparse regions compared to baseline reconstruction methods.

4. Experimental Results

The authors evaluated GA-Drive against state-of-the-art baselines (PVG, S3Gaussian, DriveDreamer4D, ReconDreamer, Deformable-GS) on the Waymo validation set.

• Quantitative Metrics: GA-Drive achieved superior performance in:
  • NTA-IoU (Novel Trajectory Agent IoU): 0.558 (vs. 0.443 for the next best), indicating better vehicle positioning.
  • NTL-IoU (Novel Trajectory Lane IoU): 56.83 (vs. 53.78), showing better lane boundary consistency.
  • FID (Fréchet Inception Distance): 49.77 (lower is better), demonstrating significantly higher photorealism compared to baselines (e.g., 76.24 for ReconDreamer).
• Qualitative Results: Visual comparisons show GA-Drive produces sharper details, fewer artifacts, and better temporal coherence than hybrid methods. It successfully handles lane shifts and maintains consistency where other methods fail (e.g., shape inconsistencies in DriveDreamer4D).
• Robustness: The model remains robust even when the underlying 3D Gaussian representation is corrupted (e.g., removing 80% of Gaussians or adding noise), thanks to the generalization capability of the diffusion model and the simulation pipeline.
• Editing Capability: The system successfully demonstrated editing tasks, such as changing weather to fog or altering car appearances, with consistent results across different camera trajectories.

5. Significance

GA-Drive represents a significant advancement in autonomous driving simulation by solving the trade-off between geometric accuracy, photorealism, and editability.

• Safety & Training: It provides a scalable way to generate diverse, high-fidelity driving scenarios (including rare weather conditions and novel camera angles) for training end-to-end autonomous systems.
• Efficiency: By decoupling geometry and appearance and utilizing a simulation pipeline, it bypasses the need for costly multi-trajectory data collection.
• Flexibility: The ability to edit scene appearance globally (across all trajectories) while preserving 3D structure opens new avenues for data augmentation and scenario testing in autonomous driving research.