To View Transform or Not to View Transform: NeRF-based Pre-training Perspective

1. Problem Statement

The paper addresses a fundamental misalignment in current autonomous driving perception systems that combine Neural Radiance Fields (NeRFs) for pre-training with View Transformation (VT) backbones for downstream tasks.

• Conflicting Priors: Existing methods (e.g., UniPAD, SelfOcc) attempt to use NeRFs for self-supervised pre-training but rely on View Transformation backbones (which project 2D features into discrete 3D voxel grids) for the actual 3D representation.
  • NeRF Assumption: Continuous, adaptive, and fine-grained functions.
  • View Transformation Assumption: Discrete, rigid, and fixed-resolution voxel grids.
• Consequences:
  • Representation Degradation: Forcing continuous NeRF knowledge into a discrete voxel grid creates a "low-pass filter" effect, resulting in blurry, ambiguous 3D representations where distinct objects merge into indistinct blobs.
  • Knowledge Waste: In current pipelines, the pre-trained NeRF network is discarded after the pre-training phase. The downstream task (detection, occupancy) relies solely on the VT backbone, failing to inherit the enhanced geometric and appearance understanding learned by the NeRF.
  • Poor Generalization: Discrete grids are highly sensitive to sensor layout changes and domain shifts, leading to performance degradation when transferring between datasets (e.g., Argoverse 2 to nuScenes).

2. Methodology: NeRP3D

The authors propose NeRP3D (NeRF-Resembled Point-based 3D detector), a unified architecture that eliminates the View Transformation bottleneck and preserves the continuous nature of NeRFs throughout the entire pipeline.

Core Architecture

• Unified Continuous Representation: Instead of converting features to a voxel grid, NeRP3D models the 3D scene directly as continuous 3D features. It samples points at arbitrary spatial locations and predicts their representations using 2D image features.
• Adaptive Sampling Strategies:
  • Ray-wise Sampling (Pre-training): For volumetric rendering (RGB/Depth), points are sampled along camera rays to optimize the radiance field.
  • Uniform Spatial Sampling (Downstream): For perception tasks (detection, occupancy), points are scattered uniformly across the 3D space around the vehicle to ensure comprehensive coverage.
• Feature Extraction (Deformable Cross-Attention):
  • 3D query points are encoded and projected to 2D image planes.
  • A deformable cross-attention mechanism aggregates features from relevant 2D image regions to the 3D points, ensuring the 3D representation is contextually aligned with the 2D observations without rigid grid interpolation.
• Task-Agnostic Heads:
  • Pre-training: Uses a rendering head (RGB and Signed Distance Function/SDF) to minimize photometric and depth reconstruction losses.
  • Downstream: The same backbone feeds into task-specific heads (e.g., detection or occupancy heads) by reshaping the continuous point embeddings. Crucially, the NeRF network is not discarded; it remains part of the architecture, preserving the learned continuous geometry.

Loss Functions

The pre-training objective combines:

1. RGB Reconstruction Loss: Standard photometric loss.
2. Depth Supervision: Utilizes LiDAR measurements where available.
3. Multi-view Consistency: Enforces consistency across different camera views to handle sparse LiDAR and unobserved regions (e.g., sky, transparent surfaces).

3. Key Contributions

1. Resolution of Prior Conflict: NeRP3D is the first approach to fully inherit the continuous function of NeRFs for both pre-training and downstream tasks, eliminating the discrete/rigid priors of View Transformation.
2. Unified Framework: It provides a single architecture that supports adaptive sampling (ray-wise for rendering, spatial for perception) without discarding the pre-trained model.
3. Superior Representation Quality: By avoiding voxel discretization, the model learns fine-grained 3D geometry and appearance, leading to sharper object boundaries and better separation of crowded objects.
4. Robust Generalization: The continuous point-based architecture is inherently more robust to domain shifts and sensor configuration changes compared to fixed-grid methods.

4. Experimental Results

Experiments were conducted on the nuScenes and Argoverse 2 (AV2) datasets.

Scene Reconstruction (Pretext Tasks)

• RGB Reconstruction: NeRP3D achieved a PSNR of 33.42 and SSIM of 0.969, significantly outperforming UniPAD (PSNR 21.14) and SelfOcc (PSNR 18.82).
• Depth Estimation: NeRP3D showed lower error rates (Abs Rel: 0.183) compared to UniPAD (0.218) and SelfOcc (0.311), demonstrating superior ability to reconstruct fine structures like poles and thin objects.
• Resolution Sensitivity: Unlike View Transformation methods which degrade at higher resolutions due to grid limitations, NeRP3D's performance improved as resolution increased, proving its continuous nature.

Downstream Perception Tasks

• 3D Object Detection: On nuScenes, NeRP3D achieved 47.3 NDS and 42.8 mAP, outperforming UniPAD (45.5 NDS, 41.6 mAP) and other SOTA methods. It showed particular strength in detecting small objects and separating crowded instances.
• 3D Occupancy Prediction: NeRP3D achieved 35.49 mIoU, surpassing UniPAD (34.05) and SelfOcc (29.65). It excelled in distinguishing thin objects (bicycles, pedestrians) and complex boundaries.
• HD Map Construction: Achieved 59.1 mAP, outperforming UniPAD (57.8) and SelfOcc (53.9), indicating better understanding of static road elements.

Generalization (Zero-Shot Transfer)

• Cross-Dataset Transfer: Pre-trained on Argoverse 2 and tested on nuScenes without fine-tuning, NeRP3D maintained high fidelity (PSNR 28.24), whereas UniPAD failed completely (PSNR 18.67) due to its dependency on fixed sensor-specific grids.

5. Significance

• Paradigm Shift: The paper argues that View Transformation is an unnecessary and harmful bottleneck for NeRF-based pre-training. It advocates for a "Point-based" continuous approach that aligns the pre-training objective with the downstream task representation.
• Efficiency of Knowledge Transfer: By retaining the NeRF network, the model effectively transfers the "understanding" of 3D geometry and appearance directly to detection and occupancy tasks, rather than just using pre-training as a feature initializer.
• Real-World Applicability: The demonstrated robustness to sensor layout changes and domain shifts makes NeRP3D a more viable solution for real-world autonomous driving systems where sensor configurations and environments vary.

In conclusion, NeRP3D demonstrates that abandoning discrete view transformation in favor of a continuous, NeRF-resembled point-based architecture leads to superior 3D scene understanding, better generalization, and state-of-the-art performance across reconstruction and perception tasks.