Low-Cost Neural Radiance Fields

Technical Summary: Low-Cost Neural Radiance Fields

Problem Statement
Neural Radiance Fields (NeRF) have established a high standard for novel-view synthesis but suffer from prohibitive training times and a heavy reliance on dense input views. These constraints limit the accessibility of NeRF for users with limited computational resources or data availability. The primary challenge addressed in this work is how to accelerate NeRF training and reduce data requirements without significantly sacrificing reconstruction quality.

Methodology
The authors conducted a comparative study of three accelerated NeRF variants—DS-NeRF, TensoRF, and HashNeRF—under constrained compute and data regimes. The study involved implementing these baselines and proposing specific architectural and training modifications:

1. TensoRF with Depth Supervision (TensoRF-DS):

   • Approach: The authors integrated a depth-supervised loss term into the TensoRF framework. Ground truth depth values were derived from COLMAP keypoints (3D locations and 2D pixel coordinates).
   • Loss Function: A Mean Squared Error (MSE) depth loss ($L_{\hat{D}}$) was calculated between the model's estimated depth and the COLMAP-derived ground truth. This was combined with the standard NeRF color loss ($L_{\hat{C}}$) using a weighting hyperparameter $\lambda_i$.
   • Dataset: Experiments were conducted on the LLFF dataset, utilizing a "low data" regime with only 9 images (4 training, 4 testing, 1 validation) compared to the full dataset.

2. TensoRF Ablation and Downsampling:

   • Downsampling: To fit within memory budgets and reduce runtime, the authors downsampled input images by a factor of 4.
   • Architecture Ablation: The feature-decoding Multi-Layer Perceptron (MLP) of TensoRF was modified by adding or removing linear layers to determine the impact on Peak Signal-to-Noise Ratio (PSNR) and runtime.

3. HashNeRF Architectural Variants:

   • Approach: The authors proposed four architectural modifications to the color and density networks following the multi-resolution hash encoding.
   • Variations:
     • Replacing linear layers with 1D convolutional layers to reduce parameters.
     • Adding residual connections from the initial input to subsequent layers in the sigma and color MLPs.
     • Adding residual connections from alternate layers (two layers prior) to preserve spatial information.
     • Deepening the network to fine-tune color variations.
   • Evaluation: Models were evaluated on the synthetic Lego scene and compared against the baseline using PSNR and training time at fixed iteration budgets (500 and 1000 iterations).

Key Results

• TensoRF-DS: The addition of depth supervision to TensoRF under low-data conditions yielded negligible improvement. While the low-data baseline suffered a significant performance drop (Test PSNR of 15.23 vs. 26.32 for full data), the depth-supervised variant only improved the Test PSNR by 0.01 (15.24). Visualizations indicated that while color outputs were similar, the underlying depth structure remained incomplete, and the depth loss gradients affected only a small portion of the voxel space due to the scarcity of COLMAP keypoints relative to pixel comparisons.
• TensoRF Downsampling: Downsampling inputs by a factor of 4 allowed the model to achieve a Test PSNR of 31.297 in just 2,500 iterations (trained in under 10 minutes), comparable to the performance of the original NeRF but with significantly reduced computational cost. Ablation studies on the feature-decoding MLP showed no significant performance gains from altering the layer count, suggesting the default architecture is already efficient.
• HashNeRF Variants: Among the four proposed architectural variants, the deepest network (Variation 4) achieved the highest PSNR of 27.119 at 1,000 iterations. However, this remained below the 28.2 PSNR reported by the original HashNeRF authors under their default configuration. When allowed to train to convergence, the proposed network surpassed the baseline, but under iso-time (fixed iteration) evaluation, it did not conclusively outperform the published baseline.

Significance and Claims
The paper concludes that under iso-time evaluation, none of the proposed extensions conclusively outperformed the published baselines. However, the work serves to characterize which extensions are viable for constrained settings and to surface design questions for future research.

• Practicality of Downsampling: The study validates downsampling as an effective technique for improving computational efficiency without significantly compromising quality, making TensoRF more accessible to researchers with limited hardware.
• Limitations of Depth Supervision: The results highlight that simply adding depth supervision to TensoRF with sparse data is insufficient; the scarcity of ground truth depth points limits the gradient propagation necessary to improve the volumetric reconstruction.
• Future Directions: The authors suggest that future work could involve calculating 3D ground truth meshes at more points to better utilize depth supervision, extending the iteration budget to verify if performance gaps close with more training, and conducting more thorough ablation studies on TensoRF components.

Ultimately, the paper positions itself as a comparative study that identifies the boundaries of current low-cost NeRF adaptations, noting that while specific improvements were not achieved under strict time constraints, the experiments provide a foundation for understanding how these models behave in resource-constrained environments.