Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting

1. Problem Statement

3D Gaussian Splatting (3DGS) has become a standard for real-time rendering and rapid scene reconstruction due to its speed and expressiveness. However, standard 3DGS relies on rasterization, which imposes inherent limitations:

• Rigid Camera Assumptions: It struggles with non-pinhole models (e.g., fisheye lenses).
• Lighting Artifacts: It produces inaccurate shadows and lacks native support for reflections or refractions.
• Relighting Limitations: Existing "relightable" 3DGS methods often rely on rasterization-style approximations (e.g., shadow mapping) to handle lighting, undermining the physical accuracy promised by ray tracing.

Conversely, Ray-Traced 3DGS methods overcome these limitations by intersecting rays with Gaussians directly. However, they suffer from a critical bottleneck: sorting. To correctly alpha-blend intersecting Gaussians along a ray, all intersections must be sorted by depth. This sorting is computationally expensive, making ray-traced reconstruction significantly slower than rasterization-based methods. Furthermore, existing differentiable ray-tracing methods for reconstruction are either non-differentiable or rely on expensive sorting during the backward pass.

2. Methodology

The authors propose a differentiable, sorting-free stochastic formulation for ray-traced 3DGS. The core innovation is replacing the deterministic, sorted alpha-blending with an unbiased Monte Carlo estimator that samples a small subset of Gaussians per ray.

A. Stochastic Forward Rendering

Instead of sorting all $N$ Gaussians intersecting a ray, the algorithm traverses them in arbitrary order. It maintains a running selection of the "visible" Gaussian based on a probabilistic decision:

• For each Gaussian $g_i$ with opacity $\alpha_i$ and depth $z_i$, the algorithm updates the current selection with probability $\alpha_i$ if $g_i$ is closer than the current selection.
• This process yields an unbiased estimate of the pixel color without ever sorting the Gaussians.

B. Stochastic Gradient Estimation (The Core Contribution)

Reconstructing the scene requires computing gradients of the pixel color with respect to Gaussian parameters (position, covariance, opacity, color). The paper addresses the challenge that the stochastic selection process involves a discontinuous branching decision (the "if" statement in the selection logic), which standard automatic differentiation cannot handle correctly.

The authors derive unbiased Monte Carlo estimators for the gradients:

1. Color Gradients ($\partial C / \partial c_i$): The estimator selects a Gaussian $I$ with probability proportional to its contribution to the final color. The gradient for this specific component is set to 1, and others to 0.
2. Opacity Gradients ($\partial C / \partial \alpha_i$): This is more complex. The gradient depends on the difference between the selected Gaussian's color and the blended color of all Gaussians behind it.
   • To avoid sorting the Gaussians behind the selected one, the method applies a second layer of stochastic sampling. It draws a second index $K$ from the Gaussians behind $I$ using a conditional probability distribution.
   • The opacity gradient is then estimated simply as the difference in colors: $(c_I - c_K) / \alpha_I$.
   • This eliminates the need for sorting in the backward pass, making the entire pipeline sorting-free.

C. Relightable 3DGS Extension

The framework extends naturally to relightable scenes:

• Formulation: Instead of fixed colors, each Gaussian has a latent feature vector decoded by a neural network to determine color based on incident lighting (BRDF).
• Shadow Rays: To compute lighting, the method traces shadow rays. The same stochastic estimator is used to calculate transmittance (visibility) along these shadow rays.
• Advantage: Unlike prior work that uses shadow maps or learned approximations, this method traces exact shadow rays stochastically, enabling physically accurate shadows and environmental lighting without specialized rasterization passes.

3. Key Contributions

1. First Differentiable Stochastic Ray Tracing for 3DGS: The first framework to use stochastic ray tracing for both reconstruction and rendering of standard and relightable 3DGS scenes.
2. Unbiased Sorting-Free Estimator: A novel Monte Carlo algorithm that estimates pixel-color gradients without sorting Gaussians, bypassing the primary bottleneck of ray-traced 3DGS.
3. Relightable Reconstruction: An extension that drives per-Gaussian shading with fully ray-traced shadow rays, replacing approximate shadow-mapping pipelines with physically accurate light transport.
4. Performance: The method matches the speed of rasterization-based 3DGS while substantially outperforming sorting-based ray tracing methods.

4. Results

The authors evaluated their method on standard benchmarks (MipNeRF360, Tanks & Temples, Deep Blending) and relightable datasets (NRHints).

• Novel View Synthesis:

  • Speed: The method is significantly faster than sorting-based ray tracing (3DGRT). On MipNeRF360, it reduces iteration time by more than half compared to 3DGRT (39.8ms vs 87.5ms total time).
  • Quality: It achieves reconstruction quality (PSNR/SSIM) comparable to standard rasterization-based 3DGS and superior to sorting-based ray tracing.
  • Comparison: It outperforms StochasticSplats (a rasterization-based stochastic method) in reconstruction quality, as StochasticSplats suffers from high variance due to near-singular opacity terms in its gradient estimator.

• Relightable Scenes:

  • Fidelity: The method produces significantly higher reconstruction fidelity for relightable scenes compared to baselines like RNG and GS3.
  • Shadows: It generates high-quality, physically accurate shadows and geometry without dedicated shadow-handling modules.
  • Generalization: It successfully relights scenes under novel environmental illumination (IBL) despite being trained only on point lights, demonstrating the robustness of the ray-traced formulation.

5. Significance

This work bridges the gap between the speed of rasterization and the physical correctness of ray tracing in 3D Gaussian Splatting.

• Efficiency: By removing the sorting requirement, it makes differentiable ray tracing feasible for large-scale scene reconstruction, a task previously too slow for practical use.
• Generality: It provides a unified framework for standard and relightable scenes, eliminating the need for hybrid approaches that mix rasterization approximations with ray tracing.
• Future Impact: The ability to perform accurate, differentiable ray tracing opens the door for high-fidelity applications in autonomous driving, digital twins, and immersive reality where accurate lighting, reflections, and non-standard camera models are critical.