3DGS$^2$-TR: Scalable Second-Order Trust-Region Method for 3D Gaussian Splatting

Technical Summary: 3DGS2-TR

Problem Statement
While 3D Gaussian Splatting (3DGS) has established itself as the state-of-the-art for real-time novel view synthesis, the training process remains a significant bottleneck. Standard 3DGS relies on the first-order ADAM optimizer, which often suffers from slow convergence in the highly non-convex and ill-conditioned parameter spaces characteristic of 3DGS. The optimization landscape is particularly challenging due to the strong coupling between geometry (position, rotation, scale) and appearance (opacity, color) parameters, as well as the nonlinearity and piecewise continuity of the rasterization process (caused by depth-based sorting).

Existing second-order approaches (e.g., 3DGS-LM, 3DGS2) aim to accelerate convergence by utilizing curvature information but face critical scalability issues. They typically rely on explicit or dense curvature representations, leading to high memory overhead ($O(n^2)$ or worse) and expensive per-iteration costs. This makes them unsuitable for large-scale scenes or distributed training settings. Furthermore, the discontinuities in the 3DGS loss function violate the smoothness assumptions required by classical second-order methods, often resulting in instability.

Methodology: 3DGS2-TR
The authors propose 3DGS2-TR, a scalable second-order trust-region optimizer designed to accelerate 3DGS training while maintaining the computational and memory efficiency of first-order methods. The method is built upon three core principles: cheap per-iteration computation, parameter-linear memory scaling, and trust-region constraints.

1. Diagonal Hessian Approximation (Matrix-Free):
   Instead of computing full Hessian matrices, 3DGS2-TR approximates curvature using only the diagonal of the Gauss-Newton matrix ($J^T J$). This is estimated efficiently via Hutchinson's method, which utilizes random Rademacher vectors to compute Hessian-vector products. This approach ensures that both computation and memory complexity remain $O(n)$, matching the complexity of ADAM. To stabilize the curvature estimates, the method employs an Exponential Moving Average (EMA) of the diagonal estimates, updated periodically (e.g., every 10 iterations) rather than at every step.

2. Parameter-Wise Trust-Region via Squared Hellinger Distance:
   To address the strong nonlinearity and discontinuities inherent in 3DGS rasterization, the authors introduce a parameter-wise trust-region technique. Unlike global or group-level constraints, this method bounds the update step for each individual Gaussian parameter.

   • Metric: The constraint is based on the squared Hellinger distance between the Gaussian distribution before and after an update. This metric is chosen because it provides a closed-form solution for unnormalized Gaussians and intuitively measures the discrepancy in the rendered appearance.
   • Normalization: To account for the fact that a Gaussian's apparent mass can change arbitrarily with viewing distance, the Hellinger distance is normalized by the determinant of the scale matrix ($\det(S)^{-1}$).
   • Mechanism: For each parameter, a trust-region radius $\eta$ is computed such that the normalized squared Hellinger distance remains below a threshold $\epsilon$. The proposed update step is then clipped to this radius ($\Delta x \cdot \text{clip}(-\eta, +\eta)$), ensuring that updates do not cause abrupt, invalid changes to the Gaussian's geometry or opacity.

3. Optimization Pipeline:
   The algorithm follows a stochastic Gauss-Newton formulation. It computes a stochastic gradient and a diagonal Hessian estimate (via Hutchinson's method), calculates a second-order update step ($\Delta x = -\hat{D}^{-1} \hat{g}$), applies the parameter-wise trust-region clipping, and updates the parameters. The method requires only a modest overhead (approximately 10% more than ADAM) due to the periodic sampling for Hessian estimation.

Key Contributions

• Scalable Second-Order Optimization: The paper presents the first second-order optimizer for 3DGS that achieves $O(n)$ complexity in both computation and memory, making it scalable to very large scenes and potentially distributed training settings.
• Diagonal Hessian Estimation: By adapting the lightweight framework of Sophia (Liu et al., 2023) to 3DGS, the method utilizes diagonal Hessian estimates to account for loss surface curvature without the prohibitive costs of full matrix operations.
• Hellinger Distance Trust-Region: The authors propose a novel, parameter-wise trust-region constraint based on the squared Hellinger distance. This provides a principled way to bound geometric changes in Gaussian primitives, addressing the instability caused by the non-smooth nature of 3DGS rasterization.

Results
Under identical parameter initialization and without the use of densification strategies, 3DGS2-TR demonstrates superior performance compared to the standard ADAM optimizer:

• Convergence Speed: It achieves better reconstruction quality on standard datasets using 50% fewer training iterations than ADAM.
• Memory Efficiency: The method incurs less than 1GB of peak GPU memory overhead (approximately 17% more than ADAM). In contrast, existing second-order methods like 3DGS-LM incur roughly 85% more memory overhead, highlighting the scalability advantage of 3DGS2-TR.

Significance
The paper claims that 3DGS2-TR bridges the gap between the convergence benefits of second-order methods and the scalability requirements of large-scale 3D content creation. By avoiding dense curvature representations and introducing a stability mechanism tailored to the specific discontinuities of Gaussian splatting, the method enables faster training times without sacrificing reconstruction quality or memory efficiency. This advancement addresses a primary bottleneck in the 3DGS pipeline, potentially facilitating broader adoption in applications requiring rapid scene reconstruction, such as virtual reality, augmented reality, and robotics.