Technical Summary: StereoGS

Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized real-time novel view synthesis (NVS) in dense-view settings, it suffers from severe overfitting and defective geometry when applied to sparse-view scenarios. In sparse settings, limited view overlap and insufficient appearance correspondences fail to provide adequate geometric constraints. Existing attempts to mitigate this using monocular depth priors face two fundamental limitations:

1. Scale Ambiguity: Monocular depth estimates are relative rather than absolute. Enforcing these as regularization often conflicts with photometric consistency derived from input images, leading to misguidance during optimization.
2. Cross-View Inconsistency: Monocular depth maps are inferred independently for each view, resulting in inconsistent depth values for the same spatial point across different viewpoints. When back-projected, these inconsistencies cause spatial misalignments and geometric artifacts.

Furthermore, existing methods that attempt to enforce cross-view consistency (e.g., via photometric loss on warped images) often lack direct gradient guidance to 3D Gaussian primitives, failing to anchor them accurately to scene surfaces.

Methodology

StereoGS proposes a sparse-view 3DGS framework that integrates stereo priors to establish reliable binocular consistency and absolute scale constraints during optimization. The method consists of three core components:

1. Consistency-Aware Dense Initialization

To overcome the sparsity and noise of standard Structure-from-Motion (SfM) point clouds in sparse settings, StereoGS employs a zero-shot multi-view depth estimator (MVSAnywhere) combined with strict geometric filtering.

• For each training view, the model estimates depth maps using the remaining images as source views.
• Cross-view reprojection errors are used to filter outliers.
• The filtered depth maps are back-projected and fused into a dense, multi-view-consistent point cloud, providing a robust geometric foundation for the Gaussians.

2. Stereo Depth Regularization

Unlike methods relying on scale-agnostic monocular constraints, StereoGS introduces a Stereo Depth Regularization term to enforce absolute scale and binocular consistency.

• Virtual Stereo Pair Construction: During optimization, a virtual right camera is constructed for each training view (treated as the left camera) via horizontal translation.
• Reference Depth Generation: The ground-truth left image and the rendered right-view image are fed into a foundation stereo model (FoundationStereo) to generate a highly accurate reference disparity map. This serves as an external, scale-aware prior.
• Consistency Filtering: A left-right consistency check is applied to the estimated disparity to identify occlusions and false matches, generating a validity mask ($M_{valid}$).
• Loss Function: The method minimizes the L1 error in inverse depth space between the rendered depth and the stereo-derived depth ($Z_{stereo}$) for valid pixels:
  $$\mathcal{L}_{depth} = \lVert M_{valid} \odot (\frac{1}{\hat{Z}} - \frac{1}{Z_{stereo}}) \rVert_1$$
  This process compels the 3D Gaussians to learn absolute scale and consistent structures.

3. Gradient-Aware Opacity Decay

To suppress overfitting and eliminate redundant primitives (floaters) without indiscriminately removing stable surface structures, StereoGS replaces periodic opacity resets with a dynamic strategy.

• Core Insight: The magnitude of the opacity gradient ($\nabla_\alpha \mathcal{L}$) reflects a Gaussian's necessity. Large gradients indicate critical contributions to reducing photometric error, while negligible gradients suggest redundancy.
• Mechanism: A relative gradient $\beta$ is computed for each Gaussian by normalizing its gradient magnitude against the mean gradient magnitude of all Gaussians.
• Decay Factor: A dynamic decay factor $\gamma$ is derived using an exponential soft-thresholding function based on $\beta$. Gaussians with small relative gradients are penalized more heavily, while those with large update demands are preserved.
  $$\gamma = 1 - (1 - \gamma_{base}) \exp(-s \cdot \beta)$$
  The opacity is then scaled as $\hat{\alpha} = \gamma \alpha$.

Key Contributions

1. Stereo Depth Regularization: A novel mechanism that utilizes foundation stereo models to enforce absolute scale constraints and binocular consistency during optimization, resolving the scale ambiguity inherent in monocular priors.
2. Gradient-Aware Opacity Decay: A dynamic pruning strategy that adjusts opacity based on relative gradient magnitudes, effectively eliminating redundant primitives while preserving essential scene geometry.
3. Consistency-Aware Dense Initialization: A robust initialization pipeline leveraging zero-shot multi-view depth estimation and geometric filtering to create a high-quality point cloud foundation.
4. State-of-the-Art Performance: The method achieves superior results in sparse-view settings without incurring additional inference overhead, as all proposed strategies are training-time only.

Experimental Results

StereoGS was evaluated on four standard datasets: LLFF, DTU, Mip-NeRF360, and Blender, using sparse training views (e.g., 3, 6, 9 views for LLFF/DTU; 12, 24 views for Mip-NeRF360).

• Quantitative Performance: StereoGS consistently outperforms existing NeRF-based (e.g., RegNeRF, FreeNeRF) and 3DGS-based (e.g., DNGaussian, FSGS, Binocular3DGS) methods across PSNR, SSIM, and LPIPS metrics. For instance, on the LLFF dataset with 3 views, StereoGS achieved a PSNR of 21.91, surpassing the previous best (Binocular3DGS at 21.44).
• Qualitative Improvements: Visual comparisons demonstrate that StereoGS significantly reduces artifacts in textureless regions and preserves fine-grained details (e.g., object boundaries, grass textures) that baseline methods fail to reconstruct. The depth maps are more coherent and accurate, with fewer holes and floaters.
• Ablation Studies: Experiments confirmed that each component (Dense Initialization, Stereo Regularization, and Gradient-Aware Decay) contributes incrementally to performance. Removing any module resulted in clear degradation in both quantitative metrics and visual quality.

Significance and Claims

The paper claims that StereoGS fundamentally addresses the critical challenges of scale ambiguity and geometric inconsistency in sparse-view 3D Gaussian Splatting. By integrating foundation stereo models and a gradient-aware pruning strategy, the framework establishes a robust geometric foundation that allows 3DGS to achieve state-of-the-art novel view synthesis quality in sparse-view conditions.

A key claim is that these improvements are achieved without additional inference overhead, maintaining the real-time rendering efficiency of vanilla 3DGS. The authors acknowledge limitations, noting that stereo models may still struggle in extremely textureless regions and that the online regularization incurs additional training time, identifying these as directions for future research.