Technical Summary: HarmoGS: Robust 3D Gaussian Splatting in the Wild via Conflict-Aware Gradient Harmonization

1. Problem Statement

Reconstructing reliable 3D representations from unconstrained "in-the-wild" image collections remains a significant challenge in computer vision. While recent advances like 3D Gaussian Splatting (3DGS) have achieved real-time, high-quality rendering, they typically assume consistent illumination and stable scene content. In real-world scenarios, training images are often corrupted by transient distractors (e.g., moving pedestrians, vehicles) and illumination-induced appearance inconsistencies across different views.

Existing methods primarily rely on image-level masking to suppress unreliable supervision. However, the authors identify two critical limitations in this approach:

1. Residual Occlusions: Masks often fail to fully exclude complex or semantically ambiguous transient objects, leaving residual noise.
2. Illumination Conflicts: Masks cannot resolve view-dependent lighting inconsistencies.

These unresolved issues lead to cross-view gradient conflicts. When optimizing a shared Gaussian representation using inconsistent views, the induced gradients may point in opposing directions (forming obtuse angles). This destabilizes the optimization process, causing visible reconstruction artifacts and floating primitives.

2. Methodology

The authors propose HarmoGS, a conflict-aware 3DGS framework that addresses these issues through a dual-view training strategy and three core components:

A. Semantic Consistency-Guided Masking

To improve the reliability of supervision before gradient formation, the method introduces an adaptive masking strategy that refines prior masks:

• Prior Mask: A binary mask ($M_{bin}$) is generated using a pre-trained SAM (Segment Anything Model) to identify potential transient regions.
• Learned Consistency Score: A lightweight predictor ($f_\psi$), guided by a frozen DINOv2 backbone, learns a pixel-wise consistency score ($S_i$) during training. This score is derived from both photometric residuals and semantic discrepancies between rendered and ground-truth images.
• Adaptive Refinement: The final mask ($M_i$) combines the binary prior and the learned consistency score. It adaptively suppresses unreliable regions (transient distractors) while correcting errors in the initial binary mask, ensuring that only reliable image evidence contributes to the loss calculation.

B. Conflict-Aware Gradient Harmonization

Recognizing that masking alone cannot eliminate all optimization conflicts, the framework employs a dual-view training approach. At each iteration, two views are sampled, and their specific gradients ($g_1, g_2$) are computed.

• Conflict Detection: A conflict is identified when the dot product of the two gradients is negative ($g_1^\top g_2 < 0$), indicating an obtuse angle ($\theta > \pi/2$).
• Orthogonal Rotation: To resolve the conflict, the method rotates both gradients within the subspace spanned by them ($V = \text{span}(g_1, g_2)$) until they become orthogonal ($\tilde{g}_1^\top \tilde{g}_2 = 0$).
  • The total angular correction ($\theta - \pi/2$) is distributed between the two gradients using an allocation parameter $\rho$.
  • The rotation preserves the original magnitudes of the gradients while eliminating negative directional interference.
• Linear Combination: The harmonized gradients are summed and expressed as a linear combination of the original gradients ($\tilde{g} = \tau_1 g_1 + \tau_2 g_2$) using analytically derived coefficients.
• Geometric Attenuation: For geometric attributes (position, rotation, scale), an additional attenuation factor is applied if conflicts are detected to further suppress artifacts.

C. Conflict-Aware Densification and Pruning

The framework extends standard 3DGS structural updates to be aware of cross-view conflicts:

• Densification: Instead of relying on single-view statistics, the method aggregates view-space gradients from both sampled views, modulated by the conflict-aware coefficients derived in the harmonization step. This guides Gaussian growth using more reliable, conflict-reconciled signals.
• Pruning: The method tracks per-Gaussian cross-view conflicts over time. If a Gaussian consistently exhibits high conflict scores (indicating persistent instability or floating artifacts), its opacity is periodically decayed. This allows the standard pruning process to naturally remove these unstable primitives.

3. Key Contributions

1. Conflict-Aware Gradient Harmonization Framework: A novel dual-view training strategy that explicitly reconciles cross-view gradient conflicts by rotating them into an orthogonal configuration. This reduces incompatible updates caused by transient distractors and illumination inconsistencies.
2. Semantic Consistency-Guided Masking: An adaptive masking mechanism that learns pixel-wise consistency scores to refine prior binary masks, effectively suppressing unreliable supervision before gradient formation.
3. Conflict-Aware Structural Updates: The introduction of densification and pruning strategies that utilize conflict-aware signals to stabilize Gaussian growth and remove persistently conflicting primitives.
4. State-of-the-Art Performance: Extensive experiments demonstrate that the method achieves superior rendering quality and reconstruction fidelity on standard in-the-wild benchmarks compared to existing NeRF and 3DGS baselines.

4. Experimental Results

The authors evaluated HarmoGS on three widely used in-the-wild benchmarks: PhotoTourism, NeRF On-the-go, and RobustNeRF.

• Quantitative Performance: The method consistently outperformed state-of-the-art baselines (including RobustNeRF, WildGaussian, SpotLessSplats, and AsymGS) across all metrics (PSNR, SSIM, LPIPS). For instance, on the NeRF On-the-go dataset, HarmoGS achieved a PSNR of 24.67 in high-occlusion scenes, surpassing the previous best (AsymGS) by a significant margin.
• Qualitative Results: Visual comparisons showed that HarmoGS effectively suppressed floating artifacts and produced cleaner renderings with more faithful scene details, particularly in scenes with complex transient occlusions and lighting variations.
• Ablation Studies:
  • Removing the dual-view setting or the gradient harmonization component led to a noticeable drop in PSNR and SSIM, confirming the necessity of conflict reconciliation.
  • The combination of the learned consistency score ($S_i$) and the binary prior mask ($M_{bin}$) yielded better results than using either component alone, demonstrating their complementary nature.
  • Conflict-aware densification and pruning resulted in a more compact Gaussian representation (fewer points) while maintaining higher quality, indicating the removal of unstable primitives.

5. Significance and Claims

The paper claims that HarmoGS provides a robust solution for 3D reconstruction in unconstrained environments by addressing the root cause of optimization instability: cross-view gradient conflicts.

• Beyond Masking: The work highlights that image-space masking is insufficient for in-the-wild 3DGS. By operating at the gradient level, the method resolves conflicts that persist even after masking, leading to more stable optimization.
• Dual-View Optimization: The authors posit that jointly considering two views allows for the explicit detection and reconciliation of conflicts, a capability absent in standard single-view optimization pipelines.
• Generalizability: The method is presented as a general framework compatible with existing 3DGS implementations (built upon Mip-Splatting) that significantly improves robustness against transient distractors and illumination changes without requiring complex scene-specific heuristics.

The authors conclude that their approach enables high-fidelity novel view synthesis in challenging real-world scenarios, advancing the applicability of 3D Gaussian Splatting to unconstrained image collections.