Technical Summary: SSA-3DGS

Problem Statement

Novel View Synthesis (NVS) methods, particularly 3D Gaussian Splatting (3DGS), rely on the assumption that input images are clean and multi-view consistent. However, real-world captures often contain screen-space artifacts: static occlusions fixed to the 2D image plane rather than the 3D world. Common examples include physical sensor defects, environmental obstructions (rain, mud on lens enclosures), capture obstructions (fingers, dashboards), and digital overlays (watermarks, UI elements).

Because these artifacts are static relative to the camera sensor, they "move" relative to the world geometry as the camera traverses the scene. Standard 3D reconstruction algorithms, which assume multi-view consistency, fail to reject these artifacts. Instead, they erroneously bake them into the 3D geometry as "floaters" (near-camera 3D Gaussians) or view-dependent effects, severely degrading the fidelity of novel-view renders. Existing robustness methods (e.g., NeRF-W, RobustNeRF, DeSplat) primarily target world-space transients or moving objects and struggle to distinguish static screen-space noise from scene geometry. Furthermore, single-image restoration techniques lack 3D structural knowledge, leading to temporal flickering and geometric inconsistencies when applied as a pre-processing step.

Methodology: SSA-3DGS

The authors propose SSA-3DGS, an unsupervised framework that jointly optimizes a 3D scene representation and a learnable 2D overlay to disentangle screen-space artifacts from the underlying geometry.

Core Formulation

The method extends the standard 3DGS pipeline by introducing a learnable 2D overlay shared across all training views. The final predicted pixel color $\hat{C}(u)$ is modeled as an alpha-blend between the clean 3D scene render $\hat{C}_{3D}(u)$ and the artifact overlay:
$$\hat{C}(u) = A(u) \cdot W(u) + (1 - A(u)) \cdot \hat{C}_{3D}(u)$$
Where:

• $W \in \mathbb{R}^{H \times W \times 3}$ is the learnable Artifact Color Map.
• $A \in \mathbb{R}^{H \times W \times 1}$ is the learnable Artifact Alpha Matte.
• Both $W$ and $A$ are defined in pixel coordinates and remain constant for all camera poses, unlike the 3D Gaussians which project differently per view.

Optimization and Regularization

The decomposition is under-constrained; without regularization, the model could degenerate into solutions where the 3D scene is over-brightened and dimmed by a uniform overlay, or where valid scene details are absorbed into the overlay. To prevent this, the authors introduce a composite loss function:
$$\mathcal{L} = \mathcal{L}_{rgb} + \lambda_{sparse}\mathcal{L}_{sparse} + \lambda_{tv}\mathcal{L}_{TV}$$

1. Photometric Loss ($\mathcal{L}_{rgb}$): Standard L1 and SSIM loss between the composited prediction and the ground truth.
2. Sparsity Regularization ($\mathcal{L}_{sparse}$): Encourages the alpha matte $A$ to be zero wherever possible. This biases the optimization to explain pixels via 3D geometry first, using the overlay only when necessary. It also prevents the "exposure collapse" degenerate solution.
3. Total Variation Loss ($\mathcal{L}_{TV}$): Encourages the overlay to consist of piecewise-smooth regions. This suppresses high-frequency noise absorption and enforces spatial coherence, reflecting the nature of many real artifacts (e.g., watermarks, mud splatters).

Initialization

Instead of initializing the overlay with constants, the method seeds $W$ and $A$ using per-pixel statistics from the training set. Pixels with stable intensity across views (likely artifacts) start nearly opaque, while pixels with high multi-view variation (likely scene content) start near-transparent.

Key Contributions

1. Unsupervised Disentanglement: A method to recover a clean 3DGS scene and segment the screen-space artifact simultaneously from corrupted multi-view recordings, without ground-truth masks or manual supervision.
2. Regularization Scheme: A novel combination of sparsity and total-variation constraints that biases the optimization toward 3D geometric explanations, ensuring the artifact layer captures only true screen-space noise.
3. Real-World Dataset: The introduction of a self-captured real-world dataset containing genuine, physically induced artifacts (mud on a lens housing, rig occlusions) to benchmark the method beyond synthetic composites.

Results

The authors evaluated SSA-3DGS on the Mip-NeRF 360 dataset (with synthetic artifacts) and their self-captured real-world dataset.

• Quantitative Performance: On the synthetic benchmark, SSA-3DGS improved reconstruction fidelity by up to 9 dB PSNR over vanilla 3DGS trained on the same corrupted inputs. It consistently outperformed the baseline 3DGS and the state-of-the-art distractor-robust method, DeSplat.
  • With MCMC densification, SSA-3DGS achieved a mean PSNR of 28.43 dB on corrupted inputs, approaching the performance of 3DGS trained on clean data (29.57 dB).
  • On the real-world "Mud" scene, the method improved PSNR by 3.0 dB over the strongest baseline (DeSplat).
• Qualitative Performance: Visual results demonstrate that SSA-3DGS successfully removes artifacts (watermarks, dashboards, mud, UI elements) and recovers clean geometry, whereas baselines often bake the artifacts into the scene as semi-transparent floaters. The method also successfully reconstructs the artifact itself as a separate overlay.
• Ablation: The study confirms that both sparsity and total-variation regularization are necessary. Removing them resulted in a drop of ~1.0 dB in PSNR and degraded FLIP scores, with the model failing to properly separate the artifact from the scene.

Significance and Claims

The paper claims that SSA-3DGS offers a viable path toward reliable 3D reconstruction from unconstrained, "in-the-wild" captures by explicitly modeling capture imperfections. Unlike generic 2D denoising applied as a pre-processing step, which breaks multi-view consistency, SSA-3DGS preserves geometric coherence by integrating the separation logic directly into the 3D optimization loop.

The authors note that the method relies on multi-view consensus; therefore, sufficiently dense captures are required. They acknowledge a limitation regarding artifacts that are only approximately static (e.g., occluders with view-dependent reflections), which are only partially captured by their static-overlay model, though the method still degrades gracefully and outperforms baselines in these scenarios. The work suggests that explicitly modeling sensor-plane obstructions is a critical step for robust NVS in real-world applications.