DenoiseSplat: Feed-Forward Gaussian Splatting for Noisy 3D Scene Reconstruction

The paper proposes DenoiseSplat, a feed-forward 3D Gaussian splatting method that achieves robust novel-view synthesis from noisy multi-view images by training end-to-end on a newly constructed noisy-clean benchmark without requiring 3D ground truth.

The Gist

Here is an explanation of the paper "DenoiseSplat" using simple language, creative analogies, and everyday metaphors.

🎥 The Big Picture: Building a 3D World from a Messy Video

Imagine you want to build a perfect, high-definition 3D model of a room using only a few photos taken with a shaky, old smartphone.

• The Goal: You want to walk around inside that 3D model and look at it from angles you never actually photographed (this is called "Novel View Synthesis").
• The Problem: Real-world photos are rarely perfect. They have graininess (noise), compression artifacts (blockiness), and blur.
• The Old Way: Most 3D reconstruction tools are like perfectionist architects. They say, "If your blueprints (photos) are smudged or torn, we can't build the house." They either refuse to work or build a shaky, distorted mess.
• The "Two-Step" Fix: Some people try to fix the photos first. They use a photo editor to clean up the grain, then feed the clean photos to the architect. But this is like cleaning a window with a rag that leaves lint behind; you might lose the fine details of the view while trying to clean it.

DenoiseSplat is a new invention that changes the game. It's like hiring an architect who is also a master restorer. They can look at your messy, grainy photos and say, "I can see the house underneath the dirt. I'll build the 3D model while ignoring the noise."

---

🧩 How It Works: The "Dual-Brain" Approach

The paper introduces a method called DenoiseSplat. Here is how it works, broken down into simple concepts:

1. The Training Ground: A "Noisy" Gym

To teach this AI how to handle bad photos, the researchers created a special gym.

• They took a huge dataset of clean 3D scenes (RealEstate10K).
• They artificially "dirty" the photos by adding four types of digital noise:
  • Gaussian: Like TV static.
  • Poisson: Like grain in a dark photo.
  • Speckle: Like raindrops on a lens.
  • Salt-and-Pepper: Like random black and white dots.
• The Rule: If a scene has noise, every photo of that scene gets the same type of noise. This mimics real life, where a camera sensor doesn't change its "graininess" just because you turn your head.

2. The Secret Sauce: The "Dual-Branch" Head

This is the paper's biggest innovation. Imagine the AI has two different jobs to do, and it splits its brain to handle them separately:

• Brain A (The Geometer): This part cares about structure. "Where are the walls? How big is the table? Where is the door?"
  • Analogy: Think of this as the skeleton of the 3D model. Bones don't care if the skin is dirty; they just need to know where the joints are. This brain ignores the color noise and focuses on the stable shapes.
• Brain B (The Artist): This part cares about appearance. "What color is the wall? Is it shiny? What is the texture?"
  • Analogy: This is the skin and paint. It takes the skeleton from Brain A and paints it. Because it knows the skeleton is solid, it can "guess" the correct colors even if the photo is noisy, filling in the gaps without getting confused by the static.

By separating these tasks, the AI doesn't get confused. The "Skeleton" stays strong and accurate, while the "Skin" gets cleaned up and smoothed out.

3. The "Cross-Branch" Safety Net (CBC)

Sometimes, even the Skeleton brain gets a little unsure near sharp edges (like the corner of a table).

• The system has a safety mechanism called CBC.
• Analogy: Imagine a construction foreman. If the "Skeleton" team says, "I'm not 100% sure about this corner," the foreman tells the "Artist" team: "Be careful here! Don't paint over the edge too roughly."
• This ensures that the final 3D model doesn't have blurry or weird edges, even when the input photos are terrible.

---

🏆 The Results: Why It's Better

The researchers tested DenoiseSplat against two other methods:

1. The "Ignore It" Method: Feeding noisy photos directly into a standard 3D tool.
   • Result: The 3D model looked like a melted wax figure. Everything was blurry and distorted.
2. The "Clean First" Method: Cleaning the photos with a standard AI, then building the 3D model.
   • Result: The photos looked clean, but the 3D model lost fine details (like the weave of a carpet or the texture of a brick). The cleaning process smoothed things out too much.

DenoiseSplat won because:

• It kept the structure sharp (the walls were straight).
• It kept the details crisp (you could still see the texture).
• It did it in one step. It didn't need to stop and clean the photos first; it just built the 3D world directly from the messy inputs.

🚀 Why This Matters

Think about Virtual Reality (VR) or Robotics.

• If a robot is navigating a dark, dusty warehouse, its camera will be noisy.
• If a VR creator is scanning a room with a cheap phone, the photos will be grainy.

Old tools would fail or produce a glitchy mess. DenoiseSplat allows us to take "imperfect" real-world data and turn it into a perfect, high-quality 3D world instantly. It's like having a magic camera that sees through the dirt to reveal the true shape of the world.

💡 Summary in One Sentence

DenoiseSplat is a smart 3D builder that doesn't need clean photos; it uses a special "two-brain" system to separate the shape of objects from the noise in the image, allowing it to build perfect 3D worlds even from grainy, messy snapshots.

Technical Summary

Here is a detailed technical summary of the paper "DenoiseSplat: Feed-Forward Gaussian Splatting for Noisy 3D Scene Reconstruction."

1. Problem Statement

Current state-of-the-art 3D reconstruction methods, particularly Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), typically assume that input images are clean and captured under controlled conditions. However, real-world multi-view data (e.g., from web videos or mobile devices) often suffers from various noise types (sensor noise, compression artifacts, low-light degradation).

Existing approaches to handle noisy inputs face significant limitations:

• Direct Application: Feeding noisy images directly into standard 3DGS pipelines (like MVSplat) causes rapid degradation in geometry estimation and texture quality.
• Two-Stage Pipelines: A common engineering solution involves denoising images in 2D first (using models like IDF or DnCNN) and then feeding them into a 3D reconstructor. This approach suffers from:
  • Over-smoothing and loss of fine details.
  • Multi-view inconsistency: Independent 2D denoising of each view can break the geometric consistency required for reliable 3D fusion.
  • Increased inference latency and system complexity.

Goal: Develop a method that performs 3D reconstruction directly from noisy multi-view inputs without per-scene optimization, preserving geometric consistency and texture details while suppressing noise.

2. Methodology

The authors propose DenoiseSplat, a feed-forward 3D Gaussian Splatting framework designed to learn noise suppression within the 3D representation itself.

A. Dataset Construction (Noisy-RE10K)

To train and evaluate the model, the authors constructed a large-scale, scene-consistent benchmark based on RealEstate10K (RE10K):

• Noise Injection: Four noise types are injected into the 2D RGB domain: Gaussian, Poisson, Speckle, and Salt-and-Pepper.
• Scene-Level Consistency: Unlike random noise injection, the same noise type and intensity level are applied to all views within a single scene. This mimics real-world acquisition where a single device captures a sequence with consistent noise characteristics.
• Training Setup: The model is trained end-to-end using noisy multi-view images as input and clean 2D renderings (from the original ground truth) as supervision. No 3D ground truth is used.

B. Network Architecture

The backbone is built upon MVSplat, a feed-forward architecture that encodes multi-view features and aggregates them into a 3D volume. Key innovations include:

1. Dual-Branch Gaussian Head (Geometry-Appearance Decoupling):
   Instead of a single shared head predicting all Gaussian parameters, the network splits into two lightweight branches:

   • Geometric Branch: Predicts position ($x$), rotation, scale, and opacity ($\alpha$). These parameters are treated as more stable cues less sensitive to high-frequency noise.
   • Appearance Branch: Predicts spherical harmonic coefficients and colors. This branch is allowed to absorb residual noise and color fluctuations, preventing noise from corrupting the structural geometry.

2. Cross-Branch Boundary-Guided Appearance Correction (CBC):
   To address errors where geometry uncertainty (e.g., at depth discontinuities) affects appearance rendering:

   • The geometry branch generates a boundary map ($E$) based on disparity gradients and a confidence map ($C$).
   • A gating mechanism uses these maps to control a residual refinement network in the appearance branch.
   • Refinement is applied primarily in low-confidence, high-boundary regions to suppress artifacts (like halos) without altering high-confidence areas.
   • Gradient Isolation: During training, gradients from the appearance branch do not backpropagate into the geometry branch, ensuring the geometry remains stable.

C. Training and Inference

• Loss Function: The model is supervised solely by 2D image losses ($L_1$ and SSIM) comparing the rendered novel views against clean ground-truth images.
• Inference: At test time, the model takes noisy multi-view inputs and camera parameters, performs a single forward pass, and outputs a clean 3D Gaussian scene and high-quality novel views. No test-time optimization or external 2D denoiser is required.

3. Key Contributions

1. Problem Formulation & Framework: First systematic study of feed-forward 3D Gaussian Splatting specifically for noisy multi-view inputs, eliminating the need for per-scene optimization.
2. Dual-Branch Architecture: A novel decoupling of geometry and appearance prediction, which stabilizes 3D structure under noise while allowing flexible color correction.
3. Scene-Consistent Benchmark: Creation of a reproducible noisy-clean dataset on RE10K with controlled, scene-level noise configurations (Gaussian, Poisson, Speckle, Salt-and-Pepper).
4. Cross-Branch Correction: Introduction of the CBC module to mitigate error propagation from geometry to appearance at boundaries.

4. Experimental Results

The method was evaluated on the noisy RE10K benchmark against:

• MVSplat-GT: Upper bound (trained/evaluated on clean data).
• MVSplat-Noisy: Clean-trained model evaluated on noisy inputs.
• Denoise-Then-MVSplat: A strong two-stage baseline (IDF denoiser + MVSplat).

Quantitative Performance:

• DenoiseSplat outperformed both the vanilla MVSplat-Noisy and the two-stage baseline across all noise types and intensities.
• Metrics: It achieved the best trade-off in PSNR (25.05 vs. 24.77 for baseline), SSIM (0.814 vs. 0.788), and LPIPS (0.260 vs. 0.272).
• Generalization: The method maintained stable performance on both seen and novel views, showing a smaller performance gap between them compared to baselines, indicating better cross-view consistency.

Qualitative Analysis:

• MVSplat-Noisy: Suffered from visible residual noise, color shifts, and structural distortions.
• Denoise-Then-MVSplat: Produced cleaner images but often resulted in over-smoothing, loss of high-frequency texture details, and "halo" artifacts near edges due to view-independent 2D denoising.
• DenoiseSplat: Successfully removed noise while preserving sharp boundaries, fine textures, and geometric consistency.

Efficiency:

• DenoiseSplat maintains high efficiency comparable to the original MVSplat (single forward pass), avoiding the latency overhead of the two-stage pipeline.

5. Significance and Future Work

Significance:
This work bridges the gap between theoretical 3D reconstruction and real-world application conditions. By moving noise suppression inside the 3D representation rather than treating it as a pre-processing step, DenoiseSplat achieves superior geometric consistency and texture fidelity. It demonstrates that feed-forward 3DGS can be robust to diverse degradations without sacrificing speed or requiring per-scene tuning.

Limitations & Future Directions:

• Noise Modeling: Currently relies on synthetic noise; real-world camera noise, motion blur, and complex compression artifacts are not yet fully modeled.
• Generalization: Experiments are limited to RE10K; cross-dataset and real-world generalization need further evaluation.
• Future Work: Incorporating real-noise captures, extending to dynamic scenes, and integrating semantic cues for higher-level 3D understanding.