Dehaze-then-Splat: Generative Dehazing with Physics-Informed 3D Gaussian Splatting for Smoke-Free Novel View Synthesis

1. Problem Statement

The paper addresses the challenge of multi-view smoke removal and novel view synthesis (NVS) for 3D reconstruction. Specifically, it targets the NTIRE 2026 3D Restoration and Reconstruction Challenge (Track 2), which involves removing physically captured smoke from multi-view images (based on the RealX3D benchmark) and synthesizing clean novel views.

Core Challenge:
There is a fundamental tension between per-image restoration quality and multi-view consistency:

• Generative Dehazing: State-of-the-art generative models (e.g., Nano Banana Pro) produce high-quality, visually compelling single-frame results. However, they process each view independently, lacking cross-view conditioning. This leads to stochastic variations in color, texture, and hallucinated details across frames.
• 3D Reconstruction Failure: When these inconsistent "pseudo-clean" images are used to train 3D Gaussian Splatting (3DGS), the optimizer attempts to reconcile conflicting photometric signals. This results in blurred renders, texture drift, geometric instability, and "floater" artifacts (Gaussians allocated to fit noise rather than scene structure).

2. Methodology: Dehaze-then-Splat

The authors propose a two-stage pipeline that combines generative dehazing with physics-informed 3DGS training to bridge the gap between 2D restoration and 3D consistency.

Stage 1: Generative Dehazing & Preprocessing

• Model: Uses Nano Banana Pro (via Gemini API) for per-frame smoke removal.
• Prompt Engineering: A structured 6-point prompt is used to ensure smoke removal while preserving scene geometry and photorealism.
• Brightness Normalization: To address inter-frame brightness shifts (up to 0.12 difference) caused by frame-wise processing, the authors apply per-channel mean/std normalization.
  • If Ground Truth (GT) is available, normalization aligns to GT statistics.
  • Otherwise, "self-normalization" aligns frames to the median statistics of the dataset.
• Resolution Alignment: Outputs are resized to match camera intrinsics exactly to ensure pixel-accurate alignment.

Stage 2: Physics-Informed 3DGS Training

Standard 3DGS is augmented with physics-informed auxiliary losses to act as implicit multi-view consistency regularizers, compensating for the inconsistencies introduced in Stage 1.

The Loss Function:
$$L = (1-\lambda_s)L_1 + \lambda_s L_{SSIM} + \lambda_d L_{DCP} + \lambda_p L_{depth} + \lambda_g L_{grad}$$

1. Depth Supervision ($L_{depth}$):
   • Uses Depth Anything V2 to generate pseudo-depth maps from original smoky images (robust to smoke).
   • Applies a scale-invariant weighted Pearson correlation loss between rendered depth and pseudo-depth.
   • Impact: The most significant contributor (+0.79 dB), anchoring geometry despite texture variations.
2. Dark Channel Prior (DCP) Regularization ($L_{DCP}$):
   • Enforces that the dark channel of the rendered image approaches zero, suppressing residual haze artifacts.
3. Dual-Source Gradient Matching ($L_{grad}$):
   • Supervises edge structures using Sobel gradients against a secondary structural reference (MB-TaylorFormer output) to ensure structural consistency.
4. MCMC Densification & Early Stopping:
   • MCMC Strategy: Uses stochastic noise injection/relocation (from gsplat) instead of manual split/clone heuristics, improving PSNR by +0.47 dB.
   • Early Stopping: The authors identify that late densification causes "floater" artifacts. They stop densification early (Step 3000, $\sim$104k–161k Gaussians). This prevents the model from overfitting to per-view inconsistencies, as the model lacks the capacity to memorize artifacts at low Gaussian counts.

Alternative Approach (Rejected)

The authors tested an End-to-End Scattering Decomposition (integrating the Koschmieder atmospheric model directly into 3DGS). This failed (10.28 dB) because, without strong scene priors, the model collapsed to a trivial solution ($\beta \to 0$), failing to separate smoke from content. This validated the necessity of the hybrid "Dehaze-then-Splat" approach.

3. Key Contributions

1. Identification of the Consistency Gap: The paper highlights that high-quality per-image generative dehazing does not guarantee multi-view consistency, and this inconsistency is the primary bottleneck for downstream 3D reconstruction.
2. Physics-Informed Regularization: Introduction of a novel loss function combining depth supervision (Pearson correlation), Dark Channel Prior, and dual-source gradient matching to enforce geometric and photometric consistency in 3DGS.
3. Capacity Control Strategy: Demonstration that early stopping of densification is critical. It prevents the model from overfitting to the stochastic variations of the generative dehazing stage, effectively acting as a regularizer.
4. Hybrid Pipeline: A successful integration of a powerful 2D generative model (Nano Banana Pro) with 3D reconstruction, overcoming the limitations of purely 2D or purely 3D approaches.

4. Results

Evaluated on the Akikaze validation scene (25 training views, 4 test views):

• Performance:
  • PSNR: 20.98 dB
  • SSIM: 0.683
  • Improvement: A +1.50 dB improvement over the unregularized baseline (19.48 dB).
• Ablation Insights:
  • Depth supervision provided the largest gain (+0.79 dB).
  • MCMC densification provided +0.47 dB over default strategies.
  • Early stopping (Step 3k vs. Step 15k) yielded +0.35 dB by preventing overfitting.
  • The optimal checkpoint occurs early (Step 2000–4000) when Gaussian counts are low ($\sim$104k), confirming that the model learns a view-averaged representation rather than memorizing artifacts.
• Comparison: The method significantly outperforms classical dehazing (DCP, ~11.9 dB) and learning-based 2D methods (MB-TaylorFormer, 17.00 dB) when used as a pipeline for NVS.

5. Significance

• Bridging 2D and 3D: The paper provides a robust framework for utilizing powerful generative 2D models for 3D tasks, a domain where they typically fail due to consistency issues.
• Practical Solution for Smoke/Haze: It offers a state-of-the-art solution for real-world scenarios involving smoke and low-light conditions, which are difficult for traditional physics-based or discriminative models.
• Future Direction: The authors suggest that future work should focus on consistency-aware dehazing (e.g., video-based generative models or multi-view attention) to narrow the gap between restoration and reconstruction further.

In summary, Dehaze-then-Splat demonstrates that while generative dehazing introduces inconsistencies, these can be effectively mitigated through physics-informed 3D constraints and careful capacity management, achieving high-fidelity smoke-free novel view synthesis.