Technical Summary: Supercharging Thermal Gaussian Splatting with Depth Estimation

Problem Statement

Efficient and robust 3D scene representation is critical for applications in autonomous driving, robotics, and surveillance. While 3D Gaussian Splatting (GS) has achieved state-of-the-art performance in Novel View Synthesis (NVS) using RGB data, its reliance on visible light makes it vulnerable in degraded environments (e.g., low-light, fog, smoke). Existing multimodal approaches attempt to integrate thermal data to improve robustness, but these methods often rely on fusing RGB and thermal modalities. This fusion introduces significant challenges:

1. Complexity and Speed: Combining multimodal data slows down the training process and complicates the pipeline.
2. Alignment Issues: Thermal and RGB cameras require precise co-registration, which is difficult to achieve.
3. Dependency: Current thermal-based GS frameworks often still depend on RGB images for supervision or joint optimization, failing to establish a truly independent thermal representation system.
4. Data Characteristics: Thermal images inherently suffer from low contrast, sparse textures, and non-uniform brightness, making pure thermal supervision unstable without geometric constraints.

The authors identify a gap in the literature: no existing framework successfully performs 3D reconstruction using only thermal infrared data (a single modality) while maintaining high quality and efficiency.

Methodology: Thermal-to-Depth Gaussian Splatting (TDg)

The paper proposes TDg, a self-contained framework that derives radiance fields using only thermal images and depth estimation, removing reliance on visible light. The core innovation is a unified 3D Gaussian representation optimized via dual rasterization.

Core Architecture

Instead of maintaining separate models for different modalities, TDg utilizes a single set of 3D Gaussians $G = \{\mu_i, \Sigma_i, \alpha_i, c_i\}$ to represent the scene.

• Unified Representation: Each Gaussian is parameterized by position ($\mu$), covariance ($\Sigma$), opacity ($\alpha$), and a thermal radiance feature ($c$).
• Dual Rendering: During optimization, the unified model is rendered into two outputs:
  1. Thermal Images: Using the radiance feature $c_i$.
  2. Depth Maps: Using the Z-coordinate of the Gaussian centers in camera space.

Key Components

1. Thermal-to-Depth Estimation:

   • A thermal-guided depth estimator ($f_\theta$), specifically the Marigold model, predicts dense depth maps ($D$) directly from the input thermal images ($I_{Th}$).
   • This provides a geometric prior to constrain the 3D distribution of the thermal radiance field, addressing the instability of pure thermal supervision in low-texture regions.

2. Loss Functions:

   • Thermal Loss ($L_{thermal}$): Combines pixel fidelity ($L_1$), structural similarity (SSIM), and a smoothness term ($L_{smooth}$). The smoothness term penalizes unnatural abrupt temperature changes, leveraging the physical property that most thermal scenes exhibit gradual temperature gradients.
   • Depth Loss ($L_{depth}$): Since monocular depth estimation suffers from scale and shift ambiguities, both the predicted depth and rendered depth are Min-Max normalized to $[0, 1]$. The loss is a combination of $L_1$ distance and SSIM between the normalized maps.

3. Progressive Joint Optimization:

   • To handle the training instability caused by sparse thermal textures, the authors employ a progressive strategy.
   • Early Training: The depth loss weight is high to enforce structural accuracy based on the estimated depth prior.
   • Late Training: The depth loss weight decays linearly ($w_{decay}$), allowing the model to focus on photometric fidelity and fine-grained thermal details.
   • The total loss is formulated as: $L_{total} = L_{thermal} + w_{decay}(t) \cdot \lambda_{depth} \cdot L_{depth}$.

Initialization Note

While the core rendering and optimization rely solely on thermal data, the authors acknowledge that the initial sparse point cloud is currently derived from RGB images using Structure-from-Motion (SfM/COLMAP). This is necessary because thermal images lack the high-frequency features required for standard SfM feature extraction. However, the optimization and radiance field derivation remain purely thermal.

Key Contributions

1. Thermal-to-Depth Gaussian Splatting (TDg): A novel, purely thermal-based GS approach that integrates depth estimation directly into the architecture.
2. Thermal-Guided Depth Module: A module integrated within GS that uses estimated depth from thermal images to constrain geometric distribution, eliminating the need for RGB supervision during the main optimization phase.
3. Comprehensive Evaluation: A rigorous benchmarking of TDg against the MSMG (Multiple Single-Modal Gaussians) baseline on two state-of-the-art datasets: RGBT-Scenes and ThermalMix.

Experimental Results

The authors evaluated TDg on 14 scenes across the RGBT-Scenes and ThermalMix datasets, comparing performance against the MSMG baseline.

• Rendering Quality:

  • PSNR: TDg achieved an average of 27.42, outperforming the baseline (27.18) by 0.01%.
  • SSIM: TDg achieved 0.8899, a 0.034% improvement over the baseline.
  • LPIPS: TDg achieved 0.175, a 1.12% improvement (lower is better) over the baseline.
  • Qualitative results showed better resolution and fewer artifacts, particularly in low-light conditions ("Dark Scenes").

• Training Efficiency:

  • TDg significantly reduced training time by 12 minutes and 47 seconds per scene on average.
  • This represents a 55% improvement in speed compared to the baseline.
  • The efficiency gain is attributed to the depth-guided supervision, which prevents the "floaters" (excessive, inefficient densification in empty space) common in the baseline by providing explicit spatial gradients.

• Failure Cases:

  • Random Initialization: Without the COLMAP-based geometric prior, the model fails to generalize to novel views, resulting in severe overfitting (PSNR drops to ~9.6).
  • Occluded Areas: The method struggles with large-scale structures (e.g., buildings) where 360-degree thermal capture is infeasible, leading to poor reconstruction in unobserved rear facades.

Significance and Claims

The paper claims that TDg successfully bridges the gap between thermal imaging and efficient 3D reconstruction by removing the dependency on RGB data for the core radiance field derivation.

• Robustness: The method enables 3D scene representation that is independent of visible light, making it suitable for surveillance, search and rescue, and industrial inspections where lighting conditions are poor or variable.
• Efficiency: By leveraging a single modality and depth-guided optimization, the method accelerates convergence and simplifies the processing pipeline compared to multimodal fusion approaches.
• Application Potential: The resulting thermal radiance fields can serve as digital twins or enrich existing ones, allowing for the identification of heat sources (people, machinery) and the analysis of heat transfer behaviors from novel viewpoints without the need for multi-modal hardware synchronization.

The authors conclude that while thermal images alone provide strong structural signals, the integration of estimated depth is crucial for guiding the optimization process, effectively balancing geometric accuracy with photometric fidelity in a single-modality framework.