Technical Summary: Towards Integrating Multi-Spectral Imaging with Gaussian Splatting

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has established itself as a state-of-the-art framework for fast, high-fidelity 3D reconstruction from multi-view RGB images, naively extending it to multi-spectral data (e.g., Red, Green, Red-Edge, Near-Infrared) presents significant challenges. The core issue is that individual spectral bands often lack the dense, feature-rich information present in RGB images. Consequently, optimizing a separate 3DGS model for each spectral band in isolation leads to degraded or unstable geometric reconstructions, even though the underlying physical geometry of the scene remains identical across all modalities. The paper addresses the need for a robust strategy to integrate multi-spectral data into the 3DGS framework to achieve balanced reconstruction quality across all bands while leveraging the potential for "spectral cross-talk" to enhance overall fidelity.

2. Methodology

The authors propose and evaluate three distinct strategies for integrating multi-spectral data (RGB, Red, Green, Red-Edge, and NIR) into the 3DGS pipeline, assuming a globally aligned sparse point cloud and camera poses are available.

2.1. Core Strategies

1. SEPARATE: A baseline approach where an independent 3DGS model is optimized for each spectral band without any shared structure or parameters. This serves as a lower-bound reference, prone to geometric artifacts in non-RGB bands.
2. SPLIT: A two-stage sequential process. First, a high-quality 3DGS model is optimized using only RGB data. The resulting geometry (means, covariances, opacities) is then duplicated and split into separate models for each spectral band. Each band-specific model is subsequently optimized independently, allowing the RGB-derived geometry to guide the reconstruction of other bands but preventing reciprocal influence.
3. JOINT: A unified pipeline where all spectral channels are optimized simultaneously. In this approach, each Gaussian primitive holds spherical harmonics coefficients for all spectral channels. During training, a channel is randomly selected, and the scene is rendered solely in that band, allowing for the joint optimization of shared structural parameters (geometry) and distinct spectral color parameters.

2.2. Optimization Enhancements for the JOINT Strategy

To improve the JOINT strategy, the authors introduce and ablate several training configurations:

• Spectral Delay (SpecDelay): To prevent spectral gradient heuristics from introducing outliers during early training, the optimization initially uses only RGB data for the first 30,000 iterations to establish robust geometry. Joint optimization of all spectral channels commences thereafter.
• Extended ADC (ExtADC): The Adaptive Density Control (ADC) densification interval is extended from the standard 25,000 to 60,000 iterations to better accommodate the larger set of training samples introduced by multi-spectral data.
• Multi-Spectral Aware Densification (MSAD): A heuristic where per-primitive gradients are tracked independently for each spectral band. Splitting or cloning occurs if any accumulated gradient across bands exceeds the threshold, ensuring densification responds to high-detail requirements in specific spectral bands.
• Spectral-Invariant Geometry (SIG): A variation where structural parameters (position, covariance, opacity) are locked to updates from RGB data only, while non-RGB bands solely refine spectral coefficients. This prevents low-signal spectral channels from distorting the 3D structure.

The authors also implement a "warm-up" step for SpecDelay, pausing densification when spectral bands are introduced to stabilize the integration of new color values without disrupting the learned RGB geometry.

3. Key Contributions

• Systematic Evaluation of Integration Paradigms: The paper formulates and compares three optimization paradigms (SEPARATE, SPLIT, JOINT) for incorporating additional spectral bands into 3DGS, identifying the limitations of independent per-band optimization.
• Optimized JOINT Framework: The authors propose a set of specific optimizations for the JOINT strategy (SpecDelay, ExtADC, MSAD, SIG) that significantly improve reconstruction quality. The combination of these modules, termed JOINT-OPTIMIZED, is presented as the most effective approach.
• Analysis of Spectral Cross-Talk: The study quantifies how the interplay between spectral bands improves reconstruction. It demonstrates that joint optimization allows spectral bands to mutually enhance geometric detail, leading to improved reconstruction quality not only in non-RGB bands but also in the RGB channel itself.

4. Experimental Results

The methods were evaluated on a multi-spectral dataset comprising seven outdoor scenes captured with a DJI Mavic 3M drone (RGB, Red, Green, Red-Edge, NIR).

• Quantitative Performance:
  • SEPARATE yielded the poorest results, particularly for non-RGB bands, confirming that isolated optimization fails to capture consistent geometry.
  • SPLIT offered balanced improvements over SEPARATE for non-RGB bands by leveraging RGB geometry but did not improve RGB reconstruction.
  • JOINT-OPTIMIZED (combining JOINT + SpecDelay + MSAD) achieved the highest performance across all metrics (PSNR, SSIM, LPIPS) for all bands. Notably, it improved RGB reconstruction by approximately 0.2–0.3 dB compared to RGB-only baselines, demonstrating the efficacy of spectral cross-talk.
• Primitive Count: The JOINT-OPTIMIZED approach required roughly two million primitives to represent a scene with high quality, whereas the SEPARATE approach required approximately one million primitives per band (totaling five million). The MSAD strategy was observed to increase the primitive count selectively in bands requiring higher detail, effectively utilizing the shared geometry.
• Ablation Studies: The ablation results confirmed that SpecDelay provided the most drastic quality increase by stabilizing early training. MSAD further improved metrics across all bands, while SIG generally yielded unsatisfactory results for non-RGB bands, though it provided the best overall RGB results in specific configurations, suggesting a trade-off between geometric flexibility and spectral fidelity.

5. Significance and Claims

The paper claims that integrating multi-spectral data directly into the spherical harmonics color components of a single, jointly optimized 3DGS model is a viable and effective strategy. The primary significance lies in demonstrating that spectral cross-talk—the mutual influence of different spectral bands on a shared geometric representation—can enhance reconstruction quality across all modalities, including the visible RGB spectrum.

The authors suggest that this approach offers practical insights for robust multi-modal 3DGS reconstruction, particularly in applications requiring detailed spectral analysis, such as plant phenotyping in agricultural environments and fruit counting. They note that while their method assumes shared geometry (suitable for visible and near-infrared spectra), it may face limitations with thermal infrared imagery where geometry can differ significantly due to emission properties. The work provides a foundation for future real-time multi-spectral applications, such as immersive visualization systems, by establishing a training pipeline that balances geometric stability with spectral fidelity.