MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging

1. Problem Statement

Medical imaging modalities such as Ultrasound, MRI, and CT provide crucial non-invasive anatomical data. However, reconstructing high-fidelity 3D surfaces from sequential 2D slices presents significant challenges:

• Sparse Data & Noise: Medical scans often suffer from noise and sparse sampling (large gaps between slices), making accurate interpolation difficult.
• Limitations of Traditional Methods: Classical approaches (e.g., Marching Cubes on voxel grids) often produce connectivity artifacts and are resolution-dependent.
• Limitations of Implicit Neural Representations (INRs): While INRs offer high fidelity, they are computationally expensive, difficult to interpret, and hard to edit manually, which hinders their utility in deformable registration and computer-assisted surgery.
• Task Decoupling: Existing methods often treat frame interpolation (image synthesis) and segmentation (surface extraction) as separate tasks, failing to leverage the geometric continuity between them.

2. Methodology: MedGS Framework

The authors propose MedGS, a novel framework based on Gaussian Splatting (GS) designed for high-fidelity 3D anatomical reconstruction. It builds upon VeGaS (Video Gaussian Splatting) and utilizes Folded-Gaussian distributions.

Core Architecture

MedGS employs a multi-task learning strategy where a unified geometric representation is shared between two distinct tasks:

1. Frame Interpolation: Reconstructing grayscale medical image intensity.
2. Segmentation: Reconstructing binary occupancy masks for surface extraction.

Unified Geometric Representation:
The scene is modeled as a collection of $N$ 3D Folded-Gaussian primitives. Each primitive $i$ is defined by:

• Shared Geometry: Mean ($m_i$), Covariance ($\Sigma_i$), and temporal deformation functions ($a_i, f_i$). These parameters are identical for both tasks, enforcing structural consistency.
• Dual Appearance Attributes:
  • $c^{img}_i$: Color coefficients for reconstructing the grayscale medical scan.
  • $c^{seg}_i$: Color coefficients for reconstructing the binary segmentation mask.

Key Mechanisms:

• Folded-Gaussians: Unlike static Gaussians, these are conditioned on frame timestamps ($t$), allowing the model to capture temporal dynamics and nonlinear spatial changes (e.g., organ deformation) efficiently.
• In-Between Frame Regularization (IBFR): To handle sparse data and noise, the model generates synthetic "pseudo-ground-truth" frames by linearly interpolating between consecutive input frames ($I_t$ and $I_{t+1}$). A regularization loss ($L^*_{interp}$) forces the Gaussians to smoothly approximate transitions, preventing overfitting to noise or collapsing into discrete layers.
• Mesh Extraction: The segmentation branch renders a continuous 3D scalar field representing occupancy probability. High-fidelity 3D meshes are extracted by querying this field at dense intervals and applying the Marching Cubes algorithm.

Objective Function:
The total loss combines image reconstruction, segmentation, and temporal regularization:
$$L_{total} = L_{img} + \lambda_{seg}L_{seg} + \lambda_{\sigma}L_{\sigma}$$

• $L_{img}$: Combines L1 reconstruction, SSIM, and the IBFR loss.
• $L_{seg}$: Minimizes the difference between rendered masks and ground truth.
• $L_{\sigma}$: Regularizes the temporal spread parameter ($\sigma_t$) to prevent Gaussians from being too sparse or diffuse.

Model Variants:

• MedGS (Full): Joint optimization of image and mask tasks.
• MedGSi: Interpolation-only (no mask branch), useful when segmentation masks are unavailable.
• MedGSs: Mask-only (no image branch), used to demonstrate the benefit of joint learning.

3. Key Contributions

1. First GS-based Framework for Sequential Medical Imaging: MedGS adapts Gaussian Splatting specifically for 3D organ mesh reconstruction from sequential 2D medical data.
2. Multi-Task Coupling: Introduces a shared-geometry, dual-appearance architecture that uses dense signals from image synthesis to regularize the geometry of segmentation, enabling high-quality surface extraction even from sparse inputs.
3. Explicit & Editable Representation: Unlike implicit models, MedGS provides an explicit representation that is efficient to train, interpretable, and allows for direct geometric manipulation.
4. Robustness to Sparsity: Demonstrates superior performance in scenarios with missing frames or noisy data through the IBFR mechanism.

4. Experimental Results

The framework was evaluated on Prostate Ultrasound, MRI (Brain, Heart, Lung, Kidney), and CTA datasets.

• Interpolation Quality (MRI):
  • MedGS significantly outperformed linear interpolation and optical flow baselines in PSNR and SSIM metrics, even when trained on every 2nd, 3rd, or 5th frame.
  • Qualitative results showed sharper anatomical boundaries compared to the blurring seen in baselines.
• Mesh Reconstruction (Prostate Ultrasound):
  • MedGS achieved the lowest Chamfer Distance (CD) and 95th Percentile Hausdorff Distance (HD95) compared to ISO-surface extraction, Poisson reconstruction, and the neural implicit method FUNSR.
  • Ablation Study: Removing the IBFR loss ($L^*_{interp}$) or temporal regularization ($L_\sigma$) resulted in degraded performance, confirming their necessity for handling sparse/noisy data.
• Clinical Application:
  • Renal Tumors: Provided precise 3D localization and orientation relative to kidney parenchyma, aiding surgical planning (partial vs. radical nephrectomy).
  • Aortic Aneurysms: Enabled detailed 3D characterization of aneurysm morphology and longitudinal tracking of growth, supporting intervention timing.

5. Significance and Impact

• Clinical Utility: MedGS offers a powerful tool for longitudinal disease monitoring (e.g., tracking tumor growth or aneurysm expansion) and surgical planning by providing clear, high-fidelity 3D visualizations from sparse or noisy 2D data.
• Efficiency: The explicit nature of Gaussian Splatting allows for faster training (~20 mins per object) and instant rendering/mesh extraction compared to heavy implicit neural networks.
• Decision Support: While not a replacement for direct tissue measurement, MedGS serves as a robust decision-support tool, enhancing the ability of clinicians to visualize complex anatomical structures and plan interventions with greater precision.

Availability: The source code is publicly available at https://github.com/gmum/MedGS.