⚡ electrical engineering

InnerGS: Internal Scenes Reconstruction and Segmentation via Factorized 3D Gaussian Splatting

InnerGS introduces a factorized 3D Gaussian Splatting framework that reconstructs detailed internal scene structures from sparse sliced data without requiring camera poses and extends this capability to text-guided segmentation by integrating language features.

Original authors: Shuxin Liang, Yihan Xiao, Wenlu Tang

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuxin Liang, Yihan Xiao, Wenlu Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, three-dimensional model of a city, but you only have a stack of flat, two-dimensional blueprints. Most computer programs today are like master architects who can only look at the outside of a building. They can tell you what the front door looks like or how the roof is shaped, but if you ask them to show you the plumbing inside the walls or the wiring behind the drywall, they hit a wall. They simply don't know how to fill in the empty space between the blueprints. This is a huge problem for fields like medicine, where doctors need to see inside a body without cutting it open, or for robots that need to understand how to grab a complex object from the inside out.

To solve this, scientists have been using a clever trick called "3D Gaussian Splatting." Think of a Gaussian not as a scary math formula, but as a fuzzy, glowing cloud of light. Imagine you are painting a 3D scene by throwing thousands of these soft, glowing clouds into the air. Each cloud has a center, a size, and a color. When you look at the scene from the outside, these clouds overlap and blend together to create a smooth, realistic picture. It's like creating a hologram out of mist. This method is incredibly fast and makes beautiful pictures, but until now, it has mostly been used to paint the surface of things. It's great for the skin of an apple, but terrible for showing the seeds and the core inside.

This is where a new paper called "InnerGS" comes in. The researchers, Shuxin Liang, Yihan Xiao, and Wenlu Tang, asked a simple but bold question: What if we could use these glowing clouds to paint the inside of things, too? They realized that the old way of painting these clouds relied on taking a picture from a specific camera angle. But when you have a stack of medical slices (like CT or MRI scans), you don't have a camera; you just have a stack of flat pages. The old method failed because it tried to force a 3D cloud into a 2D picture without knowing where the camera was standing.

The team's solution is like changing the rules of the game. Instead of throwing a cloud and then squishing it flat to fit a camera, they decided to slice the clouds themselves. They figured out a way to take a single 3D cloud and mathematically "cut" it at different depths, creating a perfect 2D shape for every single slice of the stack. It's as if they took a fluffy marshmallow and showed you exactly what the cross-section looks like at every millimeter of height, without ever needing to take a photo of it. They call this "Conditional Splatting." It's a smart way of saying, "If you are looking at slice number 5, here is exactly what that specific cloud looks like right here."

By doing this, they can take a sparse set of slices—maybe just a few pages from a thick book—and fill in all the missing pages in between with smooth, detailed 3D data. They tested this on real medical data, like brains and hearts. In their simulations, they found that their new method was much faster and more accurate than the old "squishy" way. For example, when reconstructing a brain, their method achieved a score (called PSNR) of about 32.5, which is a very high quality, and it could do it in about 26 minutes. Even cooler, they showed that this system could handle moving parts, like a wrist bending or a heart beating, by letting the clouds wiggle and change shape over time.

But the magic didn't stop at just seeing the inside. The team also showed that you can talk to this 3D model. By teaching the clouds to understand language, they made a system where you can type "show me the liver" or "find the tumor," and the computer will highlight exactly that part of the 3D volume. In tests, this language-guided segmentation was incredibly accurate, correctly identifying parts of the body about 88% of the time.

The paper doesn't claim to have solved every problem in the world. They admit that their method works best with specific types of data and that there are still challenges with things like brightness changes in the images. However, they have proven that it is possible to reconstruct the hidden, volumetric interior of objects using these glowing clouds, without needing complex camera setups. They even made their code available for anyone to try, suggesting that this could be a big step forward for medical imaging, robotics, and virtual reality, helping us finally see what's hiding inside the things we can't touch.

Technical Summary: InnerGS – Internal Scene Reconstruction and Segmentation via Factorized 3D Gaussian Splatting

Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized efficient scene rendering for external surfaces, existing methods face significant limitations when applied to internal scene reconstruction. Traditional 3DGS relies on projecting 3D Gaussians onto a 2D image plane using camera view and projection matrices. This approach is fundamentally unsuitable for internal structures (e.g., medical imaging) for two primary reasons:

Surface Bias: Standard 3DGS learns Gaussians concentrated on object surfaces, leaving the interior volume largely empty or undefined.
Pose Dependency: The projection-based pipeline requires camera poses, which are absent in sliced data modalities like MRI, fMRI, or CT slices.

Consequently, applying standard 3DGS to sparse sliced data fails to capture continuous volumetric information, leading to incomplete internal representations. Furthermore, existing internal reconstruction methods often rely heavily on specific modalities (X-ray/CT), struggle with generalization to others (MRI, fMRI), or depend on complex regularization and deep MLPs that hinder real-time inference.

Methodology

The authors propose InnerGS, a framework that adapts 3DGS for direct volumetric inference from sparse, pose-free sliced data. The core innovation lies in modeling a continuous volumetric density through Factorized 3D Gaussian Splatting.

1. Factorized Gaussian Representation

Instead of projecting a 3D Gaussian onto a 2D plane, InnerGS models the contribution of a Gaussian to a pixel $(u, v)$ at a specific depth slice $t$ directly. The density is factorized into:
$p(u, v, t) = p(u, v | t) \cdot p(t)$

$p(t)$ : A 1D Gaussian modeling the uncertainty/distribution along the depth axis.
$p(u, v | t)$ : A conditional 2D Gaussian describing the lateral spatial distribution on the image plane given the slice location $t$ .

This formulation allows the mean of the 2D splat to shift dynamically as the slice depth $t$ changes, and the extent of the splat to scale based on the distance between the 3D Gaussian center and the slice. This enables the synthesis of novel slices at arbitrary depths without camera poses.

2. Rendering Pipeline: Conditional Splatting

To render these internal structures efficiently, the paper introduces Conditional Splatting as a superior alternative to the standard "3D Ellipsoid Projection."

3D Ellipsoid Projection (Baseline): Approximates the 3D Gaussian range as a sphere and projects an identical cubic bounding box across all slices. This leads to redundant computation and overly large processing regions.
Conditional Splatting (Proposed): Dynamically adapts the bounding box for each slice based on the conditional 2D Gaussian. This results in tighter sampling, significantly reducing the number of candidate Gaussians per pixel and false positives.

The rendering process sorts candidate Gaussians by distance to the image plane and accumulates color and opacity using a differential rasterization process similar to standard 3DGS, but operating on volumetric slices rather than camera views.

3. Optimization and Extensions

Training: The model is optimized by minimizing rendering loss between predicted slices and ground truth slices. The pipeline includes standard 3DGS refinement steps (pruning, densification, cloning) adapted for the slice-based context.
Dynamic Reconstruction: For 4D data (e.g., cardiac motion, fMRI), the framework integrates a 4D Gaussian Splatting approach. A static set of 3D Gaussians represents the canonical anatomy, while a learned deformation field network predicts per-frame changes in position, covariance, color, and opacity.
Segmentation: The framework is extended for downstream tasks by integrating Vision-Language Models (VLMs). Language features are distilled into compact latent attributes attached to each Gaussian. A differentiable splatting equation aggregates these features, allowing for open-vocabulary text-guided segmentation of medical scenes.

Key Contributions

Inner Gaussian Splatting Framework: A novel approach leveraging 3DGS for direct volumetric inference, enabling high-fidelity reconstruction of complex internal structures from sparse, pose-free sliced data.
Conditional Splatting Pipeline: An efficient, slice-based rendering technique that dynamically adapts Gaussian sampling for each 2D slice, improving computational efficiency and reconstruction accuracy compared to heuristic projection methods.
Plug-and-Play Implementation: A provided CUDA implementation that demonstrates effectiveness across multiple medical datasets, including static brain and cardiac MRIs, as well as 4D dynamic sequences (wrist motion and brain fMRI).
Multimodal Segmentation: A demonstration of the framework's potential for text-guided segmentation by integrating language features, enabling natural language queries for medical scene understanding.

Experimental Results

The authors evaluated InnerGS on several datasets using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM).

Static MRI Reconstruction:
- Brain MRI: Achieved an average PSNR of 32.47 dB and SSIM of 0.9676 on the BrainWeb dataset. The model successfully reconstructed fine anatomical structures (cortical folds, deep boundaries) and smooth transitions in homogeneous areas.
- Cardiac MRI: Achieved an average PSNR of 31.50 dB and SSIM of 0.9439 on the HVSMR-2.0 dataset, accurately capturing heart chambers and vessels.
Dynamic Reconstruction:
- Wrist Motion: On a dynamic MRI dataset, InnerGS consistently outperformed a static baseline (using the previous frame as a prediction), particularly in timestamps with large anatomical motion. Average SSIM improved from 0.9429 (baseline) to 0.9649 (InnerGS).
- Brain fMRI: The model captured temporal fluctuations in signal intensity (BOLD contrast) by freezing geometry and optimizing only color/opacity, outperforming static baselines in PSNR.
Segmentation:
- On the CT-ORG subset of SAT-DS, the method achieved a mean Dice Similarity Coefficient (mDSC) of 88.17%, mean Intersection-over-Union (mIoU) of 83.72%, and Pixel Accuracy (PA) of 98.97%.

Significance and Claims

The paper claims that InnerGS addresses a critical gap in 3D reconstruction by shifting focus from surface completion to volumetric inference. Its significance lies in:

Modality Agnosticism: Unlike existing methods tailored for X-ray or CT, InnerGS is inherently compatible with various data modalities (MRI, fMRI, etc.) without requiring specific physics-based attenuation models.
Pose-Free Operation: It eliminates the need for camera poses, making it directly applicable to sliced medical data where pose information is unavailable.
Efficiency: By avoiding heavy multi-layer perceptrons (MLPs) and utilizing the factorized Gaussian structure, the method maintains the speed advantages of 3DGS, offering real-time potential and faster training compared to NeRF-based internal reconstruction.
Downstream Utility: The framework serves as a foundation for clinical tasks, demonstrated through high-quality static/dynamic reconstruction and the ability to perform open-vocabulary segmentation, potentially aiding in diagnostics and surgical planning.