💻 computer science

Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization

This paper proposes Multi-Layer Gaussian Splatting, a novel technique that compresses volumetric CT scans into an efficient, static intermediate representation with layered anatomical structures and selective activation, enabling real-time, high-quality immersive visualization on compute-constrained devices while retaining key explorative qualities of traditional path tracing.

Original authors: Constantin Kleinbeck, Hannah Schieber, Klaus Engel, Ralf Gutjahr, Daniel Roth

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Constantin Kleinbeck, Hannah Schieber, Klaus Engel, Ralf Gutjahr, Daniel Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to show a complex 3D model of a human body to a friend on a tiny, battery-powered VR headset. Usually, to make it look real with perfect lighting and shadows, you need a supercomputer to calculate how every single photon of light bounces around. But your headset isn't a supercomputer; it's more like a smartwatch trying to run a movie studio. If you try to do the heavy math in real-time, the image freezes, or the battery dies instantly.

The authors of this paper suggest a clever workaround: don't do the hard math while your friend is looking at it. Instead, do the heavy lifting beforehand, turn the result into a "snapshot" that is easy to play, and then let the headset just show that snapshot.

The "Frozen Light" Trick

Think of a standard 3D medical scan like a block of Jell-O. To see inside it, a traditional method (called path tracing) sends millions of tiny light beams through the Jell-O to see how they scatter. This creates a beautiful, lifelike image, but it takes forever to calculate.

The authors propose a new way using something called Gaussian Splatting. Imagine instead of a solid block of Jell-O, you have a cloud of millions of tiny, glowing, fuzzy balls (the "Gaussians"). Each ball has a specific color and position. When you look at them from a distance, they blend together to look like a smooth, realistic 3D object. Because these balls are pre-calculated, your headset doesn't need to do any heavy math; it just has to draw the balls. This is like swapping a live, complex orchestra performance for a high-quality recording that plays instantly on any device.

The Problem with "Static" Clouds

Here's the catch: A standard cloud of these fuzzy balls is static. It's like a statue. If you want to see just the bones, you need one statue. If you want to see the bones and the muscles, you need a completely different, separate statue. If you try to cut a hole in the statue to peek inside, the edges look messy and wrong because the "inside" was never calculated.

The paper argues that simply stacking separate models on top of each other is messy and inefficient. It suggests that we need a smarter way to build these clouds so they can be peeled back like layers of an onion without breaking the picture.

The Solution: A Layered Onion

The authors' main finding is a Layered Gaussian Splatting model. They built a system where you can train multiple "layers" of these fuzzy balls on top of each other, all in one single file.

Imagine building a 3D model of a leg:

Layer 1 (The Bones): You train the first set of fuzzy balls to perfectly represent the skeleton.
Layer 2 (The Muscles): You add a second set of balls that only fill in the gaps where the muscles are. These new balls don't touch the bones; they just sit on top, adding the extra detail.
Layer 3 (Soft Tissue): You add a third layer for the skin and fat.

Because each layer only adds new information, the file stays small. The magic is that you can now "cut" through the model in real-time. If you slice off the skin layer, the muscle layer underneath appears instantly. If you slice off the muscle, the bones appear. It's like having a single digital onion where you can peel off layers on the fly without the computer needing to rebuild the whole thing.

Cleaning Up the Mess

When you stack these layers, a new problem pops up. Sometimes, the computer tries to put fuzzy balls in places that are already hidden by a lower layer (like trying to paint the inside of a closed box). These "invisible" balls waste space and slow things down.

The paper introduces a technique called Inactive Pruning. Think of this as a smart janitor. After the model is built, the janitor walks through and sweeps away any fuzzy balls that are completely hidden behind other layers or aren't doing any work. This keeps the file size small and the rendering fast. They also fixed a glitch where transparent parts (like veins) looked weird by teaching the computer to pay attention to "transparency" (the alpha channel) during the training, ensuring the background doesn't bleed through the see-through parts.

What the Numbers Say

The authors tested this on real CT scans of human bodies (specifically a "Leg" scan and a "Fullbody" scan).

Speed: On a powerful computer, a traditional method took about 7,116 milliseconds (over 7 seconds) to render a single image. Their method took about 1.7 milliseconds on a standard PC and 10.2 milliseconds on a standalone VR headset (Meta Quest 3). That's thousands of times faster.
Size: A standard medical scan file might be hundreds of megabytes. Their compressed, layered model for a full body was only 7.6 MB (after clustering compression).
Quality: They measured how close the images were to the original using a score called PSNR. Their high-quality version scored 35.46, which is very close to the original, while the standard "denoised" path tracing on a mobile device scored much lower.

What They Don't Claim

The paper is careful not to overpromise.

It's not a magic wand for any movement. The model is static. You can't make the muscles move or the heart beat in real-time. It's a snapshot, not a movie.
It's not perfect up close. If you zoom in too far on a low-contrast area (like soft tissue), you might see the individual fuzzy balls or "smudges." The authors admit that fine details can get blurry.
It's not a replacement for diagnosis. They suggest this is great for education, planning, and visualization, but they don't claim it replaces the need for a doctor to look at the original raw scan for critical medical decisions.

The Verdict

The paper suggests that by pre-calculating the "light" and organizing it into neat, peelable layers, we can finally bring high-quality, realistic 3D anatomy to lightweight VR headsets. It's not a perfect, fully interactive world where you can play with the organs, but it bridges the gap between "ugly, fast 3D" and "beautiful, slow 3D." It allows students and doctors to slice through a virtual body in real-time on a device they can hold in their hands, making complex anatomy much easier to explore and understand.

Technical Summary: Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization

Problem Statement

Medical image visualization, particularly for Computed Tomography (CT) scans, traditionally relies on 2D slices or Direct Volume Rendering (DVR). While volumetric path tracing offers lifelike, high-fidelity 3D visualizations with realistic lighting and shadows beneficial for education and surgical planning, it is computationally prohibitive for real-time use on mobile and standalone Virtual Reality (VR) headsets. These devices lack the storage capacity and rendering power required for high-resolution path tracing or even high-quality DVR.

Existing alternatives like Gaussian Splatting (GS) offer efficient real-time rendering but suffer from significant limitations in this context:

Static Nature: Standard GS models are static; transfer functions (color/transparency) are fixed at training time.
Lack of Interactivity: Techniques like cutting, slicing, or isolating specific anatomical structures (e.g., viewing bone vs. muscle) are difficult without visual artifacts or requiring multiple separate, redundant models.
Transparency Issues: Standard GS training flattens alpha channels, leading to artifacts in semi-transparent regions, especially when trained on synthetic data with transparent backgrounds.
Redundancy: Training separate models for different anatomical views results in large storage footprints due to overlapping content.

Methodology

The authors propose a Layered Gaussian Splatting (GS) approach that creates a single, intermediate representation capable of visualizing multiple anatomical structures sequentially. The workflow consists of three stages:

1. Image Generation

High-quality training images are pre-rendered from CT volumetric data using a Monte Carlo volume path tracer.

Initialization: Instead of using Structure-from-Motion (COLMAP) or random point clouds, the authors generate an initial point cloud by sampling from outside the volume inward, stopping when a transfer function opacity threshold ( $\alpha = 0.2$ ) is reached. This ensures points are located near the surface where Gaussians will form.
Alpha Channel Handling: To address transparency artifacts, the training process employs two strategies:
- Randomized Backgrounds: The background color is randomized for each frame during training rather than flattened, preventing background color bleeding into semi-transparent areas.
- Alpha Loss: The alpha channel is explicitly included in the loss calculation to maintain detail in translucent regions.

2. Layered Training

The core innovation is a consecutive training process where multiple GS layers are stacked within a single .ply file.

Sequential Optimization: Layers are trained one after another (e.g., Bone $\rightarrow$ Muscle $\rightarrow$ Soft Tissue). Once a layer is optimized, its Gaussians are "frozen" and treated as a static background for the subsequent layer.
Differential Representation: New layers only optimize Gaussians for new information (e.g., muscle tissue) or areas where the previous layer's information is insufficient. Existing lower layers remain untouched, preserving their visual integrity.
Inactive Pruning: A critical extension to standard GS adaptive control. During the training of upper layers, some Gaussians in lower layers may become completely occluded (e.g., inside a bone) or redundant. The authors introduce a pruning mechanism that tracks Gaussian activity (position, color, and scale changes). Gaussians failing to meet a minimal activity threshold are removed. This threshold is updated via exponential decay to avoid removing well-optimized Gaussians early in the process.

3. Real-Time Rendering

The trained layered model is rendered in the Unity game engine (2022.3) using HLSL compute shaders.

Selective Loading: Only the actively rendered layers are transferred to the GPU; unused layers remain in RAM.
Compression: To reduce storage and memory footprint, the authors apply quantization and K-means clustering across all layers simultaneously. This clusters spherical harmonics coefficients (which constitute ~75% of file size) into 16k variants, leveraging the limited color palette of medical volumes.
Sorting Optimization: To handle VR stereo rendering, the authors explore sharing the sorting of splats between both eyes to halve the sorting time per frame.

Key Contributions

Layered GS Representation: A method to train and stack multiple GS renderings of the same volume into a single asset, enabling the selective display of anatomical structures (e.g., bone only, or bone + muscle) without retraining.
Inactive Pruning: A technique to remove Gaussians that are occluded or redundant during the layering process, reducing clutter and improving efficiency.
Alpha Channel Training: An extension of the GS training pipeline using randomized backgrounds and alpha loss to faithfully reconstruct semi-transparent anatomical structures.
Unity Implementation: A real-time rendering pipeline for layered GS models in Unity, supporting cutting and layer toggling on mobile and desktop VR devices.
Open Resources: The release of training and rendering code along with datasets.

Results and Evaluation

The approach was evaluated using real-world CT scans (Lower Body, Full Body, Legs) and synthetic scenes.

Performance: The method achieves interactive frame rates on both desktop VR (Varjo Aero) and standalone mobile VR (Meta Quest 3). On a PC with an RTX 4090, rendering takes approximately 1.3–1.7 ms for high-compression models, significantly outperforming path tracing (7116 ms) and denoised path tracing (66 ms).
Quality: The models achieve high visual fidelity (PSNR ~35–37, SSIM ~0.96) compared to ground truth, outperforming standard path tracing at low sample counts. While fine details may blur in low-frequency regions, high-contrast features are accurately reconstructed.
Storage Efficiency: The layered approach reduces storage requirements significantly compared to storing multiple standalone GS models. For example, a 3-layer leg scene required 34.4 MB (layered) versus 46.6 MB (standalone combined) before compression. After clustering compression, sizes dropped to 5.8 MB (Leg) and 7.6 MB (Fullbody), representing up to a 99% reduction compared to original DICOM volumes.
Comparison to Continual Learning: Compared to CLNeRF, the proposed method better represents fine anatomical structures (like bones) with fewer training images and avoids the memory overhead of replay buffers, making it more suitable for real-time VR.

Significance and Claims

The paper claims that this approach bridges the gap between the high visual fidelity of path tracing and the performance constraints of mobile VR devices. By creating a static intermediate representation that retains "explorative qualities," the method enables:

Selective Interactivity: Users can toggle layers or cut through them to visualize specific anatomical relationships (e.g., bone vs. soft tissue) in real-time, a capability previously restricted to computationally expensive path tracing or static 2D slices.
Accessibility: It makes high-quality 3D anatomical visualization feasible on affordable, standalone headsets, removing the need for tethered, high-performance workstations.
Educational Utility: The ability to interactively explore complex anatomies supports medical education and surgical planning where real-time path tracing is currently impractical.

The authors acknowledge limitations, noting that the approach is still semi-static (transfer functions are fixed) and that extreme close-ups may reveal splatting artifacts. However, they posit that the achieved level of detail is sufficient for many practical medical visualization scenarios.