💻 computer science

RT-Splatting: Joint Reflection-Transmission Modeling with Gaussian Splatting

RT-Splatting is a novel framework that enhances 3D Gaussian Splatting for semi-transparent scenes by disentangling geometric occupancy from optical opacity to jointly model high-fidelity reflections and clear transmission within a unified representation, while employing Specular-Aware Gradient Gating to mitigate optimization ambiguities and enable real-time rendering.

Original authors: Ji Shi, Xianghua Ying, Bowei Xing, Ruohao Guo, Wenzhen Yue

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ji Shi, Xianghua Ying, Bowei Xing, Ruohao Guo, Wenzhen Yue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect 3D photo of a car with a glass window. When you look at the window, you see two things happening at once:

Reflection: You see the shiny image of the sky or trees bouncing off the glass.
Transmission: You see the car seats and dashboard through the glass.

Current 3D photo technologies (called "3D Gaussian Splatting") are great at making things look real, but they get confused by glass. They tend to do one of two bad things:

They make the reflection blurry, like a smudged mirror.
They make the glass look like a solid, opaque wall, blocking your view of the car seats inside.

The paper introduces RT-Splatting (Reflection-Transmission Splatting), a new way to fix this. Here is how it works, using simple analogies:

1. The "Double-Identity" Glass

Think of a standard 3D model of a window as a single piece of plastic. It has to decide: "Am I solid (blocking light) or am I see-through?" It can't really be both at the same time without getting messy.

RT-Splatting gives every tiny piece of the window (called a "Gaussian") two separate identities:

Identity A (The Surface): This decides how much the piece acts like a solid mirror to catch reflections. It says, "I am a solid surface here!"
Identity B (The Filter): This decides how much light gets through the piece. It says, "I am clear here, let the light pass."

The Analogy: Imagine a stained-glass window.

The Surface Identity is the glass pane itself, which is solid enough to reflect the sun.
The Filter Identity is the colored glass, which lets light through but changes its color.
By separating these two jobs, the computer can make the glass look shiny and see-through at the same time, without one job ruining the other.

2. The "Smart Traffic Cop" (Gradient Gating)

When the computer tries to learn what the scene looks like, it makes mistakes. Sometimes, it tries to fix a blurry reflection by accidentally putting "ghost" objects (floaters) inside the glass, which then block the view of the car seats behind it.

To stop this, the authors added a Specular-Aware Gradient Gating mechanism.
The Analogy: Think of this as a smart traffic cop at a busy intersection.

When the computer sees a complex, shiny reflection (like a sparkly car hood), the traffic cop puts up a "Do Not Enter" sign for the "Transmission" lane.
This stops the computer from trying to "fix" the reflection by messing up the view through the glass.
If the area is just a plain, clear window, the cop lets the traffic flow normally so the computer can learn what's behind the glass.

This keeps the "ghosts" away and ensures the background stays clear.

3. The Result: A Hybrid Renderer

Because of this separation, the system uses a hybrid pipeline:

It treats the glass as a surface to calculate the shiny reflections (like a mirror).
It treats the glass as a volume to calculate what shines through it (like a window).

What Can You Do With This?

The paper shows that because the system understands the difference between the "surface" and the "see-through" parts, you can edit the scene easily.

You can make the glass more or less transparent.
You can make the reflections shinier or duller.
You can even change the tint of the glass (like turning clear glass into blue glass) without breaking the 3D model.

Summary

RT-Splatting solves the "glass problem" in 3D photos by giving every tiny piece of glass two jobs: one to act as a mirror and one to act as a window. It also uses a smart filter to stop the computer from getting confused and creating fake objects. The result is a 3D scene that looks real, with sharp reflections and clear views through windows, all while running fast enough to be used in real-time.

Technical Summary: RT-Splatting

Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized real-time novel view synthesis, existing methods struggle with semi-transparent specular surfaces (e.g., glass windows, plastic films) where reflection and transmission coexist. Standard 3DGS approaches often hallucinate "floaters" behind surfaces to capture high-frequency specular highlights, which inadvertently occludes the background, corrupting transmission clarity. Conversely, methods attempting to preserve transmission often fail to model sharp reflections, resulting in blurry specularities.

Current state-of-the-art variants face specific limitations:

Deferred Shading approaches (e.g., 3DGS-DR, Ref-GS) typically store only the nearest surface properties in G-buffers, making it fundamentally difficult to handle transparency and accumulate background radiance.
Multi-stage pipelines (e.g., TransparentGS) often reconstruct the background separately by masking out transparent regions. This fails in scenarios where the background is exclusively visible through the transparent object (e.g., viewing a car interior solely through its windows).
Joint optimization ambiguity: Simultaneously optimizing reflection and transmission is ill-posed. Residual errors in fitting high-frequency specular reflections often generate misleading gradients that leak into the transmission branch, causing the model to create erroneous floaters to compensate for the mismatch.

Methodology

The authors propose RT-Splatting, a hybrid surface-volume rendering framework that disentangles geometric occupancy from optical opacity within a unified Gaussian representation.

1. Occupancy-Opacity Factorization

The core innovation is the factorization of the standard per-Gaussian opacity into two distinct, learnable attributes:

Geometric Occupancy ( $\sigma$ ): Encodes the probability that a ray interacts with the substance of the Gaussian. This determines the Gaussian's role as a surface element for reflection modeling.
Optical Opacity ( $\alpha$ ): Specifies the conditional probability that a ray is absorbed or scattered once an interaction occurs. This controls light attenuation for transmission.

This factorization allows a single set of Gaussians to represent a surface that is geometrically solid (high $\sigma$ for sharp reflections) yet optically clear (low $\alpha$ for clear transmission). The effective opacity used for volumetric compositing is defined as $\alpha_{eff} = \sigma \alpha$ .

2. Hybrid Reflection-Transmission Rendering

The framework employs a two-pass rendering pipeline:

Deferred Pass (Reflection): Uses the probabilistic formulation of geometric occupancy to aggregate first-hit surface attributes (normals, roughness, material features) into G-buffers. A specular shading network ( $f_{spec}$ ) then computes high-frequency, view-dependent reflections based on these attributes.
Forward Pass (Transmission): Operates as a standard volumetric renderer, accumulating background radiance ( $C_{trans}$ ) using the effective opacity $\alpha_{eff}$ .
Subsurface Transport & Blending: To model intrinsic material appearance (e.g., colored glass), the method introduces an intrinsic scattered color ( $C_{scatter}$ ) and a transmissivity ratio ( $\tau$ ). These are combined with the transmitted background to form a subsurface component ( $C_{sub}$ ).
Dynamic Attenuation: A learnable attenuation factor ( $\beta$ ) modulates the subsurface component based on the strength of the specular highlight. This mimics the perceptual observation that transmitted details are visible through faint reflections but suppressed by strong specularities. The final color is computed as $C = C_{spec} + \beta C_{sub}$ .

3. Specular-Aware Gradient Gating

To address the ambiguity in joint optimization, the authors introduce a Specular-Aware Gradient Gating mechanism.

Mechanism: The system estimates the complexity of specular patterns in a pixel by computing the local variance of the specular component ( $C_{spec}$ ).
Function: A gating weight $g(x)$ is calculated based on this variance. During backpropagation, this weight attenuates the gradients flowing from the image loss to the transmission branch in regions dominated by complex specular patterns.
Effect: This prevents the transmission branch from "hallucinating" floaters to compensate for reflection fitting errors, thereby preserving the clarity of the transmitted background while allowing full supervision in non-specular regions.

4. Optimization and Regularization

The method performs joint optimization of Gaussian primitives, factorized opacities, and the shading function. To prevent "ghost" geometries (Gaussians with high occupancy but near-zero opacity) from accumulating in diffuse regions, a transparent mask regularization is applied. This uses a pre-trained semantic mask (from SAM2) to supervise the optical opacity map, ensuring that geometric presence aligns with semantic transparency.

Key Contributions

Unified Surface-Volume Representation: A novel factorization of Gaussian opacity that enables a single set of primitives to simultaneously model sharp specular reflections (as a surface) and clear transmission (as a volume).
Specular-Aware Gradient Gating: A mechanism that suppresses misleading gradients in highly specular regions, effectively reducing floaters and improving background clarity in the transmission branch.
Hybrid Rendering Pipeline: A framework that integrates deferred shading for reflections with volumetric forward rendering for transmission, supporting flexible scene editing and real-time performance.

Experimental Results

The authors evaluated RT-Splatting on public benchmarks (Ref-Real, NeRF-Casting, EnvGS, T&T) and self-captured scenes featuring thin semi-transparent surfaces.

Quantitative Performance: RT-Splatting achieved State-of-the-Art (SOTA) performance across PSNR, SSIM, and LPIPS metrics on both entire images and specifically within transparent regions. It outperformed baselines including 3DGS, 2DGS, GaussianShader, 3DGS-DR, Ref-GS, and EnvGS.
Qualitative Improvements: Visual comparisons demonstrated that RT-Splatting successfully renders sharp reflection details without occluding the background, a failure mode common in prior methods. It effectively handles scenarios where the background is only visible through the transparent object.
Efficiency: The method maintains real-time rendering speeds (approx. 33 FPS) and competitive training times (~0.9 hours), comparable to standard 3DGS variants.
Ablation Studies: Removing any key component (occupancy factorization, joint optimization, scattering, attenuation, or gradient gating) resulted in significant performance degradation, validating the necessity of the full framework.

Significance and Claims

The paper claims that RT-Splatting provides a robust solution for the long-standing challenge of jointly modeling reflection and transmission in real-world scenes. By decoupling geometric occupancy from optical opacity, the method overcomes the inherent trade-offs between reflection fidelity and transmission clarity that plague existing Gaussian Splatting variants.

The authors emphasize that their approach naturally enables flexible scene editing, allowing for independent manipulation of surface roughness, transparency levels, and material tints. While the current framework is designed for thin semi-transparent surfaces (approximating straight-path transmission) and does not explicitly model complex refraction or multi-bounce light transport, it establishes a unified representation that significantly advances the state of the art for reflective and transparent scene reconstruction.