💻 computer science

3DSS: 3D Surface Splatting for Inverse Rendering

The paper introduces 3D Surface Splatting (3DSS), a differentiable renderer that leverages a coverage-based compositing model and oriented surface samples to jointly recover shape, spatially-varying materials, and illumination for high-quality inverse rendering while natively bridging to mesh-based workflows.

Original authors: Mae Younes, Adnane Boukhayma

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mae Younes, Adnane Boukhayma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a box of thousands of tiny, flat, colored stickers (called surfels). Each sticker has a specific color, a specific angle it's facing, and a specific size. Your goal is to arrange these stickers in 3D space so that when you take a photo of them from any angle, the picture looks exactly like a real, shiny, 3D object (like a teapot or a car).

This is the core idea behind 3DSS (3D Surface Splatting), a new method for "Inverse Rendering." Inverse rendering is like being a detective who looks at a finished photograph and tries to figure out three things:

Shape: What does the object look like in 3D?
Material: Is it made of shiny metal, rough plastic, or soft fabric?
Lighting: Where was the sun or the lamp when the photo was taken?

Here is how 3DSS solves this puzzle, explained through simple analogies:

1. The Problem with Old Methods

Before this, there were two main ways to solve this puzzle, and both had flaws:

The Mesh Method (The Origami Approach): Imagine trying to build a shape out of a single sheet of paper folded into triangles. It looks sharp and real, but if you want to change the shape, you have to cut and glue the paper. If you try to make it too detailed, the paper gets wrinkled and messy. It's rigid and hard to fix.
The Volume Method (The Fog Approach): Imagine the object is made of thick fog. You can shape the fog easily, but because it's fog, the light gets mixed up inside. If you try to figure out the material, the "fog" makes it look like a blurry mix of colors rather than a distinct surface. It's hard to tell where the actual surface is.

2. The 3DSS Solution: The "Smart Sticker" Approach

3DSS uses a third way: Surface Splatting. Instead of a rigid sheet or a foggy cloud, it uses those thousands of independent stickers.

The Stickers (Surfels): Each sticker is a tiny piece of the surface. It knows its own position, its own angle (normal), and its own material properties (is it shiny? is it red?).
The Magic Trick (Differentiability): The system is "differentiable," which means it can learn from its mistakes. If the photo it creates doesn't match the real photo, the system knows exactly how to nudge the stickers (move them, rotate them, or change their color) to get closer to the truth. It does this mathematically, without needing to cut or glue anything.

3. How It Handles Overlaps (The "Layer Cake" Analogy)

The hardest part of this puzzle is when stickers overlap. If you look at a corner of a box, you might see the front wall and the side wall overlapping in your vision.

Old Way: Previous methods tried to pick just one sticker per pixel (the one closest to the camera). This caused jagged, pixelated edges and made it impossible to see what was behind the front layer.
3DSS Way: 3DSS uses a clever sorting trick called Interval Grouping. Imagine the stickers are sorted by how deep they are. If two stickers are close enough in depth, 3DSS realizes they belong to the same layer (like the front wall). If there is a gap in depth, it knows they belong to a different layer (like the side wall).
The Result: It builds a "layer cake" of stickers. It blends them together smoothly. This creates perfectly smooth edges (anti-aliasing) and allows the system to see through transparent-looking edges to the layers behind, just like real life.

4. How It Handles Lighting (The "Pre-Painted" Analogy)

In many systems, the computer tries to calculate the lighting after it has mixed all the stickers together. This is like mixing different paints and then trying to figure out what the original colors were. It's messy and inaccurate.

3DSS does it the other way around: Forward Shading.

Before the stickers are even arranged into a picture, the system calculates the lighting for each individual sticker. It asks, "If this specific sticker were here, under this specific light, what color would it be?"
It paints each sticker with its final color before they are stacked. Then, it just stacks the already-painted stickers. This ensures the materials (like shiny metal vs. matte plastic) stay true and don't get "muddy" when mixed.

5. The Results

The authors tested 3DSS on a benchmark called Stanford-ORB, which uses real photos of objects like teapots, cars, and salt shakers.

New Views: When they asked the system to show the object from a new angle it had never seen before, 3DSS produced clearer, sharper images than methods using rigid meshes or foggy volumes.
New Lighting: When they asked the system to show the object under a different light (like moving the sun), 3DSS got the reflections and shadows right, whereas other methods often got the materials wrong.
Shape: It reconstructed the 3D shape just as accurately as the best existing methods, but without the "wrinkled paper" problems of the mesh methods.

Why This Matters

The paper claims that 3DSS is the first method to successfully bring Surface Splatting into the world of Inverse Rendering. It bridges the gap between the flexibility of point-based methods (easy to move and change) and the realism of surface-based methods (sharp edges, correct materials).

Most importantly, because the final result is just a collection of oriented points, it can be easily converted back into a standard 3D mesh (a triangle model) that game engines and 3D software can use immediately. It's like taking a pile of smart stickers and instantly turning them into a solid, usable 3D model.

Technical Summary: 3D Surface Splatting (3DSS) for Inverse Rendering

Problem Statement

Inverse rendering aims to reconstruct geometry, materials, and illumination from multi-view photographs to create editable, relightable assets. Current differentiable rendering paradigms face complementary limitations:

Triangle Mesh Rasterizers: While producing sharp images with explicit geometry, they struggle with topology evolution. Optimization often leads to degenerate elements, self-intersections, and broken manifolds, requiring careful regularization.
Volumetric Renderers (NeRFs/3DGS): These offer connectivity-free flexibility and high novel-view synthesis quality. However, their image formation composites radiance along rays rather than at surfaces. This volumetric blending mixes material attributes (like normals) across primitives, resulting in pixel values that do not correspond to the reflectance of any single physical surface, undermining physically-based decomposition.
Surface Splatting: Historically absent from differentiable inverse rendering, surface splatting (specifically Elliptical Weighted Average or EWA) offers a middle ground: it uses oriented point samples (surfels) with explicit surface semantics but lacks a differentiable formulation for visibility and surface separation. Classical methods rely on hard, discontinuous depth tests (ternary z-buffering) to separate surfaces, which are incompatible with gradient-based optimization and prevent correct anti-aliasing at overlapping edges.

Methodology: 3D Surface Splatting (3DSS)

3DSS introduces the first differentiable surface splatting renderer designed for physically-based inverse rendering. It represents scenes as unstructured sets of oriented surfel primitives and reconstructs continuous surfaces via EWA splatting.

1. Surfel Representation and Rasterization

Each surfel is defined by a center position, two tangent vectors (encoding orientation and anisotropic scaling), and shading attributes (albedo, metallic, roughness). The surface normal is implicitly derived from the cross product of the tangents.

T-Matrix: The renderer uses a 4×4 T-matrix to map surfel parameterization to clip space, enabling analytic ray-surfel intersection and bounding box calculation in a numerically stable manner.
Forward Shading: Unlike deferred shading, 3DSS shades each surfel before rasterization using a split-sum Image-Based Lighting (IBL) model. This avoids the non-commutativity issues of shading reconstructed attributes (e.g., averaging normals) and keeps the rasterization kernel compact.

2. Multi-Layer Surface Separation (Depth-Interval Grouping)

A core innovation is the reformulation of surface separation. Instead of binary/ternary depth tests, 3DSS treats surface separation as an interval-merging problem:

Each surfel occupies a depth interval $[z - \varepsilon_z, z + \varepsilon_z]$ based on its depth and kernel extent.
Primitives are sorted by their interval start ( $z_{start}$ ).
Surfels are grouped into surface layers if their depth intervals overlap. A gap between the running maximum depth of a group and the start of the next primitive indicates a new surface layer.
This allows for an arbitrary number of surface layers per pixel in a single rasterization pass, resolving self-occlusion and visibility discontinuities without hard decisions.

3. Coverage-Based Compositing

The renderer derives opacity directly from the accumulated EWA reconstruction weight within each layer:

Coverage Opacity: The accumulated weight $W_k$ of a layer is mapped to a coverage opacity $\alpha_k$ using a smooth function ( $1 - \exp(-\gamma \cdot W_k / 2\pi)$ ). This weight arises naturally from the signal reconstruction process.
Compositing: Layers are composited front-to-back using the standard "over" operator, where the residual transmittance of preceding layers determines the visibility of deeper layers.
Gradient Flow: This mechanism ensures that even at sparsely covered edges (silhouettes), where a single surfel might otherwise produce a constant normalized color, the sub-unity coverage opacity modulates the final pixel value. This provides informative gradients for geometry optimization at object boundaries.

4. Anti-Aliasing and Density Control

Object-Space MIP Filtering: To prevent sampling aliasing, 3DSS employs an object-space MIP filter. It precomputes the inverse covariance matrix and normalization factor at the surfel center, approximating the Jacobian of the ray-splat intersection. This suppresses high frequencies without the per-pixel arithmetic cost of full Jacobian evaluation.
Adaptive Density Control: The method maintains a "partition of unity" assumption (overlapping supports) via a density-aware splitting criterion. It compares a surfel's scale against the local inter-sample spacing (rather than a global scene threshold) and accumulates screen-space gradient norms directly from the T-matrix backward pass to identify regions needing refinement.

Key Contributions

Multi-Layer Surface Separation: An algorithm that partitions sorted primitives into arbitrary surface layers via interval merging, replacing binary/ternary depth tests and supporting multiple layers per pixel.
Coverage-Based Differentiable Compositing: A model where per-layer opacity is derived from accumulated EWA weights, yielding anti-aliased silhouettes and informative gradients at sparse edges, which single-layer formulations cannot provide.
Efficient Object-Space MIP Filter: A center-precomputed approximation that suppresses sampling aliasing with negligible visual loss and minimal computational overhead.
Bridge to Mesh Workflows: The optimized representation (oriented point samples) is natively compatible with established mesh reconstruction algorithms (e.g., Screened Poisson Surface Reconstruction), facilitating a direct pipeline from point-based inverse rendering to mesh-based workflows.

Results

Evaluated on the Stanford-ORB benchmark (14 objects, 7 scenes with ground-truth scans, HDR imagery, and measured lighting), 3DSS was compared against mesh-based (NVDiffRec), implicit (NeRF, NeRFactor), and Gaussian-splatting (3DGS-based) baselines.

Novel View Synthesis: 3DSS achieved the highest scores across all metrics (PSNR, SSIM, LPIPS), outperforming both mesh-based and implicit baselines. The point-based representation's topological flexibility allowed for better recovery of fine details compared to fixed-grid mesh methods.
Novel Scene Relighting: 3DSS achieved state-of-the-art structural similarity (SSIM) and perceptual (LPIPS) scores, surpassing NVDiffRec and implicit methods. It reproduced specular highlights and diffuse shading more faithfully, particularly on objects with high specularity.
Geometry Reconstruction: 3DSS achieved geometry scores (depth, normals, Chamfer distance) on par with the strongest baselines (NVDiffRec, InvRender). Qualitative analysis showed smoother and more coherent surface normals compared to mesh-based methods.
Fixed-Geometry Validation: When surfels were sampled from ground-truth scans with fixed geometry, the renderer produced high-fidelity shading, confirming the surfel representation's compatibility with existing mesh assets and the physical plausibility of the shading pipeline.

Significance and Claims

The paper claims that 3DSS occupies a distinct and practically valuable position in differentiable rendering by combining the connectivity-free flexibility of point-based methods with the surface-correct image formation of mesh rasterizers.

Differentiability: By formulating surface separation and opacity as continuous, differentiable processes, 3DSS enables joint optimization of shape, spatially-varying BRDFs, and illumination without the topological constraints of meshes or the physical ambiguity of volumetric blending.
Workflow Integration: The method provides a "direct bridge" between point-based inverse rendering and downstream mesh-based workflows, as the optimized surfels can be directly fed into standard surface reconstruction algorithms.
Revival of Surface Splatting: The work demonstrates that surface splatting, previously hindered by visibility discontinuities, can be successfully adapted for inverse rendering, offering a viable alternative to the dominant mesh and volumetric paradigms.

The authors note limitations, including the use of a simplified split-sum IBL shading model (lacking global illumination) and the assumption of opaque surfaces, but emphasize that the rendering backend is agnostic to the shading model, allowing for future extensions to more complex light transport.