💻 computer science

InvSplat: Inverse Feed-Forward Scene Splatting

InvSplat is a feed-forward multi-view reconstruction framework that directly predicts a structured 3D Gaussian representation with intrinsic material attributes, enabling physically grounded inverse rendering, accurate material recovery, and stable novel view synthesis in a single forward pass.

Original authors: Polina Karpikova, Wenjing Bian, Haofei Xu, Hendrik Lensch, Andreas Geiger

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Polina Karpikova, Wenjing Bian, Haofei Xu, Hendrik Lensch, Andreas Geiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a set of photos taken of a room from different angles. Usually, if you want to create a 3D model from these photos, you have two choices:

The Slow, Perfect Way: You spend hours (or days) manually tweaking the 3D model until it looks perfect. This is like sculpting a statue by hand, chipping away slowly. It's accurate, but it takes forever for every single new room.
The Fast, "Flat" Way: You use a smart AI that looks at the photos and instantly spits out a 3D model. But, this model is often just a "skin" painted with colors. It doesn't really understand that a shiny ball is made of metal or that a wall is rough plaster. If you try to move the camera or change the lighting, the model looks weird or breaks.

InvSplat is a new method that combines the speed of the second approach with the deep understanding of the first. Think of it as a super-fast, instant 3D sculptor that also understands physics.

Here is how it works, using some everyday analogies:

1. The "Magic Cloud" of Tiny Orbs

Instead of building a 3D model out of triangles (like a low-poly video game character), InvSplat builds the scene out of millions of tiny, invisible 3D clouds (called Gaussians).

Imagine a foggy room where every single drop of fog knows exactly where it is, how big it is, and how it spins.
In the past, these "clouds" only knew how to reflect light to look like a picture.
InvSplat's Twist: Each of these tiny clouds is now "smart." It carries a full ID card with specific properties:
- Albedo: What color is the object if there were no shadows? (Like the true color of a red apple).
- Metallic: Is it shiny like a car or dull like a rock?
- Roughness: Is it smooth like glass or bumpy like sandpaper?
- Geometry: Where is it, and what shape is it?

2. The "One-Click" Recipe

Old methods that tried to figure out these properties (like "is this shiny?") had to do a slow, step-by-step calculation for every single scene. It was like a chef tasting a soup, adding salt, tasting again, and repeating for an hour.

InvSplat is like a high-tech microwave. You put the photos in (the "ingredients"), press a button, and in 1.5 seconds, it serves you a fully cooked 3D meal. It doesn't taste-test; it uses a massive library of knowledge (trained on thousands of scenes) to guess the recipe instantly.

3. Why "Consistency" Matters

If you look at a shiny spoon in a photo, a "flat" AI might paint the reflection on the spoon. But if you move your head, the reflection should move. Flat AIs often get confused here, making the reflection look like a sticker that doesn't move correctly.

Because InvSplat builds a real 3D object with real materials, the reflections and shadows behave correctly no matter where you stand.

The Analogy: Imagine a flat painting of a mirror vs. a real mirror. If you walk past the painting, the reflection in the painting stays the same. If you walk past a real mirror, the reflection changes. InvSplat builds the "real mirror," so the reflections update naturally as you move.

4. The "Relighting" Superpower

Because the AI separates the material (what the object is made of) from the lighting (where the light is coming from), you can do something magical: Relighting.

Imagine you took a photo of a room with a lamp on. InvSplat figures out that the wall is "white paint" and the lamp is "bright yellow."
Once it knows this, you can tell the computer: "Okay, turn off that lamp and turn on a blue neon sign."
The computer instantly recalculates how the blue light hits the white paint, creating a brand-new image that looks physically real, without needing to take a new photo.

Summary of What It Does

Input: A few photos of a scene with known camera positions.
Process: A single, fast pass through a neural network (no slow waiting).
Output: A 3D scene made of "smart clouds" that know their own color, shininess, and roughness.
Result: You can walk around this 3D scene, change the lights, and see realistic reflections and shadows, all generated in real-time.

In short, InvSplat is the first tool that can look at a few photos and instantly build a physically accurate 3D world that you can edit and explore, doing in seconds what used to take hours or was impossible to do consistently.

Technical Summary: InvSplat: Inverse Feed-Forward Scene Splatting

Problem Statement

Inverse rendering aims to recover 3D geometry and physically meaningful surface reflectance from images, a prerequisite for applications like relighting, material editing, and realistic AR/VR content insertion. Existing methods face a trade-off between efficiency, physical interpretability, and multi-view consistency:

Optimization-based approaches (e.g., IRIS) achieve high fidelity but require computationally expensive per-scene fitting, limiting practical utility.
Learning-based approaches (e.g., DiffusionRenderer, MVInverse) offer faster inference but predominantly operate in image space. They lack explicit 3D scene representations, leading to multi-view inconsistencies and an inability to support robust novel view synthesis or physically grounded editing.
Feed-forward 3D reconstruction methods (e.g., pixelSplat, MVSplat) generate explicit 3D representations (like 3D Gaussians) efficiently but focus on synthesizing RGB appearance rather than disentangling intrinsic material properties (albedo, metallic, roughness). They often fail to reproduce sharp specular highlights or allow explicit relighting.

Methodology

InvSplat introduces a feed-forward framework that directly predicts a structured 3D Gaussian representation with intrinsic material attributes from posed multi-view images in a single forward pass.

Scene Representation

The scene is represented by $M$ 3D Gaussian primitives. Unlike standard 3DGS which uses spherical harmonics for appearance, each Gaussian $j$ in InvSplat is parameterized by:

Geometry: Mean ( $\mu_j$ ), Rotation ( $q_j$ ), Scale ( $s_j$ ), Opacity ( $\sigma_j$ ), and Surface Normal ( $n_j$ ).
Intrinsic Materials: Diffuse Albedo ( $a_j$ ), Metallicity ( $m_j$ ), and Roughness ( $r_j$ ).

This allows for a disentangled, physically grounded representation where illumination is separated from material properties.

Architecture

The model employs a dual-branch feed-forward network:

Geometry Branch: Inspired by ReSplat, it uses a ResNet backbone to extract multi-scale features. A Multi-view Geometry Encoder (transformer) aggregates cross-view information, followed by a Multi-view Feature Matching module that constructs a depth-candidate cost volume.
Intrinsic Branch: Utilizes a DINOv2 ViT-L/14 backbone to encode input views. A Multi-view Intrinsic Translator (a 36-block transformer) refines these features through intra-view and inter-view self-attention to extract cross-view intrinsic features.
Decoding Heads: Features from both branches are processed by six decoding heads:
- Depth: Estimated via a DPT head using combined features.
- Gaussian Parameters: Rotation, scale, and opacity are refined using a Point Transformer that lifts 2D features into 3D context based on predicted depth.
- Materials & Normals: Albedo, metallicity, roughness, and normals are predicted via DPT heads. Albedo prediction includes skip connections to higher-resolution image cues.
Unprojection & Rendering: Predicted depth maps are unprojected to 3D to form Gaussian centers. The scene is rendered using a differentiable Gaussian rasterizer to produce maps for albedo, metallic, roughness, normal, and depth.

Training

The model is trained on the InteriorVerse dataset using a combination of pixel-wise and perceptual losses:

Material Supervision: $L_1$ and LPIPS losses for albedo, metallic, and roughness.
Depth Supervision: An affine-invariant loss (inspired by MoGe) to handle scale ambiguity.
Normal Supervision: Cosine similarity loss.
The training jointly optimizes the geometry encoder, intrinsic translator, and decoding heads.

Key Contributions

First Feed-Forward Inverse Rendering Framework: InvSplat is the first method to predict physically based 3D Gaussian primitives with intrinsic material parameters from multi-view images in a single forward pass, eliminating the need for per-scene optimization.
Unified 3D Representation: By integrating priors from image-based material estimation and 3D reconstruction models, it produces high-quality 3D material properties with strong generalization to real-world data.
Enabling Relighting and Consistent NVS: The disentangled 3D Gaussian representation supports novel view synthesis with decomposed materials, enabling physically based relighting and view-dependent effect modeling that is consistent across viewpoints.

Results

Experiments were conducted on synthetic (InteriorVerse, Structured3D) and real-world (RealEstate10K, DL3DV) datasets.

Quantitative Performance: On the InteriorVerse test set, InvSplat achieves performance comparable to 2D baselines (like MVInverse) in terms of PSNR, SSIM, and LPIPS for albedo and normals, while providing an explicit 3D representation.
Multi-View Consistency: On the Structured3D dataset, InvSplat demonstrates superior cross-view consistency (lower reprojection RMSE) compared to 2D methods, particularly for challenging attributes like metallicity and roughness.
Qualitative Results: Visualizations show that InvSplat avoids "baking" reflections into the albedo, producing finer-grained details and more consistent predictions across views compared to 2D baselines which often exhibit drift around reflective surfaces.
Efficiency: The method achieves reconstruction in approximately 1.5 seconds, enabling real-time rendering.

Significance

The paper claims that InvSplat bridges the gap between efficient feed-forward inference and high-fidelity inverse rendering. By moving from image-space predictions to an explicit 3D Gaussian representation with intrinsic parameters, the method overcomes the multi-view inconsistency issues of prior learning-based approaches and the computational bottlenecks of optimization-based methods. This enables instantaneous scene decomposition, facilitating downstream applications such as relighting and material editing that were previously difficult to achieve with feed-forward models.