💻 computer science

GaNI: Global and Near Field Illumination Aware Neural Inverse Rendering

The paper presents GaNI, a two-stage neural inverse rendering framework that reconstructs geometry, albedo, and roughness from co-located light and camera images by introducing specific techniques to handle near-field illumination, strong specular reflections, and moving light sources in multi-object scenes.

Original authors: Jiaye Wu, Saeed Hadadan, Geng Lin, Matthias Zwicker, David Jacobs, Roni Sengupta

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaye Wu, Saeed Hadadan, Geng Lin, Matthias Zwicker, David Jacobs, Roni Sengupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out what a room looks like, what the furniture is made of, and how light bounces around it, but you can only take photos with your phone in a dark room while holding a flashlight.

This is the challenge the paper GaNI tackles. It's a new computer program that can turn a bunch of photos into a perfect 3D model of a room, telling you exactly how shiny, rough, or colorful every object is.

Here is the breakdown of how it works, using some everyday analogies.

The Problem: The "Flashlight Fog"

Previous computer programs were great at figuring out a single object (like a vase) in a dark room. But when you put a whole room full of objects (a shoe rack, a table, a coffee mug) together, they got confused.

Why? Because of two things:

The Flashlight Effect: When you hold a flashlight close to a wall, the part right under the light is blindingly bright, but the corner is dark. Old programs assumed the light was far away (like the sun), so they couldn't handle this "near-field" brightness change.
The Echo Effect: Light doesn't just bounce once. It hits a red wall, bounces to a white table, and makes the table look slightly pink. This is called Global Illumination. Old programs ignored these "light echoes," leading to messy, inaccurate 3D models.

The Solution: GaNI (The Two-Stage Detective)

The authors built a system called GaNI that acts like a two-step detective to solve this mystery. Instead of trying to guess the shape and the material at the same time (which is too hard), it solves them one by one.

Stage 1: Building the Skeleton (Geometry)

First, the program tries to figure out the shape of the room.

The Analogy: Imagine trying to sculpt a statue out of clay while someone is shining a blinding flashlight directly at your hands. The glare makes it hard to see the curves.
The Fix: GaNI uses a special trick called Surface Angle Weighting. It knows exactly where the flashlight is. If the light hits a surface straight on (creating a harsh glare) or at a weird angle (creating weird shadows), the program says, "Okay, I'm going to ignore this specific pixel for a moment because the glare is lying to me." It focuses only on the parts of the image where the shape is clear.
The Result: It builds a clean, accurate 3D skeleton of the room, ignoring the confusing glare.

Stage 2: Painting the Skin (Materials)

Once the 3D shape is built, the program moves to the second stage: figuring out what the objects are made of (their Albedo or color, and Roughness).

The Analogy: Imagine you have the 3D skeleton, but now you need to paint it. But here's the catch: the light source (your flashlight) is moving around the room in every photo.
The Problem: Old programs assumed the light stayed in one spot. If the light moved, they got confused and thought the object changed color.
The Fix: GaNI uses a "Smart Memory Bank" (a Radiance Cache). Think of this like a librarian who remembers how light bounced off a table when the flashlight was in the corner, and how it bounced when the flashlight was on the ceiling. Even though the light moves, the program remembers the "light echoes" (global illumination) and subtracts them out to find the true color of the object.
The Smoothness Trick: Sometimes, the 3D skeleton has tiny bumps or errors. If the program tries to guess the roughness based on a bumpy skeleton, the result looks noisy (like static on a TV). GaNI adds a "smoothing rule" that says, "If the surface looks bumpy but shouldn't be, let's smooth out the roughness guess."

Why It Matters

The paper tested this against other top methods (like IRON and WildLight).

The Competition: Other methods either got the shape wrong because of the flashlight glare, or they got the materials wrong because they couldn't handle the light bouncing between objects.
GaNI's Win: GaNI produced 3D models that looked much more realistic. The "re-renders" (photos of the 3D model taken from new angles) looked almost identical to the real photos.

The Bottom Line

Think of GaNI as a master photographer who can look at a messy, dark room lit by a moving flashlight and say:

"I know that bright spot is just a glare, not a shiny object. I know that pinkish tint on the table isn't the table's color; it's just the red wall reflecting light onto it. Let me strip all that away and show you the real 3D room."

It's a big step forward for Virtual Reality (VR), robotics, and computer graphics, allowing us to create digital twins of real-world rooms using nothing more than a smartphone and a dark room.

1. Problem Statement

Inverse rendering aims to decompose a scene into its constituent components: geometry, material properties (albedo and roughness), and lighting. While existing methods have achieved success with single objects or human faces, they struggle with multi-object indoor scenes, particularly when using a co-located light and camera setup (e.g., a smartphone flashlight and camera).

The specific challenges addressed in this paper are:

Near-field Illumination: In co-located setups, the light source is close to the scene, causing significant intensity variations (inverse square law) across different parts of the scene. Existing methods often assume distant or uniform lighting.
Global Illumination (GI): Multi-object scenes exhibit strong inter-reflections (bounces) between objects. Standard inverse rendering techniques often fail to model these complex light bounces, leading to incorrect material estimation.
Specular Reflections: The proximity of the light source creates strong specular highlights that can corrupt geometry reconstruction.
Moving Light Sources: Unlike static light setups, co-located capture involves moving the light/camera, meaning the illumination changes for every view. Existing neural radiosity methods assume static illumination.

2. Methodology: GaNI

The authors propose GaNI, a two-stage neural inverse rendering pipeline designed to handle these complexities.

Stage 1: Geometry Recovery with Volume Rendering

The first stage reconstructs the scene geometry using a modified version of NeuS (Neural Signed Distance Fields).

Implicit Near-Field Modeling: Standard NeuS does not account for the distance between the light source and the surface. GaNI modifies the color network to accept the distance ( $t$ ) from the co-located light as an input. Specifically, it conditions the network on $1/t^2$ to model the inverse quadratic fall-off of light intensity.
Surface Angle Weighting: To handle strong specular reflections from the flashlight that degrade geometry, the authors introduce a surface angle loss function. This function adaptively down-weights pixels where the light ray is either perpendicular (strong specular highlight) or parallel (grazing angle with strong Fresnel effects) to the surface normal. This prevents these regions from dominating the geometry optimization.
Supervision: The system uses standard reconstruction loss (linearized log loss) and, for real data, utilizes Structure-from-Motion (SfM) point clouds to supervise the Signed Distance Field (SDF).

Stage 2: BRDF Estimation with Surface Rendering

Once geometry is established, the second stage estimates material properties (albedo and roughness) using surface rendering and a modified InvNeRad (Inverse Neural Radiosity) framework.

Light Position-Aware Radiance Cache: Standard InvNeRad assumes static lighting. Since GaNI uses a moving light, the authors propose a single radiance cache network that takes the light position ( $x_l$ ) and orientation as inputs. This allows the network to share computed radiance across different light positions rather than training a separate network for every light position.
Direct Illumination Conditioning: To help the network model the sharp discontinuities of direct illumination, the system explicitly computes and conditions the network on the direct illumination term ( $L_{direct}$ ), calculated based on visibility, light intensity, and distance.
Roughness Regularization: To address noise in roughness predictions caused by minor geometry imperfections, a Total Variation (TV) smoothness prior is applied to the roughness map.
Optimization: The system optimizes for spatially varying albedo and roughness by minimizing a composite loss function including reconstruction loss, radiometric prior, roughness regularization, and a Left-Hand Side (LHS) consistency loss.

3. Key Contributions

GaNI Framework: A novel two-stage neural inverse rendering system capable of reconstructing geometry and reflectance for multi-object scenes using co-located light and camera.
Near-Field & Specular Handling:
- Implicit modeling of near-field illumination effects via distance-conditioned color networks.
- A novel surface angle loss function to suppress geometry errors caused by strong specular inter-reflections.
Dynamic Global Illumination:
- A light position-aware radiance cache that extends InvNeRad to handle moving light sources, enabling efficient modeling of global illumination without training thousands of separate networks.
- Integration of explicit direct illumination calculations to aid the neural network.
Roughness Regularization: Introduction of a total-variation prior to stabilize roughness estimation in the presence of imperfect geometry.

4. Experimental Results

The authors evaluated GaNI on 3 synthetic scenes (Bedroom, Shelf, Counter) and 4 real-world scenes (Window Sill, Table, Shoe Rack, Coffee Table).

Comparison with State-of-the-Art (SOTA):
- vs. IRON [37]: GaNI significantly outperforms IRON (which is designed for single objects) in both geometry (lower Chamfer distance and Depth L1 error) and reflectance (lower MSE for albedo and roughness). IRON fails to handle the complex inter-reflections in multi-object scenes.
- vs. WildLight [4]: GaNI produces better geometry and reflectance, particularly in scenes with strong concavity and inter-reflections where WildLight (which assumes static global illumination) fails.
- vs. Natural Illumination (NeILF++ [36]): While NeILF++ works in natural light, GaNI (using a dark room with a flashlight) produces significantly better reflectance (albedo) and comparable or slightly better geometry. The co-located setup resolves the ambiguity between shading and material properties better than natural light.
Ablation Studies:
- Removing the surface angle weighting leads to artifacts in geometry due to inter-reflections.
- Replacing the second stage with IRON's method (Ours-Abl) results in strong artifacts in albedo and poor roughness recovery, proving the necessity of the light-position-aware radiance cache.

5. Significance

GaNI represents a significant step forward in scene-scale inverse rendering. By successfully addressing the unique challenges of near-field lighting and dynamic global illumination in co-located setups, it enables the reconstruction of high-fidelity geometry and physically based materials for complex indoor environments. This moves the field beyond single-object reconstruction toward full room-scale digital twins, which is crucial for applications in VR/AR, robotics, and computational photography. The method demonstrates that a simple, accessible setup (smartphone flashlight in a dark room) can yield superior results to more complex or unconstrained capture strategies when the underlying physics (near-field and GI) are correctly modeled.