💻 computer science

Ambient-robust Inverse Rendering using Active RGB-NIR Imaging

This paper presents an ambient-robust inverse rendering method that leverages active RGB-NIR imaging and a novel three-stage pipeline to achieve accurate geometry and reflectance reconstruction by utilizing NIR flash illumination to stabilize shading against varying ambient lighting conditions.

Original authors: Hoon-Gyu Chung, Jinnyeong Kim, Hyunwoo Kang, Seung-Hwan Baek

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hoon-Gyu Chung, Jinnyeong Kim, Hyunwoo Kang, Seung-Hwan Baek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a perfect photo of a shiny, complex statue to recreate it in a video game. The problem? The room is full of unpredictable, messy sunlight and shadows. If you try to scan the statue now, the computer gets confused: "Is that shiny spot because the statue is made of metal, or because a sunbeam hit it?" This confusion makes it hard to separate the object's true shape and color from the lighting around it.

This paper presents a clever solution: a robot that uses "invisible flashlights" to see through the mess.

Here is how it works, broken down into simple steps:

1. The Problem: The "Ambient Noise"

Think of normal cameras like people trying to listen to a whisper in a crowded, noisy concert hall. The "whisper" is the true color and shape of the object, and the "noise" is the ambient light (sunlight, room lamps) bouncing everywhere. Existing methods try to guess the whisper by listening very hard, but they often get it wrong because the noise is too loud.

2. The Solution: The "Invisible Flash"

The authors built a special camera system that sees two things at once:

Visible Light (RGB): What our eyes see.
Near-Infrared (NIR): A type of light humans cannot see, but the camera can.

They mounted a powerful NIR flash on a robot arm. Because this light is invisible to humans, the robot can blast the object with a bright, controlled spotlight without annoying people or changing the "mood" of the room.

The Analogy: Imagine trying to read a book in a dark room with a flickering candle (ambient light). It's hard to see the text clearly. Now, imagine you have a flashlight that shines a beam of invisible light. You can turn it on to read the text perfectly, and then turn it off to see the candlelight again. The invisible light gives you a clean, stable reference that the candle can't mess up.

3. The Robot "Scanner"

The team built a mobile robot (like a Roomba with an arm) that carries this special camera and flash. It drives around an object, stopping at many different angles to take hundreds of photos.

It takes a photo with the NIR flash ON.
It takes a photo with the NIR flash OFF.
By subtracting the "OFF" photo from the "ON" photo, the computer mathematically removes all the messy ambient light, leaving only the clean, pure reflection from their invisible flashlight.

4. The Three-Step "Magic Trick" (The Algorithm)

The computer uses these photos in a three-stage process to build a perfect 3D model:

Stage 1: The Rough Sketch (Using Normal Photos)
First, it looks at the regular color photos (the ones with the messy room light) to get a rough idea of the object's shape. It's like sketching the outline of a face before adding details. This step is good enough to get the general shape, even if the lighting is weird.
Stage 2: The "Truth" Layer (Using the Invisible Flash)
This is the most important part. The computer looks at the "clean" photos taken with the invisible NIR flash. Because this light is controlled and stable, the computer can perfectly calculate:
- How rough or smooth the surface is (like sandpaper vs. glass).
- Whether the material is metallic or plastic.
- The exact 3D bumps and dents.
  Think of this as the computer putting on "X-ray glasses" to see the true texture of the object, ignoring the messy room light entirely.
Stage 3: The Final Paint Job (Combining Everything)
Now that the computer knows the exact shape and texture (from Stage 2), it goes back to the regular color photos. It uses the "truth" it learned to figure out the object's true colors (albedo) and what the room's lighting actually looks like. It separates the object's paint from the shadows cast by the room.

5. The Result: A "Re-lightable" Object

The final output is a digital 3D model that is "physically correct."

The Magic: Because the computer knows the true shape and material, you can take this model and put it into a video game or a virtual world. You can change the lighting from "sunny beach" to "dark cave," and the object will react naturally. The shiny parts will shine, and the matte parts will stay dull, just like in real life.

Why This Matters

Previous methods either needed a dark, controlled studio (which is impractical for real life) or failed when the lighting was messy. This method works in the real world—indoors or outdoors—because the "invisible flashlight" cuts through the noise.

The authors also released a new dataset (a collection of these scans) so other researchers can test their own ideas against this new standard. They proved that by using this invisible light, they can reconstruct objects much more accurately than before, even when the sun is shining brightly or the room is dimly lit.

Technical Summary: Ambient-robust Inverse Rendering using Active RGB-NIR Imaging

Problem Statement

Inverse rendering aims to reconstruct the geometry and reflectance of objects from images. However, existing methods struggle significantly under uncontrolled ambient illumination. Passive approaches relying solely on RGB images face inherent ambiguities between surface reflectance and environmental lighting, leading to inconsistent and inaccurate reconstructions. While active illumination techniques can resolve these ambiguities, traditional visible-light flash systems are impractical for "in-the-wild" scenarios because strong visible light disturbs human observers and the environment, typically requiring darkroom conditions.

Methodology

The authors propose an ambient-robust inverse rendering framework that leverages active RGB-NIR imaging. The core insight is that Near-Infrared (NIR) light is imperceptible to humans, allowing for the use of strong, controlled NIR flash illumination without disturbing the scene. This provides stable point-light shading that remains invariant to ambient lighting conditions, while simultaneously capturing naturally illuminated RGB images for visible-spectrum reflectance.

The system and algorithm operate through the following components:

1. Active Imaging System

The authors developed a mobile robotic platform consisting of:

A four-wheeled mobile base (AgileX Ranger Mini V2).
A robotic arm (AgileX Piper) mounted with a pixel-aligned RGB-NIR camera (JAI FS-1600) and a synchronized NIR flash (Advanced Illumination AL295).
Operation: The system autonomously scans objects by moving the base in a circular trajectory and varying the arm's elevation. It captures High Dynamic Range (HDR) images with and without the NIR flash.
Data Processing: The NIR flash-only image ( $I_{NIR}$ ) is isolated by subtracting the ambient-only NIR image ( $I_{NIR-off}$ ) from the flash-on image ( $I_{NIR-on}$ ). Object masks are generated using SAM3, and camera poses are estimated via COLMAP.

2. Three-Stage Inverse Rendering Pipeline

The reconstruction process is divided into three sequential stages to ensure robustness:

Stage 1: Geometry Initialization
- Uses multi-view ambient RGB images to initialize the scene geometry.
- Employs 2D Gaussian Splatting to represent the scene, optimizing geometric parameters (center, scale, rotation, opacity) and view-dependent radiance.
- The radiance parameters are discarded after this stage; only the geometric structure is retained.
Stage 2: NIR Flash Inverse Rendering (The Core Innovation)
- Uses multi-view NIR flash images to refine geometry and estimate NIR reflectance.
- Because the NIR flash dominates the scene, the shading is stable and robust to ambient variations.
- Modeling: NIR reflectance is modeled using a basis BRDF formulation (weighted sum of Disney BRDFs). The system optimizes NIR diffuse albedo, roughness, and metallic parameters for each Gaussian.
- Cross-Spectral Transfer: The estimated roughness and metallic parameters are assumed to be wavelength-independent (micro-geometry and material-type properties). These are shared with the subsequent RGB stage to constrain the solution space.
Stage 3: RGB Environment Inverse Rendering
- Recovers the RGB diffuse albedo and the RGB environment map using the multi-view ambient RGB images.
- Geometry, roughness, and metallic parameters are fixed from Stage 2.
- Rendering: Uses a physics-based Monte Carlo integration with Multiple Importance Sampling (MIS) to solve the rendering equation, separating direct environment lighting from indirect illumination.
- Regularization: A specific edge-aware loss term uses the shadow-free NIR albedo to guide the RGB albedo estimation, preventing shadows from being "baked" into the diffuse texture.

Key Contributions

Ambient-Robust Framework: A three-stage RGB-NIR inverse rendering method that utilizes active NIR flash to decouple geometry and reflectance estimation from uncontrolled ambient lighting.
Automated Active Imaging System: A mobile robotic platform capable of dense, object-centric multi-view capture of synchronized RGB-NIR data without disturbing the environment or observers.
RGB-NIR Inverse Rendering Dataset: The first dataset of its kind, containing multi-view RGB-NIR images (with and without flash) captured under multiple ambient illumination conditions for both real-world and synthetic scenes.

Results and Evaluation

The method was evaluated on both synthetic (Mitsuba 3) and real-world datasets across diverse lighting environments.

Quantitative Performance: The proposed method significantly outperforms state-of-the-art passive RGB methods (e.g., NeILF, TensoIR, GS-IR, R3DG, IRGS) and active RGB methods (e.g., WildLight) in terms of PSNR, SSIM, LPIPS, and geometric error (RMSE, MAE).
Ambient Robustness: The system produces consistent reflectance and environment map estimations across four different indoor and outdoor lighting conditions, whereas passive methods fail to disentangle lighting from material properties.
Relighting: The reconstructed PBR assets enable realistic relighting under novel illumination setups, demonstrating natural specular highlights and shading.
Ablation Studies:
- Removing the NIR flash stage (Stage 2) leads to a substantial drop in accuracy, confirming the necessity of controlled NIR point-light shading.
- The method remains robust across different material types (diffuse, specular, metallic) and scene complexities (multiple objects).

Significance and Claims

The paper claims to address the critical limitation of ambient sensitivity in inverse rendering. By exploiting the imperceptibility of NIR light, the authors demonstrate that it is possible to achieve accurate, automatic, and robust scene reconstruction in everyday environments without the need for darkrooms or intrusive visible lighting. The work establishes a new benchmark for inverse rendering under uncontrolled lighting and provides the necessary hardware and dataset infrastructure to advance the field. The authors position this as a step toward practical, automatic, and robust scene reconstruction for real-world applications.