💻 computer science

Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark

This paper proposes a 3D-aware neural modeling approach for robust low-light RGB-NIR imaging that effectively recovers clean RGB images by fusing noisy RGB observations with NIR cues in 3D space, eliminating the need for clean RGB supervision while demonstrating superior generalization across varying noise levels.

Original authors: Muyao Niu, Mingze Ma, Yifan Zhan, Qingtian Zhu, Zhihang Zhong, Wei Guo, Chang Wen Chen, Yinqiang Zheng

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muyao Niu, Mingze Ma, Yifan Zhan, Qingtian Zhu, Zhihang Zhong, Wei Guo, Chang Wen Chen, Yinqiang Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to take a perfect photo in a pitch-black room. You point your camera, press the shutter, and get back a grainy, chaotic mess of static. This is the daily struggle of "low-light imaging" in computer vision. For decades, scientists have tried to fix this by either waiting longer for the camera to gather light (which causes blurry motion) or by using complex math to guess what the picture should look like. But there's a clever trick: Near-Infrared (NIR) light. While our eyes can't see it, cameras can, and NIR behaves like a ghostly flashlight that reveals the shape and texture of objects even in total darkness, though it shows them in a monochrome, gray scale. The big question has always been: Can we use this ghostly gray map to fix our noisy, colorful photos without needing a "perfect" reference photo to teach the computer what to do? Usually, to teach a computer to clean up a photo, you need to show it a dirty version and a clean version side-by-side. But in the real world, getting that clean version in the dark is often impossible.

This paper, titled "Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark," proposes a new way to solve this puzzle. The authors, a team from the University of Tokyo and other institutions, suggest that instead of trying to clean a single 2D image, we should build a 3D model of the scene first. Think of it like this: if you have a blurry, noisy photo of a Lego castle taken in the dark, and a clear, gray photo of the same castle taken with an infrared camera, the old methods try to paste the gray details onto the color photo like a sticker. This paper argues that's the wrong approach. Instead, they build a virtual 3D sculpture of the castle in the computer's mind. They use the clear gray (NIR) photo to define the shape of the sculpture, and then they use the noisy color (RGB) photo to paint it, but they do it in a way that forces the paint to stick only where the shape makes sense.

The researchers found that by combining these two types of light in a 3D space, they could reconstruct a clean, colorful image without ever seeing a "clean" version of the original photo. They discovered two major hurdles in this process. First, the computer kept trying to learn the random static (noise) in the color photo as if it were real detail, creating a weird "checkerboard" pattern. To fix this, they taught the computer to listen to the infrared photo for its "rhythm" (frequency) rather than the noisy color photo, effectively silencing the static. Second, they realized that the same shade of gray in the infrared photo could correspond to many different colors in real life (a gray wall could be red, blue, or green). To solve this, they added a special "color code" system that lets the computer guess the right color from a list of possibilities, rather than just blending them into a muddy brown.

The team tested their idea on both computer-generated scenes and real-world captures, including a scene with an owl and another with a rover. They compared their method against many existing techniques, including those that rely on massive amounts of clean training data. The results showed that their 3D-aware method produced sharper images with better colors and fewer artifacts, even when the noise was extremely severe. They demonstrated that this approach works across different levels of darkness and noise without needing the impossible "clean" reference photos that other methods require. While the system currently works best on still scenes and struggles if the cameras aren't perfectly aligned, the paper suggests this is a significant step toward making cameras that can see clearly in the dark, using only the light they can already capture.

Technical Summary: Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark

Problem Statement

Robust low-light imaging remains a fundamental challenge in computer vision. While conventional strategies like long-exposure and burst photography suffer from motion blur and visibility issues, and existing deep learning-based enhancement methods often rely on "noisy RGB–NIR–clean RGB" triplets, these supervised models exhibit poor cross-domain generalization and limited robustness under varying noise levels. A critical gap exists in achieving robust low-light enhancement using only Near-Infrared (NIR) and noisy RGB observations, without relying on any form of clean RGB supervision.

Methodology

The paper proposes a novel 3D-aware neural implicit fusion architecture that leverages a volume rendering framework to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space. The approach is built upon Multi-Layer Perceptrons (MLPs) and volume rendering, redesigned to exploit the complementary properties of RGB-NIR modalities without clean RGB ground truth.

Key Architectural Components

3D-Aware Fusion Structure (NIR-Conditioned MLP):
- Unlike a parallel structure where separate MLPs handle RGB and NIR, the proposed method employs an NIR-conditioned structure.
- An NIR MLP estimates volume density ( $\sigma$ ) using the NIR observation.
- The output of the NIR MLP is fed into the RGB MLP to provide structural guidance. Crucially, a gradient-stopping strategy prevents the noisy RGB observations from degrading the learning of the NIR representation.
- This design allows the RGB MLP to leverage the structural information encoded in NIR while maintaining the integrity of the NIR representation, effectively recovering fine 2D texture details that pure 3D consistency might miss.
NIR-Modulated Positional Encoding (NIR-P.E.):
- Standard sinusoidal positional encoding (P.E.) applied to 3D coordinates ( $x$ ) and view directions ( $d$ ) often leads to "checkerboard artifacts" under severe noise, as the network overfits to high-frequency noise rather than structural details.
- Analysis of the frequency domain reveals that clean RGB and NIR share similar smooth frequency distributions, whereas noisy RGB is dominated by random high-frequency noise.
- To address this, the authors apply P.E. to the NIR estimation ( $c_N$ ) instead of the spatial coordinates. The RGB MLP takes $x$ , $d$ , and the encoded $\gamma(c_N)$ as input.
- This acts as a frequency-aligned modulation, amplifying structurally consistent frequencies present in NIR while suppressing noise-driven components from the noisy RGB supervision.
Color Code MLP (C.C. MLP):
- The mapping from NIR to RGB is inherently ill-posed; a single NIR value can correspond to multiple distinct RGB colors (e.g., different colored objects with similar reflectance in NIR). Standard MLPs, due to their smoothness, tend to produce "blended" or averaged RGB estimates for points with identical NIR values.
- To resolve this ambiguity, a Color Code MLP is introduced to learn a probabilistic distribution of colors for each NIR value.
- It predicts a non-uniform log probability distribution ( $\pi$ ) and uses Gumbel-Softmax to sample a one-hot color code ( $\delta$ ).
- This code is fed into the RGB MLP, enabling the model to distinguish between different RGB colors that share the same NIR value, thereby preserving diverse color representations.

Optimization

The model is optimized by minimizing the L2 photometric loss between the rendered estimations and the observations. Notably, the loss function does not include clean RGB supervision. Instead, it minimizes the difference between the rendered NIR/RGB and the observed noisy RGB/NIR. The noisy RGB observation is amplified by a hyperparameter $\phi$ to account for exposure differences.

Key Contributions

A New 3D-Aware Fusion Model: The paper proposes a model for RGB-NIR dark imaging that effectively fuses noisy RGB observations with NIR cues in 3D space without requiring clean RGB supervision.
Insightful Architectural Components: Two specific components are introduced to unlock the complementary potential of RGB-NIR modalities:
- NIR-Modulated Positional Encoding: To suppress noise-induced overfitting and eliminate checkerboard artifacts.
- Color Code MLP: To resolve the fundamental NIR-to-RGB ambiguity and prevent color blending.
Datasets: The authors contribute both synthetic and real-world multi-view datasets containing NIR images and noisy RGB observations to facilitate future research.

Results and Evaluation

The proposed framework was evaluated on both synthetic datasets (generated via Mitsuba3) and real-world captures (using a JAI FS-3200T10GE-NNC camera).

Synthetic Data: The model was tested across five noise levels ( $s \in \{1/10, \dots, 1/200\}$ ). It consistently outperformed state-of-the-art supervised methods (SANet, NAID), unsupervised methods (NVEU), and other NeRF-based approaches (RawNeRF, LLNeRF) in terms of SSIM, PSNR, and LPIPS.
Real-World Data: Evaluations included non-reference metrics (PI, MUSIQ, MANIQA) and Human Subjective Evaluation (HSE). The proposed method achieved the best scores across all metrics, demonstrating superior structural integrity and color fidelity compared to existing baselines.
Ablation Studies: Removing the NIR-P.E. resulted in severe checkerboard artifacts. Removing the C.C. MLP led to blending artifacts where distinct colors were merged. The full model was necessary for high-fidelity reconstruction.
RAW Domain Generalization: The method was also evaluated in the 12-bit RAW domain, showing it generalizes effectively and outperforms baselines like RawNeRF and NVEU without requiring white balance or color correction.

Significance and Claims

The paper claims that robust low-light enhancement can be achieved using only NIR and noisy RGB observations, eliminating the need for difficult-to-acquire clean RGB supervision data. By incorporating 3D-aware neural modeling, the method generalizes across different noise levels and scenarios where traditional supervised models fail. The work demonstrates that implicit neural representations, when carefully redesigned to account for the specific characteristics of RGB-NIR modalities (frequency alignment and color ambiguity), can effectively recover clean images from extremely noisy inputs. The authors emphasize that this approach provides a new perspective for dark imaging, offering a solution that is both robust and generalizable without the constraints of paired training data.