💻 computer science

Frequency-Decomposed INR for NIR-Assisted Low-Light RGB Image Denoising

This paper proposes FDINR, a Near Infrared-assisted low-light image denoising method that employs a frequency-decoupled implicit neural representation with cross-modal differentiated supervision to effectively restore luminance and high-frequency texture details while enabling arbitrary-resolution reconstruction.

Original authors: Ligen Shi, Zengyu Pang, Chang Liu, Shuchen Sun, Jun Qiu

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ligen Shi, Zengyu Pang, Chang Liu, Shuchen Sun, Jun Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a beautiful photo of a forest at night using a drone. The problem is, it's so dark that your camera's "visible light" sensor (the RGB camera) is struggling. It's like trying to read a book in a pitch-black room; you end up squinting so hard that the image comes out grainy, blurry, and the colors look weird (too green or too purple).

However, your drone also has a special "Night Vision" camera (the NIR camera). This camera sees the world differently. It doesn't care about color, but it sees the shapes and textures of the trees and rocks very clearly, even in the dark, because it uses a different kind of light (infrared) that cuts through the darkness better.

The challenge for scientists has always been: How do we combine the "color" from the bad night photo with the "sharp shapes" from the night vision photo without making a mess?

Previous methods tried to just "stitch" these two photos together like a puzzle, but the pieces didn't fit perfectly. The colors got muddy, and the sharp edges turned into smudges.

The New Solution: FD-INR (The "Frequency Chef")

This paper introduces a new method called FD-INR. Think of it not as a puzzle, but as a master chef who knows exactly how to separate ingredients before cooking them.

Here is how it works, using simple analogies:

1. The "Frequency" Separation (The Kitchen Counter)

Imagine the image is a soup.

Low Frequencies are the broth: The big, smooth flavors (the overall colors and brightness).
High Frequencies are the spices and crunch: The tiny details (the leaves on a tree, the texture of bark).

In a dark photo, the "broth" (colors) is actually okay, but the "spices" (details) are drowned in noise (static). In the night vision photo, the "spices" are perfect, but there is no "broth" (it's black and white).

Old methods tried to mix the two soups together immediately, resulting in a salty, grainy mess. FD-INR says: "Let's separate the broth from the spices first!"

2. The Two Specialized Chefs (Dual Branches)

The system uses two different "AI chefs" working in parallel:

Chef A (The Color Specialist): Looks only at the dark RGB photo. Their job is to fix the broth (colors and brightness). They are told to ignore the tiny details because they are too noisy. They make sure the sky is blue and the grass is green, not purple or gray.
Chef B (The Detail Specialist): Looks only at the Night Vision (NIR) photo. Their job is to fix the spices (sharp edges and textures). They are told to ignore the colors because they don't exist in that photo. They make sure the leaves look crisp and the tree bark looks rough.

3. The "Magic Blender" (Implicit Neural Representation)

Usually, computers work with a fixed grid of pixels (like a mosaic). If you zoom in, the mosaic gets blocky.
FD-INR uses something called Implicit Neural Representation (INR). Imagine instead of a mosaic, you have a smooth, infinite liquid.

You can zoom in as close as you want, and the image stays smooth and sharp.
It doesn't care if the photo is 100 pixels wide or 10,000 pixels wide. It just "knows" the shape of the image mathematically. This allows the drone to take a photo and then zoom in later without it ever getting blurry.

4. The "Smart Taste Tester" (Adaptive Weighting)

Sometimes, the Night Vision camera might be a little too bright, or the Color camera might be too dark.
FD-INR has a Smart Taste Tester (an uncertainty-based loss function). It constantly tastes the soup and asks: "Is the color too strong? Is the texture too weak?"
It automatically adjusts the recipe on the fly, telling Chef A to add more color or Chef B to sharpen the edges, ensuring the final dish is perfectly balanced.

Why is this a Big Deal?

No More "Ghost" Colors: Because the two chefs work separately, the Night Vision camera doesn't accidentally turn the red car into a gray blob. The colors stay true to the original scene.
Super Sharp Details: The texture from the Night Vision camera is perfectly pasted onto the color, so you can see individual leaves even in total darkness.
Zoom Forever: Because it uses the "infinite liquid" math (INR), you can take a small photo and blow it up to a huge size, and it will still look crisp. This is huge for drones that might need to zoom in on a specific object later.

The Bottom Line

This paper is like teaching a computer to stop trying to force two different languages to speak at once. Instead, it teaches the computer to translate the "Shape" language from one camera and the "Color" language from the other, and then perfectly blend them into a single, crystal-clear, high-definition image that looks like it was taken in broad daylight, even when it's pitch black outside.

1. Problem Statement

The paper addresses the critical challenge of low-light image restoration, specifically for Unmanned Aerial Vehicles (UAVs) operating in extreme environments (dawn, dusk, night, shadows).

Core Issues: Under low-light conditions, RGB sensors suffer from severe quantum and electronic noise. This leads to:
- Loss of high-frequency structural details (textures).
- Color bias and contrast degradation.
- A trade-off in existing methods where noise suppression often results in over-smoothing or "hallucinated" artifacts.
Limitations of Current Solutions:
- Single-modal methods: Struggle to recover details lost to noise without introducing artifacts.
- Existing RGB-NIR fusion: Most methods perform fusion in the spatial domain or use decomposition that ignores physical inconsistencies. They often fail to distinguish between reliable low-frequency color information (from RGB) and reliable high-frequency structural information (from NIR), leading to color distortion and spectral leakage.
- Discrete Representations: Traditional grid-based CNNs lack the ability to adaptively handle varying noise levels and cannot perform arbitrary-resolution reconstruction.

2. Methodology: FD-INR

The authors propose FD-INR (Frequency-Decomposed Implicit Neural Representation), a framework that treats image restoration as a continuous function mapping problem rather than a discrete grid processing task.

A. Core Concept: Frequency-Decoupled Representation

The method leverages a statistical prior: RGB signals are reliable in low frequencies (global luminance/color), while NIR signals are reliable in high frequencies (structural textures).
The reconstructed image $\hat{I}(x)$ is modeled as a linear superposition of two continuous functions:
$\hat{I}(x) = F_{LF}(x; \theta) + G_{HF}(x; \phi) - \beta$

$F_{LF}$ (Low-Frequency Branch): Captures global color consistency and luminance. It uses sparse feature grids to enforce a low-pass filter effect, naturally suppressing high-frequency noise.
$G_{HF}$ (High-Frequency Branch): Reconstructs sub-pixel structural details. It uses dense feature grids to bypass the "spectral bias" of standard INRs, injecting fine textures from the NIR signal.
$\beta$ : A learnable bias term to calibrate the DC component (luminance offset) between the two modalities.

B. Wavelet-Domain Spectral Supervision

To ensure strict adherence to the frequency decoupling, the authors employ Multi-scale Discrete Wavelet Transform (DWT) as a supervision mechanism:

Spectral Decoupling Regularization ( $L_{reg}$ ): Forces the low-frequency branch to have zero energy in high-frequency subbands and the high-frequency branch to have zero energy in low-frequency subbands. This prevents "energy leakage" (e.g., NIR luminance contaminating RGB colors).
Loss Functions:
- Low-Frequency Loss: Uses Charbonnier loss on RGB wavelet approximation subbands to ensure color fidelity.
- High-Frequency Loss: Combines $L_1$ and SSIM loss on NIR detail subbands to preserve structural integrity.
- Gradient Consistency Loss: Ensures spatial edge sharpness by aligning gradients with the NIR reference.
- Zero-Mean Centering: Constrains the high-frequency branch to output zero-mean residuals, preventing it from altering global brightness.

C. Adaptive Optimization

Homoscedastic Uncertainty Weighting: Instead of fixed weights, the model learns task-specific noise variances ( $\sigma_k$ ) to automatically balance the contributions of the low-frequency, high-frequency, and gradient losses during training.
Muon Optimizer: The authors utilize the Muon optimizer for the MLP linear layers to accelerate convergence in high-dimensional coordinate mappings, addressing the slow training speeds often associated with INRs.

3. Key Contributions

FD-INR Framework: The first implicit neural representation method to explicitly decouple RGB and NIR tasks into frequency-specific branches within a continuous coordinate space. It integrates the physical prior of "RGB-constrained low frequencies" and "NIR-enhanced high frequencies."
Adaptive Mechanism: Introduces a homoscedastic uncertainty-based loss weighting strategy and a zero-mean centering constraint to solve color distortion and irradiance mismatch issues common in cross-modal fusion.
Resolution Independence: By utilizing INR, the method achieves arbitrary-resolution reconstruction, allowing for super-resolution without retraining, a significant advantage over discrete grid-based methods.
Optimization Efficiency: The integration of the Muon optimizer significantly improves convergence speed for coordinate-based feature mapping.

4. Experimental Results

Datasets: Evaluated on the Dark Vision Dataset (DVD), a real-world RGB-NIR benchmark with synchronized pairs.
Performance:
- Quantitative: Achieved State-of-the-Art (SOTA) results with the lowest NIQE (Natural Image Quality Evaluator, indicating better perceptual quality) and highest NCC (Normalized Cross-Correlation).
- Qualitative: Visual comparisons show FD-INR produces sharper edges, purer color manifolds, and fewer artifacts compared to Naive DWT fusion and deep learning baselines (e.g., DVN).
Ablation Studies: Confirmed that removing any component (e.g., the zero-mean constraint or frequency decoupling regularization) leads to significant degradation, such as overexposure, color desaturation, or blurred textures.

5. Significance

This work represents a paradigm shift in low-light image restoration by moving from discrete spatial fusion to continuous frequency-decoupled representation.

Robustness: It effectively handles the extreme noise levels found in UAV operations where single-modal sensors fail.
Generalization: The zero-shot, instance-specific nature of the method allows it to adapt to new scenes without large-scale pre-training.
Downstream Impact: By restoring high-fidelity structural details and color accuracy, it significantly enhances the reliability of downstream computer vision tasks like object detection and 3D reconstruction in all-weather, low-light scenarios.