NEC-Diff: Noise-Robust Event-RAW Complementary Diffusion for Seeing Motion in Extreme Darkness

This paper introduces NEC-Diff, a novel diffusion-based framework that synergistically combines noisy RAW images and event camera data through physics-driven constraints and adaptive feature fusion to achieve high-fidelity motion reconstruction in extreme low-light conditions, supported by the newly constructed REAL dataset.

The Gist

Imagine trying to take a photo of a bustling city street at 3:00 AM with a standard camera. The result? A grainy, blurry mess. You have two problems:

1. Too little light: The camera has to crank up its sensitivity (gain), which turns the image into static noise.
2. Too much motion: If you leave the shutter open longer to catch more light, moving cars turn into ghostly streaks.

For decades, scientists have tried to fix this, but they usually had to choose between removing the noise (making the image smooth but blurry) or keeping the details (keeping the noise).

Enter NEC-Diff, a new "super-camera" software that solves this by using two different types of eyes working together.

The Two Eyes: RAW and Events

Think of the system as having two distinct helpers:

1. The RAW Eye (The Photographer): This is a standard camera, but it captures the raw, unprocessed data before the computer tries to make it look "pretty." It sees the whole scene and knows the general brightness, but in the dark, it's full of static noise.

   • Analogy: Imagine trying to read a book in a dark room with a shaky flashlight. You can see the words (the scene), but the light flickers so much it's hard to read clearly.

2. The Event Eye (The Motion Detective): This is a special camera that doesn't take "photos." Instead, it only records changes. If a pixel doesn't change, it stays silent. If a car drives by or a leaf falls, it screams, "Something moved here!" It is incredibly fast and sensitive, but in total darkness, it gets confused and starts shouting random noise.

   • Analogy: Imagine a security guard in a pitch-black room who only yells when he hears a footstep. In a quiet room, he's perfect. But if there's a storm outside, he might start yelling about the wind, confusing you about what's actually moving.

The Problem: Both are Noisy

The paper points out a flaw in previous attempts: scientists tried to combine these two, but they mostly focused on using the Event camera to find edges, ignoring the fact that both cameras are screaming with noise in the dark. If you just mash them together, you get a noisy mess.

The Solution: NEC-Diff (The Smart Mediator)

The authors created a system called NEC-Diff that acts like a brilliant editor, using a technique called Diffusion (think of it as a "reverse noise generator" that learns to turn static into a clear picture).

Here is how it works, step-by-step:

1. The "Cross-Check" (Collaborative Noise Suppression)

Instead of letting the two cameras work alone, NEC-Diff makes them help each other clean up their own mess.

• The RAW camera tells the Event camera: "Hey, that area is actually bright, so those random 'shouts' you're making are probably just noise, not movement."
• The Event camera tells the RAW camera: "That area is dark and blurry, but I see a sharp edge here! Don't smooth that out, or you'll lose the detail."
• The Result: They act like two people trying to solve a puzzle in the dark. One has the picture of the whole box (RAW), and the other has the sharp edges (Events). By comparing notes, they can tell what is real and what is just static.

2. The "Trust Meter" (SNR-Guided Fusion)

The system doesn't just blindly mix the two. It constantly checks a Signal-to-Noise Ratio (SNR) meter for every tiny part of the image.

• Analogy: Imagine a conductor leading an orchestra. If the violin section (RAW) is playing loudly and clearly, the conductor listens to them. If the violin is quiet but the drums (Events) are hitting a perfect rhythm, the conductor focuses on the drums.
• NEC-Diff dynamically decides: "In this dark corner, the RAW image is too noisy, so I'll trust the Event camera. In this bright spot, the Event camera is confused, so I'll trust the RAW image."

3. The "Magic Paintbrush" (Diffusion)

Once the system has cleaned up the data and decided which parts to trust, it feeds this "best guess" into a Diffusion Model.

• Analogy: Think of a noisy, static-filled TV screen. A diffusion model is like a smart AI that knows what a clear TV screen should look like. It slowly peels away the static, using the clues from the RAW and Event cameras to fill in the missing details perfectly. It doesn't just guess; it reconstructs the scene based on physics and the clues it gathered.

The New Playground: The REAL Dataset

To prove this works, the team couldn't just use fake computer simulations. They built a special rig with two cameras (one RAW, one Event) mounted on a car and drove it around in extremely dark conditions (as low as 0.001 lux—that's darker than a full moon!).
They created a massive new dataset called REAL (Raw and Event Acquired in Low-light) containing 47,800 pairs of these images. This is like giving the AI a massive library of "nightmare scenarios" to practice on so it becomes an expert.

Why This Matters

Previous methods were like trying to fix a blurry photo by either blurring it more (to hide noise) or sharpening it until it looked jagged.
NEC-Diff is different because:

1. It understands the physics of how light and noise work.
2. It uses two different types of vision to cross-check each other.
3. It uses AI diffusion to reconstruct the image with high fidelity.

The Bottom Line:
NEC-Diff allows us to see clearly in the "photon-starved" darkness—places where human eyes and standard cameras go blind. Whether it's for self-driving cars at night, search-and-rescue missions, or just taking better photos at a concert, this technology turns the "grainy mess" of the dark into a crisp, high-definition reality.

Technical Summary

1. Problem Statement

Imaging dynamic scenes in extremely low-light conditions (photon-starved environments, e.g., <0.8 lux) presents a fundamental challenge for conventional cameras:

• Photon Scarcity: Limited photons lead to severe noise (shot noise, readout noise) and significant loss of texture details.
• Trade-off Dilemma: Existing Low-Light Image Enhancement (LLIE) methods struggle to balance noise suppression with texture preservation. Aggressive denoising often results in oversmoothing (loss of detail), while insufficient denoising leaves residual noise.
• Limitations of Existing Solutions:
  • sRGB-based methods: Rely on ISP pipelines that introduce nonlinear transformations, distorting noise characteristics and making separation difficult.
  • RAW-based methods: Preserve linear light response but still suffer from information loss in dynamic scenes due to short exposure times.
  • Event-based methods: Offer high dynamic range (HDR) and microsecond resolution but lack intensity information in smooth regions and are themselves noisy under low light.
  • Hybrid methods: Often fail to effectively model the intrinsic noise of both modalities simultaneously, leading to unstable fusion or poor reconstruction in ultra-dark scenarios.

2. Methodology: NEC-Diff

The authors propose NEC-Diff, a diffusion-based framework that fuses RAW images and Event streams to achieve high-fidelity reconstruction. The framework consists of three core modules:

A. Event–RAW Collaborative Noise Suppression (ECNS)

This module addresses the severe noise in both modalities by exploiting their complementary physical properties:

• Illumination-Guided Event Denoising: Since RAW images have a linear response to illumination, they provide a reliable "illumination prior." This prior guides the denoising of event streams, which are dominated by Poisson-distributed shot noise correlated with light intensity.
• Event-Assisted Image Denoising: Denoised events provide high-dynamic-range edge cues and motion information. These cues guide the RAW image denoising process, helping to distinguish signal from noise in weakly textured regions without oversmoothing fine details.
• Intensity Consistency Constraint: A physics-driven loss function ($L_{cons}$) enforces a logarithmic relationship between the denoised RAW image ($\hat{R}$) and the refined event stream ($\hat{E}$), ensuring physical consistency:
  $$\hat{E}(t) \cdot C \approx \log \frac{\hat{R}(t)}{\hat{R}(t-\Delta t)}$$

B. SNR-Guided Reliable Information Extraction (SRIE)

Recognizing that signal reliability varies spatially (e.g., events are reliable at edges but absent in smooth dark areas; RAW is reliable in brighter areas but noisy in shadows), the authors introduce an adaptive fusion mechanism:

• SNR Estimation: The module dynamically estimates the Signal-to-Noise Ratio (SNR) maps for both the denoised RAW image and the event stream.
• Adaptive Weighting: A lightweight network generates spatial weight maps ($W_{img}, W_{evt}$) via channel-wise softmax. This allows the system to selectively fuse features, prioritizing the modality with higher SNR in specific regions (e.g., relying on events for edges and RAW for smooth brightness).

C. Cross-Modal Attentive Diffusion (CAD)

The final reconstruction is performed using a diffusion model:

• Feature Fusion: The weighted features from both modalities are fused using bidirectional cross-modal attention. This allows events to inject dynamic edge details into the image features, while image features provide stable global brightness to guide event features in flat regions.
• Conditional Diffusion: The fused multi-modal features serve as conditional inputs ($F_{fused}$) for a DDIM (Denoising Diffusion Implicit Models) sampler. This guides the noise prediction network ($\epsilon_\theta$) to progressively reconstruct the high-quality image from noise, ensuring robustness in low-SNR regions.

3. Key Contributions

1. Novel Framework (NEC-Diff): A diffusion-based hybrid imaging framework that effectively disentangles sensor noise from fragile signals in extreme darkness by leveraging the linear illumination prior of RAW and the motion sensitivity of events.
2. Physics-Driven Constraints: Introduction of an Intensity Consistency Loss that mathematically links the denoised outputs of both modalities, ensuring physical plausibility and improving denoising accuracy.
3. Adaptive Fusion Strategy: Development of an SNR-guided Reliable Information Extraction module that dynamically selects the most trustworthy features from each modality, overcoming the limitations of static fusion strategies.
4. REAL Dataset: Construction of the Raw and Event Acquired in Low-light (REAL) dataset.
   • Scale: 47,800 pixel-aligned triplets (Low-light RAW, Events, High-quality sRGB GT).
   • Conditions: Illumination levels ranging from 0.001 to 0.8 lux (extreme darkness).
   • Hardware: Coaxial imaging system with controlled motion and exposure to avoid blur, capturing real-world dynamic scenes.

4. Experimental Results

The authors evaluated NEC-Diff on both synthetic (LLRVD-simu) and real-world (REAL) datasets against state-of-the-art (SOTA) methods in sRGB, RAW, Event, and Hybrid categories.

• Quantitative Performance:
  • On the REAL dataset, NEC-Diff achieved 24.51 dB PSNR, 0.742 SSIM, and 0.201 LPIPS.
  • It outperformed the next best hybrid method (ELEDNet) by a significant margin (+2.93 dB PSNR) and surpassed RAW-only methods by ~1.8 dB.
• Qualitative Performance:
  • NEC-Diff demonstrated superior texture preservation and color fidelity compared to methods that produced oversmoothed results or residual noise.
  • In dynamic scenes, it successfully recovered motion details that were blurred or lost in other methods.
• Ablation Studies:
  • Removing the ECNS module caused a 3.45 dB drop in PSNR, highlighting the importance of collaborative denoising.
  • Dual SNR-guided fusion outperformed direct fusion and image-only SNR fusion, proving the necessity of evaluating both modalities' reliability.

5. Significance

• Theoretical Advancement: The paper bridges the gap between event-based sensing and conventional imaging by establishing a rigorous physical constraint (intensity consistency) between the two modalities, moving beyond simple feature concatenation.
• Practical Impact: The proposed method enables high-quality imaging in scenarios previously considered impossible (e.g., night driving, search and rescue in total darkness) without motion blur.
• Community Resource: The release of the REAL dataset fills a critical gap in the field, providing the first large-scale, pixel-aligned benchmark for low-light event-RAW imaging, facilitating future research in extreme low-light computer vision.

In summary, NEC-Diff represents a significant leap forward in low-light imaging by combining the strengths of diffusion models, physics-based constraints, and multi-modal sensor fusion to solve the noise-texture trade-off in photon-starved environments.