💻 computer science

DeLux: Cross-Modal Local Artifact Restoration in Video Using Neuromorphic Data

The paper introduces DeLux, a novel cross-modal pipeline that leverages neuromorphic event streams as structural priors to effectively detect and inpaint local lighting artifacts in RGB video, significantly outperforming existing RGB-only and event-guided baselines in restoring obscured details.

Original authors: Bartosz Stachowiak, Dariusz Brzezinski

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bartosz Stachowiak, Dariusz Brzezinski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a photo of a beautiful sunset, but the sun is so bright it blinds your camera. The result is a photo with a giant, white, washed-out blob where the trees should be, or maybe a strange rainbow flare across the screen. This is what happens to standard cameras (RGB cameras) when the light gets too intense or tricky. They "saturate," meaning they lose all the details in those bright spots, and once that information is gone, it's usually impossible to get it back just by looking at the photo again.

This paper introduces a new system called DeLux (which stands for Deluxe Lux restoration) that solves this problem by using a "second pair of eyes" that never gets blinded.

The Problem: The Blinded Camera

Think of a standard camera like a human eye. If you stare directly at a blinding light, your vision goes white, and you can't see anything behind it. In the digital world, this creates "artifacts" like glare, lens flares, flickering lights, or overexposure. Existing software tries to fix these photos by guessing what might be there, but it often fails because the original data is completely gone.

The Solution: The "Super-Seeing" Eye

The authors realized that while the standard camera gets blinded, a special type of camera called a neuromorphic event camera does not.

The Analogy: Imagine the standard camera is a person taking a photo with a slow shutter speed. If a flash goes off, the photo is ruined. The neuromorphic camera, however, is like a person with super-fast reflexes who only notices changes in light. It doesn't take a "picture" in the traditional sense; it records millions of tiny "blips" of light changes per second. Even when the sun is blinding the standard camera, this event camera can still "see" the details of the trees and buildings because it's sensitive to the tiny shifts in brightness that the standard camera misses.

How DeLux Works: The Four-Step Kitchen

The authors built a modular pipeline (a step-by-step recipe) called DeLux that combines these two cameras to fix the bad photos. Here is how it works, using a kitchen analogy:

The Backup Chef (Event-to-Video):
First, the system takes the raw "blips" from the event camera and turns them into a smooth, black-and-white video. Think of this as the Backup Chef who can see clearly in the dark and bright light. This video serves as a "map" of what the scene actually looks like, free from the blinding glare.
The Spotter (Artifact Detection):
Next, a smart AI looks at the original, ruined photo and asks, "Where exactly is the problem?" It draws a digital mask (like a stencil) over the bright, washed-out spots. It doesn't try to fix the whole picture yet; it just identifies the damaged areas.
The Fusion Station (Multimodal Fusion):
This is the magic step. The system takes the ruined photo, the clear "map" from the Backup Chef, and the stencil from the Spotter, and blends them together. It's like a chef who knows exactly which part of the soup is burnt. They only scrape off the burnt part and replace it with fresh ingredients from the Backup Chef's pot, leaving the rest of the soup untouched.
The Fixer (Artifact Removal):
Finally, the system uses that blended information to "paint over" the bad spots in the original photo. Because it has the clear details from the event camera, it can restore the trees, the road, or the building that were hidden behind the glare.

Why This is a Big Deal

The paper claims that previous methods tried to fix these problems in isolation:

Some tried to fix glare but failed with lens flares.
Some tried to fix overexposure but just made the whole image darker (like putting on sunglasses) rather than restoring the hidden details.
Some tried to use the event camera to make the whole image brighter, but that often kept the ugly glare rays intact.

DeLux is different because it is the first system to specifically target local damage. It doesn't just brighten the whole image; it surgically removes the bad light and replaces only the missing details using the event camera's data.

The Results

The authors tested DeLux on:

Fake data: They created computer-generated photos with fake glare and flares to test if the system could fix them.
Real data: They used footage from real cars driving in sunny conditions.

The results showed that DeLux was much better at restoring the hidden details than any other method. It achieved a score of over 0.99 (on a scale where 1.0 is perfect) for how well the restored image matched the original scene. In real-world car footage, it reduced the severity of the glare by up to 88%.

Limitations (The Fine Print)

The authors are honest about what DeLux can't do yet:

Static Scenes: Event cameras rely on movement or changes in light to work. If a scene is completely still and the light doesn't change, the event camera might go silent, and DeLux can't help.
Synthetic Training: They had to teach the system using fake, computer-generated glare because it's hard to find real photos where you have both the "ruined" version and the "perfect" version of the same scene.
Flicker: Because the system fixes each frame individually, sometimes the restored video might flicker slightly between frames, though the authors note this is a direction for future work.

In short, DeLux is a clever way of using a "super-sensitive" camera to help a "normal" camera fix its mistakes, allowing us to see clearly even when the sun is trying to blind us.

Technical Summary: DeLux – Cross-Modal Local Artifact Restoration in Video Using Neuromorphic Data

1. Problem Statement

Conventional RGB cameras are susceptible to severe lighting artifacts such as lens flare, glare, overexposure, and flicker, particularly in outdoor scenes with extreme dynamic ranges. These artifacts often cause localized information loss where pixel intensities are fully saturated or clipped, rendering recovery via standard RGB-only post-processing impossible. Existing restoration methods typically address these issues in isolation (e.g., only glare or only overexposure) or rely on global tone mapping. The latter approach preserves the structural integrity of optical artifacts (like flare rays) while compressing dynamic range, failing to recover details completely obscured by the degradation. Furthermore, current literature lacks a generalized paradigm capable of simultaneously addressing diverse, spatially discrete lighting degradations.

2. Methodology: The DeLux Pipeline

The authors propose DeLux, a modular, end-to-end proof-of-concept pipeline that leverages neuromorphic event streams as a structural prior to guide the targeted detection and inpainting of lighting artifacts in RGB video. The framework explicitly decouples artifact detection from multimodal fusion and image inpainting. The pipeline consists of four distinct stages:

Event-to-Video Reconstruction: Asynchronous event data from a neuromorphic camera is converted into dense grayscale video frames using pre-trained models (E2VID or E2VID++). These reconstructions serve as a secondary modality, capturing brightness changes with microsecond precision and a dynamic range exceeding 140 dB, even when RGB sensors are saturated.
Artifact Detection: A dedicated UNet-based segmentation network processes the input RGB frame to generate a pixel-wise confidence map (artifact mask) indicating the likelihood of artifact presence (glare, flare, overexposure, flicker). This step is trained on synthetically generated artifact masks to generalize across conditions.
Multimodal Fusion: The RGB frame, the event-based reconstruction, and the artifact mask are concatenated along the channel dimension. A convolutional module blends these inputs to create a unified representation. Crucially, the artifact mask acts as a manual-override interface, allowing the fusion behavior to be configured at inference without retraining. This stage ensures that the structural details from the event stream are utilized primarily in corrupted regions.
Artifact Removal: A final UNet-based module performs targeted inpainting. It receives the fused representation and outputs a clean, artifact-free RGB image. The model is trained to restore corrupted regions while preserving unaffected content.

Training Strategy:
The pipeline is trained via joint optimization of the detector, fusion, and remover modules, while the event-to-video reconstructor remains frozen. The training objective combines three loss components:

Detection Loss ( $L_{det}$ ): Huber loss between predicted and ground-truth artifact masks.
Reconstruction Loss ( $L_{recon}$ ): A combination of Mean Absolute Error (MAE), Multi-Scale Structural Similarity (MS-SSIM), VGG perceptual loss, and Total Variation (TV) regularization.
Non-Mask Loss ( $L_{nm}$ ): A consistency term that penalizes edits in non-artifact regions, ensuring the model does not alter clean pixels.

Data:
The system is trained on a hybrid dataset comprising synthetic artifacts injected into clean frames (using CARLA and Cityscapes) and real-world neuromorphic data (DSEC, E2VID). For real-world evaluation, the authors curated specific temporal windows from automotive footage where severe artifacts occur naturally.

3. Key Contributions

Novel Research Direction: The paper introduces "cross-modal local artifact restoration," a paradigm that treats localized lighting artifacts as modality-specific information loss mitigated by integrating a secondary sensing modality (neuromorphic events) to guide RGB inpainting.
DeLux Framework: The first system designed to detect and remove multiple types of lighting artifacts within a single deep learning pipeline by fusing standard video with neuromorphic event data.
Rigorous Benchmark: The authors establish a benchmark comprising both synthetic and real-world lighting artifacts, providing an evaluation protocol for future cross-modal restoration models. They release synthetic artifact generation tools and curated real-world datasets to foster further research.

4. Experimental Results

The authors evaluated DeLux against state-of-the-art baselines, including single-modality restoration models (Flare7K++, Wu et al., SHDR), a unified single-network approach (DAD†), and event-guided HDR models (HDRev-Diff).

Synthetic Data Performance: DeLux achieved the highest performance across all artifact types (glare, flares, overexposure) in terms of MS-SSIM, PSNR, and MAPE. It achieved an average MS-SSIM of over 0.99 across all artifact types, significantly outperforming RGB-only baselines and event-guided HDR models.
Real-World Performance: On real-world automotive footage, DeLux demonstrated up to an 88% reduction in artifact severity (measured by the custom Strong Artifact Suppression metric, SAS). In contrast, global tone-mapping baselines like HDRev-Diff often failed to suppress localized artifacts, resulting in negative SAS scores for certain sequences.
Efficiency: DeLux operates at 16.5 FPS with 90.2 million parameters on an NVIDIA A100, making it an order of magnitude faster and lighter than diffusion-based baselines (e.g., HDRev-Diff at 1.1 FPS with 1.6B parameters).
Ablation Studies: The study confirmed that removing the event modality (DeLux-NE) or the explicit detector (DeLux-ND) severely degrades performance. The non-mask loss ( $L_{nm}$ ) was found to be critical for balancing detection accuracy with controlled, localized reconstruction.

5. Significance and Claims

The paper claims that DeLux represents a significant departure from global enhancement approaches. By explicitly decoupling artifact localization from multimodal fusion, the framework prevents the unwarranted alteration of clean image regions, allowing the network to selectively use the secondary modality only where the RGB sensor has failed.

The authors position DeLux as a template for addressing severe localized image degradations using secondary sensing modalities. They argue that this cross-modal approach moves beyond single-sensor constraints, offering a path toward highly reliable image processing systems capable of operating under adverse lighting conditions.

Limitations Acknowledged by the Authors:

Static Scenes: Event cameras inherently struggle in stationary scenes where no events are triggered, though the modular architecture allows for the integration of other modalities (e.g., thermal) to mitigate this.
Synthetic Evaluation: While high-quality synthetic overlays were used, they cannot fully reproduce the complex physical interplay of light in real lens assemblies.
Temporal Consistency: As the current implementation restores frames independently, outputs are not guaranteed to be temporally consistent, potentially resulting in visible flicker. The authors identify multi-frame modeling as a necessary direction for future work.
Optimization: The reliance on a frozen E2VID reconstructor limits full end-to-end optimization, suggesting future pipelines could ingest raw event representations directly.