💻 computer science

Dynamic Weight-based Temporal Aggregation for Low-light Video Enhancement Under Extreme Noise

The paper proposes DWTA-Net, a novel recurrent deep-learning framework for low-light video enhancement that combines Mamba-based multi-frame alignment with dynamic weight-based temporal aggregation guided by optical flow to effectively suppress extreme noise while preserving temporal consistency and fine details.

Original authors: Ruirui Lin, Guoxi Huang, Nantheera Anantrasirichai

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruirui Lin, Guoxi Huang, Nantheera Anantrasirichai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to watch a video recorded at night during a heavy storm. The picture is dark, grainy with "snow" (noise), and colors look washed out. If you try to brighten a single frame, the noise gets worse. If you try to fix the whole video at once, the image might flicker or look blurry because the camera moved.

This paper introduces a new tool called DWTA-Net to fix these messy, dark videos. Think of it as a two-step "video doctor" that uses a clever trick to clean up the noise without making the picture fuzzy.

Here is how it works, explained simply:

The Problem: The "Short Memory" Issue

Most current video cleaners are like people with short-term memory. They look at a few frames (maybe 5 or 10) and try to fix them.

The Analogy: Imagine trying to clean a dirty window by looking at it for only a split second. You might miss the dirt that is hidden behind a smudge.
The Result: These methods often leave behind noise or make the video flicker because they can't see the "big picture" of what the scene looks like over a longer time.

The Solution: A Two-Stage Process

DWTA-Net fixes this by working in two distinct stages, like a construction crew that first builds the frame and then does the fine polishing.

Stage 1: The "Group Photo" (Structure & Color)

First, the system looks at a short sequence of frames (like a group photo) and lines them up perfectly.

What it does: It uses a special AI brain (called Mamba) to understand the whole scene at once. It fixes the brightness, corrects the colors, and makes sure the shapes (like a fence or a tree) look solid.
The Analogy: This is like gathering a group of blurry photos and stacking them on top of each other to get a clear outline of the subject. It fixes the "big picture" first.

Stage 2: The "Smart Blender" (Denoising & Consistency)

This is the paper's big innovation. Instead of just averaging the frames (which makes everything blurry), it uses a recurrent method. This means it remembers what it saw in the past and blends it with the current frame, but only where it makes sense.

The Analogy: Imagine you are trying to hear a friend's voice in a noisy room.
- If your friend is standing still (static region), you can listen to them for a long time to filter out the background noise. DWTA-Net does this by "blending" many past frames together to smooth out the grain.
- If your friend starts running (dynamic region), you can't blend their past positions with their current one, or they will look like a ghost. DWTA-Net detects this movement and stops blending, keeping the moving object sharp.
How it decides: It uses "optical flow" (a way to track how pixels move) to create a dynamic weight map. It's like a smart dimmer switch that says, "Blend heavily here because it's still," and "Don't blend here because it's moving."

The Secret Sauce: The "Texture-Aware" Loss

To teach the AI how to do this, the researchers created a special scoring system (a "loss function") called Texture-Adaptive Loss.

The Analogy: Think of a painter.
- When painting a rough, textured wall (like grass or brick), the painter wants to keep every little detail and crack visible.
- When painting a smooth wall (like a clear blue sky), the painter wants it to be perfectly smooth and free of speckles.
The Result: The AI learns to be "tough" on smooth areas to remove noise, but "gentle" on textured areas to keep the details. It doesn't treat the whole image the same way.

What Did They Find?

The team tested DWTA-Net on real-world videos taken in very dark, noisy conditions (like a horse race filmed after sunset).

The Result: Compared to other top methods, DWTA-Net removed more noise, kept the colors natural, and didn't make moving objects look blurry or ghostly.
The Proof: When they looked at a static part of the video (the sky), DWTA-Net produced a cleaner image than even the best single-image cleaners, proving that looking at the "long-term history" of the video really helps.

In short: DWTA-Net is a video enhancer that acts like a smart editor. It fixes the colors first, then uses a "memory" to smooth out noise in quiet parts of the video while keeping moving parts sharp, all guided by a rule that says "keep the details where they matter."

Technical Summary: Dynamic Weight-Based Temporal Aggregation for Low-Light Video Enhancement Under Extreme Noise

Problem Statement
Low-light video enhancement (LLVE) faces significant challenges due to severe noise, low contrast, and color degradation, particularly in outdoor "in-the-wild" scenarios. While learning-based methods offer fast inference, they often struggle under heavy real-world noise because they fail to sufficiently exploit long-term temporal cues. Existing approaches typically rely on sliding-window processing or 3D convolutions that consider only a short neighborhood of frames. These methods are computationally heavy, memory-intensive, and unable to fully leverage long-term temporal redundancy, which is crucial for denoising under extreme conditions. Furthermore, models trained on synthetic or mild noise often generalize poorly to complex, spatially varying real-world noise, leading to temporal inconsistencies such as flickering and ghosting.

Methodology: DWTA-Net
The authors propose DWTA-Net, a novel deep-learning recurrent framework based on a two-stage architecture designed to handle heavy real-world noise while maintaining temporal consistency.

Stage I: Multi-Frame Enhancement (Local Restoration)
- Goal: Restore local structure, color, and brightness while ensuring short-term temporal consistency.
- Mechanism: A short sequence of neighboring frames is first aligned to a reference frame using a PCD (Pixel-wise Convolutional Deformable) module to generate motion-compensated features.
- Backbone: These aligned features are processed by a Mamba-based U-Net-like backbone. Conventional convolutional blocks are replaced with Visual State-Space (VSS) blocks. Unlike standard convolutions with limited receptive fields, VSS blocks utilize a selective scan mechanism to capture long-range global dependencies, which is critical for restoring structural coherence in heavily degraded scenes.
Stage II: Dynamic Recurrent Refinement (Long-Term Aggregation)
- Goal: Suppress noise and stabilize details over long sequences by accumulating long-term temporal evidence.
- Mechanism: Departing from hidden-state recurrence, DWTA-Net performs aggregation directly in the spatial (pixel) space, where low-light noise manifests as high-frequency artifacts.
- Dynamic Weighting: The stage employs a recurrent formulation guided by optical flow (using GMFlow). At each timestep $t$ , the refined output from the previous step ( $O'_{t-1}$ ) is warped to align with the current Stage I output ( $O_t$ ).
- Adaptive Blending: A dynamic weight map $\omega$ $ω$ is computed based on the motion residual $R = |O_t - O^W_{t-1}|$ $R = ∣ O_{t} - O_{t - 1}^{W} ∣$ .
  - In static regions (low residual), the weight emphasizes accumulation from the warped previous frame to suppress noise.
  - In dynamic regions (high residual), the weight favors the current frame to preserve motion sharpness.
- This strategy balances noise suppression with motion sharpness, overcoming the limitations of fixed-window averaging.
Texture-Adaptive Loss
- To address the spatial variance of low-light noise, the authors introduce a loss function that balances fine detail preservation with spatial smoothness.
- A texture-aware map ( $M_T$ ) is derived from high-frequency components of the output using a 2D Discrete Wavelet Transform (2D-DWT).
- The total loss combines a pixel-wise $\ell_2$ $ℓ_{2}$ reconstruction loss with a weighted sum of:
  - Perceptual Loss ( $L_{VGG}$ ): Applied to highly textured regions ( $M_T \to 1$ ) to preserve fine details.
  - Total Variation Loss ( $L_{TV}$ ): Applied to homogeneous regions ( $M_T \to 0$ ) to encourage smoothness and suppress noise.

Key Contributions

DWTA-Net Framework: A novel Mamba-based recurrent framework tailored for heavy real-world noise, featuring a two-stage design that separates local restoration from long-term temporal aggregation.
Spatial Recurrent Aggregation: A dynamic weight-based temporal aggregation strategy that operates in spatial space rather than feature space. It utilizes motion residuals to adaptively adjust blending weights, balancing temporal consistency with motion-aware sharpness.
Texture-Adaptive Loss: A custom loss function that dynamically weights perceptual and smoothness constraints based on local texture complexity, ensuring detail preservation in textured areas while smoothing homogeneous regions.

Experimental Results
The model was trained on the DID paired low-light video dataset and evaluated on both quantitative metrics (PSNR, SSIM, LPIPS) and qualitative "in-the-wild" scenarios (specifically the Horse sequence).

Quantitative Performance: On the DID dataset, DWTA-Net achieved the highest PSNR (24.27 dB) and LPIPS (0.115) among compared methods, outperforming state-of-the-art image-based methods (e.g., Retinexformer, WaveMamba) and video-based baselines (e.g., EDVR, SMOID).
Denoising Capability: In a supplementary evaluation on a static sky region of the Horse footage (using a pseudo-ground truth generated by frame averaging), DWTA-Net achieved the highest PSNR (27.81 dB) and SSIM (0.839), surpassing even ideal static-scenario image-based methods.
Visual Quality: Qualitative comparisons show that DWTA-Net preserves fine details (e.g., fences, cables, grass texture) and natural colors without the over-smoothing artifacts seen in methods like Starlight-denoiser or the distracting artifacts introduced by SDSD-net.
Ablation Studies: Removing Stage I led to drastic drops in PSNR and perceptual quality, confirming its role in basic restoration. Removing Stage II resulted in higher LPIPS, demonstrating the necessity of long-term temporal aggregation for visual quality. The texture-adaptive loss was shown to be critical for balancing noise removal and texture preservation.

Significance and Claims
The paper claims that DWTA-Net provides a robust solution for low-light video enhancement under extreme noise conditions where traditional pipelines and existing learning-based methods fail. By leveraging long-term temporal redundancy through a recurrent, motion-aware aggregation strategy, the framework achieves superior noise suppression and temporal consistency. The authors position their work as a practical advancement that addresses the specific challenge of generalizing to complex, real-world noise distributions while preserving fine structural details, validated through extensive benchmarks and challenging "in-the-wild" evaluations.