Technical Summary: Low-light Image Enhancement via Multi-scale Attention combined with Fourier Transform (MSFT)

1. Problem Statement

Low-light image enhancement (LLIE) aims to recover high-quality, normal-light images from inputs captured in challenging environments characterized by insufficient illumination, such as dark indoor settings or outdoor scenes with dramatic brightness variations. These conditions lead to luminance degradation, substantial image blur, and noise.

Existing deep learning-based LLIE methods face several limitations:

• Texture and Illumination Recovery: Many struggle to accurately capture real-world illumination distributions and restore fine texture details simultaneously.
• Adaptive Limitations: Mainstream methods often fail to adaptively adjust brightness based on regional texture information, leading to issues like overexposure or underexposure.
• Computational and Prior Dependencies: While diffusion models offer high realism, they often rely on assumed degradation processes or heavy pre-trained priors, limiting their applicability in real-world scenarios where degradation is ambiguous. Furthermore, methods relying on semantic priors often demand excessive computational resources.
• Noise and Background Handling: Techniques like SNR-aware optimization improve signal-to-noise ratios but often fail to capture deep-scale information or adequately enhance background regions containing auroral foreground elements.

2. Methodology

The authors propose MSFT (Multi-scale Attention combined with the Fourier Transform), a supervised, frequency-domain deep learning network. The architecture is a two-stage, U-shaped framework that integrates Convolutional Neural Networks (CNNs) and Transformers, guided by Fourier transform principles.

Core Architectural Components

A. Fourier Transform-Guided Multi-Scale Attention (FTG-MSA)
This is the core recovery module embedded within the network's scales. It leverages the observation that frequency domain amplitude encodes brightness information, while phase relates to structural details.

• Guidance Construction: The network creates a four-channel feature brightness map by concatenating the low-light image with a maximum gray value prior map (derived from the illumination layer via Retinex theory).
• Frequency Fusion: The module performs Fast Fourier Transform (FFT) on both the low-light image features and the four-channel guidance map. It adds the amplitude spectrum of the guidance map to the amplitude spectrum of the low-light image while preserving the phase of the low-light image.
• Inverse Transformation: An Inverse FFT (IFFT) reconstructs the guided feature map, which serves as a brightness prior to prevent over/under-exposure.
• Attention Mechanism: This fused amplitude information guides a multi-head self-attention mechanism. The attention weights are dynamically calculated to selectively draw relevant features, enriching image content while preserving texture.

B. Multi-Shape Synergistic Attention (MSSA)
Designed to extract deep-scale texture information and integrate high-dimensional sparse features, the MSSA module is inserted at the highest channel scale. It comprises three serially connected components:

1. Spatial and Channel Synergistic Attention (SCSA): Adaptively weighs channel importance and augments critical spatial information.
2. Multi-Shape Attention (MSA): Consists of parallel Dilated Square Attention (DSA) and Dilated Rectangle Attention (DRA).
   • DSA: Uses a hyperbolic tangent (Tanh) activation instead of Softmax to avoid low-pass filter limitations, enhancing high-frequency components.
   • DRA: Aggregates information in rectangular regions to capture multi-shape features.
3. CMUNeXt: A lightweight module that extracts global information across all channels simultaneously, reducing parameters and computational costs while integrating the spatial-channel information.

C. Network Structure
The overall framework is a three-scale U-shaped architecture.

• Encoding: Low-light images and the four-channel guidance map are processed through convolutional layers and FTGT (Fourier Transform-Guided Transformer) blocks to generate hierarchical features.
• Decoding: Multi-scale features from the encoder are fed directly into corresponding decoder channels to guide the reconstruction, eliminating the need for additional feature extraction and reducing redundancy.
• Loss Function: The model is trained using Charbonnier loss to minimize the error between the enhanced output and the ground truth.

3. Key Contributions

1. MSFT Architecture: The proposal of a two-stage integrated network combining Fourier transforms with a CNN-Transformer Cross-mixed U-net framework. This hybrid design balances the CNN's efficiency in capturing local features with the Transformer's ability to model global context.
2. FTG-MSA Module: The design of a self-attention mechanism that incorporates frequency-domain amplitude information as a dynamic guidance signal. This allows the network to adaptively adjust enhancement weights based on the maximum gray value in low-light areas, preventing exposure artifacts.
3. MSSA Module: The introduction of a Multi-Shape Synergistic Attention module in the highest-dimensional guidance space. This module effectively integrates sparse information, homogenizes channel density, and extracts deep-scale texture information using a combination of SCSA, MSA (DSA/DRA), and CMUNeXt.
4. Superior Performance: Extensive experiments demonstrate that MSFT outperforms state-of-the-art (SOTA) methods across multiple datasets (LOL, SID, SMID, SDSD) in terms of PSNR and SSIM, particularly in preserving texture details and restoring illumination without overexposure.

4. Experimental Results

The authors evaluated MSFT on seven datasets: LOL-v1, LOL-v2-real, LOL-v2-synthetic, SID, SMID, SDSD-indoor, and SDSD-outdoor.

• Quantitative Performance:
  • On the SDSD-outdoor dataset, MSFT achieved a PSNR of 41.76 dB and an SSIM of 0.988. This represents an improvement of 11.92 dB over Retinexformer and a 13.80% improvement in SSIM.
  • Compared to the SOTA supervised method Retinexformer, MSFT achieved significant PSNR gains across all benchmarks, ranging from 0.11 dB to 11.92 dB.
  • Compared to the unsupervised method Retinexformer, improvements were even more pronounced, with gains up to 16.48 dB on certain datasets.
• Efficiency:
  • MSFT has 1.25 million parameters and 18.07 GFLOPs. This is relatively low compared to other high-performing models like SNR-Net (4.01M parameters, 26.35 GFLOPs) and offers a better balance of performance and computational cost.
• Ablation Studies:
  • Removing the Retinex guidance (four-channel input) caused a significant drop in performance (e.g., ~9 dB on SDSD-outdoor), validating the importance of illumination priors.
  • Removing the MSSA module led to substantial declines in both PSNR and SSIM, confirming its role in structural similarity and texture restoration.
  • Removing the FFT component within FTGT resulted in the most severe performance degradation, proving that frequency-domain guidance is indispensable for global consistency.
• Qualitative Results: Visual comparisons show MSFT effectively captures background textures and maintains realistic illumination, whereas other methods often introduce dark spots, artifacts, or overexposure in bright regions.

5. Significance and Limitations

Significance:
The paper claims that MSFT provides a robust solution for low-light enhancement by effectively fusing frequency-domain amplitude information with multi-scale attention mechanisms. It offers a reference for constructing models that use frequency-domain guided attention to restore degraded images, achieving a balance between precise illumination recovery and detailed texture enhancement without the heavy computational burden of diffusion models or the limitations of fixed ViT structures.

Limitations:
The authors acknowledge several constraints:

• Light Pollution: The method is limited in handling images polluted by strong light, as it struggles to remove noise from overexposed parts.
• Real-time Processing: It is not sufficiently efficient for real-time processing of natural light scenarios.
• Data Dependency: The model requires a large amount of paired data for training and is not currently suitable for unsupervised augmentation fields lacking paired data.
• Noise Sensitivity: The model does not account for images acquired in scenarios with severe noise pollution; it requires relatively clean paired datasets or pre-cleansing.

Future Work:
The authors suggest future research directions include:

• Truly integrating the Fourier transform into the attention layer to perform attention calculations directly in the frequency domain (using amplitude and phase spectra).
• Exploring the application of diffusion models in the frequency domain, potentially by optimizing the diffusion structure to reduce computational resources while leveraging its generative capabilities for unsupervised enhancement.