M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement

Technical Summary: M2RETINEXFORMER

Problem Statement
Low-light image enhancement is a challenging task due to complex degradations such as amplified noise, artifacts, and color distortion. While Retinex-based deep learning methods, particularly Retinexformer, have achieved strong results by decomposing images into reflectance and illumination components, they rely exclusively on single-modality RGB information. This limitation restricts the network's ability to reason about scene geometry and the spatial distribution of light, leading to difficulties in distinguishing between dark regions caused by shadows, occlusion, or distance.

Methodology
The authors propose M2Retinexformer (Multi-Modal Retinexformer), a framework that extends the one-stage Retinex-based framework (ORF) of Retinexformer by integrating complementary multi-modal cues. The architecture consists of an Illumination Estimator and a Multi-Modal Corruption Restorer, enhanced by two primary components:

1. Modality Extractor: This module extracts and aligns auxiliary features at multiple scales to inject them into the restoration pipeline. The paper utilizes three specific modalities:

   • Depth: Extracted using a frozen Depth-Anything-V2 model to provide illumination-invariant geometric structure, helping to disambiguate dark regions.
   • Luminance: Augmented luminance features derived from NTSC luminance calculations, enriched with Sobel edges, local contrast, and multi-scale pyramid cues.
   • Semantic Features: Extracted using a frozen DINOv3 backbone to capture object-aware representations, preserving color consistency and structural integrity.

2. Multi-Modal Cross-Attention Block (MMCAB): This is the core fusion module. It operates within a progressive refinement pipeline (cascading three identical stages) and employs:

   • Cross-Attention: Allows RGB features to selectively query complementary information from the auxiliary modalities (Depth, Luminance, Semantic).
   • Illumination-Guided Self-Attention: Retained from the original Retinexformer, where attention is modulated by illumination priors.
   • Adaptive Gating: A learnable mechanism that dynamically balances the self-attention output and the multi-modal cross-attention output based on the reliability of the auxiliary cues. This prevents noisy or redundant modalities from degrading the main RGB branch.

The framework is trained using a combination of L1 loss and a perceptual loss (VGG-19 based) to preserve high-level semantic structures and fine textures.

Key Contributions

• Novel Architecture: The introduction of M2Retinexformer, which extends Retinexformer by incorporating depth, luminance, and semantic features via a Multi-Modal Cross-Attention Block and an adaptive gating mechanism.
• Modular Design: A flexible architecture where modalities adhere to a unified interface, allowing for the integration of additional priors without modifying the core network.
• Systematic Analysis: Extensive ablation studies demonstrating the individual and combined effects of each auxiliary modality on performance.
• Efficiency: The model utilizes frozen encoders for auxiliary modalities, resulting in 2M trainable parameters (48M total), which is significantly more efficient than prior multi-modal approaches like ModalFormer (198M parameters).

Results
Evaluations were conducted on seven benchmarks: LOL-v1, LOL-v2 (Real and Synthetic), SID, SMID, and SDSD (Indoor and Outdoor).

• Quantitative Performance: M2Retinexformer achieved the best or second-best performance on the majority of datasets. Notably, it achieved a PSNR of 24.89 on LOL-v1, 23.85 on LOL-v2 Real, and 30.48 on SDSD Indoor, outperforming the baseline Retinexformer and other state-of-the-art methods.
• Qualitative Performance: Visual comparisons indicate that the proposed method produces well-exposed images with natural colors and reduced noise, effectively suppressing the color distortion and residual noise observed in the baseline Retinexformer.
• Ablation Findings: On the LOL-v2 Real dataset, the addition of depth provided the most significant performance gain (+2.00 dB over baseline), followed by luminance. The study noted that adding all modalities simultaneously does not always yield consistent improvements, highlighting the importance of effective modality selection and the role of adaptive gating.

Significance and Claims
The paper claims that M2Retinexformer demonstrates the effectiveness of integrating heterogeneous modalities within a Retinex-based framework. The key insight is that depth provides geometric context robust to lighting variations, while luminance and semantic features offer content-aware guidance. The authors assert that their approach achieves overall improvements over existing methods while maintaining a modular and extensible design.

However, the paper modestly acknowledges a limitation: the benefits of multi-modal fusion depend heavily on modality reliability. In scenarios where auxiliary features are unstable (such as video-based captures in SMID and SDSD datasets with varying exposure characteristics), the performance gains may diminish. The framework is presented as a promising direction for future advances, offering a foundation for accommodating additional priors in low-light enhancement tasks.