A Comprehensive Survey on Underwater Image Enhancement Based on Deep Learning

This paper provides a comprehensive survey on deep learning-based underwater image enhancement by systematically reviewing physical models, data construction, and evaluation metrics; categorizing recent algorithms across six key dimensions; conducting unbiased quantitative and qualitative comparisons of state-of-the-art methods; and outlining future research directions.

The Gist

Imagine you are trying to take a beautiful photo of a coral reef, but you are doing it through a thick, murky window that is covered in blue-green fog. The colors are washed out, the details are blurry, and the light is dim. This is exactly what happens when we take pictures underwater.

This paper is a comprehensive "user manual" and "state of the union" report for a field called Underwater Image Enhancement (UIE). The authors are essentially saying, "We have a lot of different tools to fix these blurry, blue photos, but nobody has ever organized them all into one clear guide. So, we did it."

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Blue Fog"

When light travels underwater, it gets eaten up by the water and the dirt in it.

• The Red Light Dies First: Imagine a flashlight beam going underwater. The red, orange, and yellow parts of the light get absorbed quickly, leaving only the blue and green. That's why everything looks like it's wearing a blue filter.
• The Fog: Particles in the water scatter the light, making the image look hazy, like looking through a dirty shower curtain.
• The Darkness: Deep down, there is no sun, so it's pitch black without a flashlight.

2. The Old Way vs. The New Way

• The Old Way (Non-Deep Learning): In the past, scientists tried to fix these photos using math formulas and physics rules. It was like trying to fix a broken watch with a hammer and a ruler. Sometimes it worked, but often the "rules" didn't fit the messy reality of the ocean.
• The New Way (Deep Learning): Now, researchers use AI (Artificial Intelligence). Think of this as training a super-smart apprentice. You show the AI thousands of bad photos and the "correct" versions, and it learns to guess how to fix them on its own. This paper focuses entirely on these AI methods.

3. The "Six Ways" to Fix the Photo

The authors organized all the different AI methods into six categories, like sorting tools in a toolbox:

• Network Architecture (The Blueprint): How is the AI built? Some are built like a standard factory line (Convolution), some use "spotlights" to focus on specific details (Attention), and some use a new type of brain called a "Transformer" (like the ones that power chatbots).
• Learning Strategy (The Teacher): How does the AI learn?
  • Adversarial Learning: Imagine a forger trying to make fake money and a police officer trying to catch them. The AI tries to make the photo look real, and a "critic" AI tries to spot the fake. They fight until the photo is perfect.
  • Rank Learning: Instead of asking "Is this perfect?", the AI asks, "Is this photo better than that one?" It learns to rank them.
• Learning Stage (The Process): Does the AI fix the photo in one big jump (Single Stage), or does it do a rough draft first and then polish it (Coarse-to-Fine)? Some even use a "Diffusion" process, which is like slowly turning a blurry sketch into a sharp painting by removing noise step-by-step.
• Helping Hands (Assistance Tasks): Sometimes the AI gets help from other tasks. For example, if the AI also tries to guess how deep the water is (Depth) or what object is in the picture (Semantic Segmentation), it gets better at fixing the colors.
• Domain Perspective (The Translator): Real underwater photos are hard to get. So, we often train AI on computer-generated photos. But computer photos look different from real ones. This section is about teaching the AI to translate "computer style" to "real ocean style" so it doesn't get confused.
• Disentanglement & Fusion (The Separation): The AI tries to separate the "bad stuff" (the blue fog, the darkness) from the "good stuff" (the actual fish and coral). It strips away the fog and then fuses the clean image back together.

4. The Great Experiment (The Race)

The authors noticed that every researcher tested their AI on different computers with different settings, making it impossible to know who was actually the best.

• The Solution: They took the top 17 AI algorithms and ran them all on the same test track, with the same rules.
• The Result: It was a tight race!
  • For photos where they knew the "perfect" answer (like a test with a key), the UIE-DM (a diffusion-based model) won.
  • For real-world photos where there is no "perfect" answer, UGAN (a forger/cop model) did the best job at making things look natural.

5. What's Next? (The Future)

The paper ends by saying, "We are getting good, but we aren't there yet." Here are the problems they need to solve next:

• Better Training Data: We need better ways to create fake underwater photos that look exactly like real ones, maybe using video game engines (like Unreal Engine) to simulate the ocean perfectly.
• Does it Help Robots? We fix photos so robots can see better. But surprisingly, some studies show that making the photo too pretty might actually confuse the robot's object detector. We need to figure this out.
• Chat with the AI: What if we could tell the AI, "Make the fish look redder" or "Brighten the left side"? Using text to guide the image fix is the next big thing.
• Uneven Light: Most AI assumes the light is even. But underwater, a flashlight creates a bright spot and a dark shadow. The AI needs to learn to handle that uneven lighting.

Summary

This paper is a map for anyone wanting to navigate the world of underwater photo fixing. It tells us what tools exist, how they work, which ones are currently the fastest, and where the road is still under construction. It's a celebration of how far we've come, but a reminder that the ocean is still a very tricky place to see clearly.

Technical Summary

1. Problem Statement

Underwater imaging is critical for resource exploration, surveillance, and entertainment, but images captured underwater suffer from severe quality degradation due to the physical properties of water. The primary issues include:

• Color Distortion: Differential attenuation of light wavelengths causes images to appear blue-green.
• Contrast Reduction & Haze: Light scattering by suspended particles and muddy water creates a hazy, blurry effect.
• Low Illumination: Light absorption at depth leads to dark, low-light conditions.
• Lack of Ground Truth: Obtaining paired distorted-reference images in real-world scenarios is nearly impossible, making supervised learning difficult.

While numerous Deep Learning-based Underwater Image Enhancement (DL-UIE) algorithms exist, there has been no systematic review categorizing them by contribution, nor a fair, unified comparison of their performance across different benchmarks.

2. Methodology

The authors conducted a comprehensive survey and a rigorous experimental evaluation.

A. Systematic Taxonomy

The paper categorizes existing DL-UIE algorithms into six primary dimensions based on their main contributions:

1. Network Architecture:
   • Convolution: Standard convolutions, residual/dense connections (e.g., UWCNN, UWNet).
   • Attention Mechanisms: Spatial and channel attention to focus on specific features (e.g., WaveNet, ADMNNet).
   • Transformers: Leveraging long-range dependencies (e.g., PTT, U-Trans).
   • Frequency Domain: Utilizing Fourier transforms and Wavelet decomposition to separate frequency sub-bands (e.g., UHD, UIE-WD).
   • NAS: Automated network structure search (e.g., AutoEnhancer).
2. Learning Strategy:
   • Adversarial Learning: Using GANs to improve perceptual quality (e.g., FUnIEGAN).
   • Rank Learning: Optimizing relative quality rankings rather than absolute pixel values (e.g., URanker).
   • Contrastive Learning: Learning from positive/negative pairs to handle missing labels (e.g., TACL).
   • Reinforcement Learning: Using reward functions based on no-reference metrics (e.g., HPUIE-RL).
3. Learning Stage:
   • Single Stage: Direct mapping from distortion to enhancement.
   • Coarse-to-Fine: Multi-stage refinement (e.g., MBANet).
   • Diffusion Models: Using generative diffusion processes (e.g., UIE-DM).
4. Assistance Tasks: Integrating semantic segmentation, depth estimation, or object detection to provide auxiliary priors (e.g., Joint-ID, SGUIE).
5. Domain Perspective: Addressing the domain gap between synthetic and real data via knowledge transfer, domain translation, or diversified outputs.
6. Disentanglement & Fusion: Separating physical parameters (attenuation, background light) or fusing different color spaces (RGB, HSV, LAB) and water types.

B. Experimental Evaluation

To address the lack of fair comparisons in existing literature, the authors performed a unified quantitative and qualitative evaluation:

• Algorithms: 17 State-of-the-Art (SOTA) DL-UIE algorithms were tested.
• Datasets:
  • Full-Reference (Paired): UIEB, LSUI, EUVP, UFO-120.
  • No-Reference (Unpaired): Challenge-60, U45, UCCS, EUVP-330.
• Metrics:
  • Full-Reference: PSNR, SSIM.
  • No-Reference: UIQM, UCIQE, and the data-driven URANKER.
• Settings: All models were re-trained or tested under unified conditions (batch size 8, 256x256 resolution, specific augmentation, no external pre-trained feature extractors) to ensure fairness.

3. Key Contributions

1. Comprehensive Taxonomy: The first survey to categorize DL-UIE algorithms across six distinct technical perspectives (Architecture, Strategy, Stage, Assistance, Domain, and Disentanglement), providing a clear roadmap for researchers.
2. Unified Benchmarking: A rigorous, fair comparison of 17 SOTA methods across multiple datasets and metrics, eliminating inconsistencies caused by varying experimental setups in original papers.
3. Resource Compilation: Creation of a centralized GitHub repository (https://github.com/YuZhao1999/UIE) containing code, datasets, and models.
4. Identification of Future Directions: Highlighting critical gaps such as the need for game-engine-based data synthesis, the impact of UIE on downstream tasks, and the integration of vision-language models.

4. Results

• Full-Reference Performance: UIE-DM (Diffusion Model) achieved the best overall performance in terms of PSNR and SSIM on paired datasets, indicating superior fitting ability. However, the top 5 algorithms showed minimal quantitative differences, suggesting a saturation in pixel-level accuracy.
• No-Reference/Generalization Performance:
  • UGAN achieved the best overall performance on no-reference metrics (UIQM, UCIQE), demonstrating superior generalization to real-world distorted images.
  • UIE-DM and CECF also performed highly on the URANKER metric.
• Visual Analysis: While many algorithms achieved similar pixel statistics, visual results varied significantly. Some methods still suffered from residual blue-green casts or artifacts, highlighting that high PSNR does not always equate to perceptually superior images.

5. Significance and Future Outlook

This survey serves as a foundational reference for the underwater vision community. It clarifies that while pixel-level restoration is maturing, perceptual quality and generalization remain challenging.

The authors propose several critical directions for future research:

• High-Quality Data Synthesis: Using game engines (e.g., Unreal) to generate realistic, controllable paired underwater data.
• Downstream Task Impact: Investigating the surprising finding that some enhancement methods may degrade object detection performance, necessitating a balance between visual quality and machine vision utility.
• Multimodal Integration: Leveraging large-scale vision-language models (e.g., CLIP) to incorporate global semantic understanding into enhancement.
• Non-Uniform Illumination: Developing methods specifically for artificial lighting scenarios common in deep-sea exploration.
• Reliable Metrics: Developing no-reference metrics that better correlate with human subjective aesthetics.

In conclusion, the paper asserts that UIE is still an emerging field with significant room for improvement, particularly in bridging the gap between algorithmic optimization and real-world application requirements.