AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions

Technical Summary of AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions

Problem Statement
Robust visual recognition in underwater environments remains a significant challenge due to complex distortions such as turbidity, low illumination, occlusion, and color distortion. While underwater imaging hardware has advanced, the development of reliable computer vision algorithms has been hindered by a lack of extensive, annotated datasets that reflect real-world environmental variability. Existing datasets often suffer from limitations including narrow scope (focusing solely on specific scenarios like fish identification or coral monitoring), insufficient data volume, inconsistent annotations, or a lack of diversity in imaging conditions (depth, lighting, turbidity). Furthermore, many existing resources rely heavily on synthetic data or controlled environments, failing to address the intra-class variations and inter-class similarities inherent in live underwater ecosystems.

Methodology
The paper introduces AQUA20, a comprehensive benchmark dataset designed to address these gaps. The methodology encompasses three primary components:

1. Dataset Construction (AQUA20):

   • Scale and Composition: The dataset comprises 8,171 high-quality underwater images categorized into 20 distinct marine species, including fish, coral, sharks, turtles, divers, and various invertebrates.
   • Data Sources: Images were curated primarily from the existing EUVP dataset (originally for image enhancement) and supplemented with other publicly available sources to ensure adequate representation for all 20 classes.
   • Annotation Protocol: A rigorous multi-stage annotation process was employed. Initial labeling was followed by independent reviews from three annotators to flag problematic cases (e.g., multiple species, occlusion, blur). A final consensus round resolved disagreements, ensuring high-confidence labels.
   • Challenges Incorporated: The dataset explicitly captures real-world adversities, including inter-class similarity (e.g., octopus vs. squid), high intra-class variation (e.g., different fish sizes/angles), and environmental factors like turbidity, uneven lighting, and occlusion.
   • Split: The data is divided into an 80/20 train-test split, with each class containing at least 10 test samples.

2. Benchmarking Framework:

   • Models Evaluated: Thirteen state-of-the-art deep learning models were evaluated, ranging from lightweight CNNs (SqueezeNet, MobileNetV2, ShuffleNet) to standard and deep CNNs (VGG19, ResNet50, DenseNet, EfficientNet, RegNet) and transformer-based architectures (ViT, SwinV2, MaxViT, ConvNeXt).
   • Training Protocol: All models were initialized with ImageNet-pretrained weights. A two-stage training strategy was used: first, training only the classifier layers with the backbone frozen; second, fine-tuning the entire network. No data augmentation was applied during training to strictly evaluate model robustness against the raw dataset conditions.
   • Metrics: Performance was assessed using Accuracy (Top-1, Top-2, Top-3), Precision, Recall, F1-Score, and ROC-AUC.

3. Explainability Analysis:

   • The study utilized Gradient-weighted Class Activation Mapping (Grad-CAM) and Local Interpretable Model-Agnostic Explanations (LIME) to visualize model attention. This analysis compared high-performing models (ConvNeXt) against lower-performing ones (InceptionV3) to understand how models interpret underwater features.

Key Results

• Top Performance: ConvNeXt emerged as the best-performing model, achieving a Top-1 accuracy of 90.69%, a Top-3 accuracy of 98.82%, and the highest overall F1-score of 88.92%. Its performance suggests that modern CNN architectures incorporating transformer-inspired designs (e.g., larger kernels, layer normalization) are highly effective for underwater classification.
• Transformer Performance: Transformer-based models like SwinV2B and ViT-B16 demonstrated strong capabilities (Top-3 accuracies of 98.76% and 98.45%, respectively) but were consistently outperformed by ConvNeXt.
• Lightweight Models: Models such as EfficientNetB0, MobileNetV2, and ShuffleNetV2 achieved competitive Top-3 accuracies (exceeding 95%) with significantly fewer parameters, validating their potential for resource-constrained underwater robotics.
• Baseline Limitations: VGG19, despite having the largest parameter count (139M), performed poorly (Top-1: 77.98%), highlighting the inefficiency of older, non-optimized architectures for this domain.
• Class-Specific Insights: Performance correlated with data support; classes with abundant samples (e.g., Fish, Coral) achieved high F1-scores (>0.90), while rare classes (e.g., Octopus, Marine Dolphin with ~10 test samples) showed lower recall. The "Diver" class achieved perfect scores, indicating that distinct visual features can compensate for low sample counts.
• Error Analysis: The primary source of error involved inter-class confusion between visually similar species (e.g., Coral vs. Starfish/Sea Anemone) and intra-class diversity. Confusion matrices revealed that even top models struggle with textured surfaces and branching structures common in marine life.
• Explainability Findings: Grad-CAM and LIME visualizations revealed that ConvNeXt focused on biologically relevant features (e.g., turtle flippers, dolphin midsections), whereas lower-performing models like InceptionV3 often attended to irrelevant background areas (e.g., seabed), correlating with their lower accuracy.

Significance and Claims
The authors claim that AQUA20 serves as a foundational resource for advancing underwater visual recognition by providing a diverse, real-world benchmark that addresses the scarcity of annotated data. The paper asserts that:

1. AQUA20 fills a critical gap by offering a large-scale, class-diverse dataset tailored specifically for classification under challenging environmental conditions, unlike previous datasets limited to specific tasks or synthetic data.
2. ConvNeXt establishes a strong baseline, demonstrating that modern CNNs can effectively handle underwater distortions and outperform both traditional deep CNNs and pure transformer architectures in this specific domain.
3. Explainability is vital for understanding model failures in underwater settings, as visualizing attention maps helps identify whether models rely on semantic features or spurious correlations.
4. Future Research Directions: The dataset is intended to support future work in marine biology, underwater robotics (AUVs/ROVs), and image enhancement. The authors suggest that future research should focus on expanding species coverage, developing specialized image enhancement techniques to counter environmental challenges, and exploring domain adaptation to improve robustness across diverse underwater settings.

The paper concludes that while substantial progress has been made, significant room for improvement remains in underwater species recognition, and AQUA20 provides the necessary infrastructure to drive these advancements.