SMART-Ship: A Comprehensive Synchronized Multi-modal Aligned Remote Sensing Targets Dataset and Benchmark for Berthed Ships Analysis

1. Problem Statement

Remote sensing (RS) for maritime surveillance faces significant challenges due to the dynamic nature of marine environments, the diversity of ship scales and categories, and the limitations of single-sensor imaging (e.g., cloud cover affecting optical sensors, lack of texture in SAR). While multi-modal data fusion offers a solution, existing datasets suffer from critical gaps:

• Task Fragmentation: Most datasets are designed for specific tasks (e.g., only detection or only generation) rather than a unified benchmark.
• Annotation Granularity: Existing datasets often rely on coarse bounding boxes (HBB/OBB) rather than precise polygonal annotations, lacking instance-level identifiers and fine-grained categories.
• Modality & Temporal Misalignment: Many datasets lack strict spatiotemporal registration across diverse modalities (RGB, SAR, PAN, MS, NIR) within short time windows, which is essential for capturing dynamic scene changes.
• Domain Bias: Most datasets focus on urban/land scenarios, lacking the complex spectral characteristics of water bodies and port infrastructure.

2. Methodology: The SMART-Ship Dataset

The authors propose SMART-Ship, a comprehensive dataset designed to bridge these gaps.

• Data Composition:
  • Scale: 1,092 registered multi-modal image sets covering 38,838 ship instances.
  • Modalities: Five complementary modalities: Visible-light (RGB), Synthetic Aperture Radar (SAR), Panchromatic (PAN), Multi-spectral (MS), and Near-infrared (NIR).
  • Synchronization: Images are acquired within a one-week window (average time difference < 0.01 days, std dev 2.86 days) to ensure spatiotemporal consistency.
  • Resolution: High-resolution modalities (PAN, RGB, SAR) have ~0.8m GSD; low-resolution modalities (MS, NIR) have ~3.0m GSD.
• Annotation Strategy:
  • Polygonal Annotations: Unlike standard bounding boxes, ships are annotated with precise polygons (average 12–13 vertices per ship) to capture complex hull shapes.
  • Hierarchical Labels: Includes 61 fine-grained ship categories and unique instance-level identifiers (566 unique IDs) for re-identification.
  • Change Masks: Includes change region masks for temporal analysis (e.g., ship arrival/departure).
• Benchmark Tasks: The dataset supports five fundamental downstream tasks:
  1. Multi-modal Ship Detection
  2. Cross-modal Ship Re-identification (Re-ID)
  3. Cross-modal Image Generation
  4. Pan-sharpening
  5. Cross-modal Change Detection

3. Key Contributions

1. First Unified Multi-modal Maritime Dataset: SMART-Ship is the first dataset to provide synchronized, fine-grained annotations across five modalities specifically for maritime scenarios, addressing the lack of port-specific resources.
2. Rich Annotation Granularity: It introduces polygon-level annotations and instance-level IDs, enabling tasks like precise shape analysis and cross-modal re-identification, which are impossible with coarse bounding boxes.
3. Comprehensive Benchmarking: The authors established standardized benchmarks for five distinct tasks and evaluated state-of-the-art (SOTA) methods for each, providing a baseline for future research.
4. Insightful Analysis: The paper provides deep analysis of modality affinity, revealing that while PAN+MS fusion is highly effective, RGB+SAR fusion remains challenging due to domain gaps and temporal misalignment.

4. Experimental Results & Findings

The authors evaluated representative methods across the five tasks:

• Task 1: Multi-modal Detection:
  • Single-Modality: RGB achieved the highest mAP (68.7%), while SAR was lowest (50.9%) due to lack of texture. However, SAR proved crucial in occluded scenes.
  • Fusion: ICAFusion achieved the best performance. Notably, PAN+MS fusion yielded significant gains over single modalities, whereas RGB+SAR fusion showed mixed results, often failing to outperform RGB alone due to high heterogeneity and misalignment.
• Task 2: Cross-modal Re-ID:
  • Performance: PAN↔RGB achieved the highest accuracy (Rank-1 ~96%) due to spectral similarity. RGB↔SAR and SAR↔PAN performed poorly (Rank-1 ~35-40%), highlighting the severe domain gap between optical and radar.
  • Finding: Modality affinity is a stronger determinant of performance than the specific loss function used.
• Task 3: Cross-modal Generation:
  • Metrics vs. Perception: A trade-off exists between pixel-level metrics (PSNR/SSIM) and perceptual quality (FID). Methods like CycleGAN achieved better FID (semantic structure) but lower PSNR, while WflmGAN achieved high PSNR but produced semantic distortions (e.g., misclassifying land as water).
  • Difficulty Hierarchy: RGB↔PAN (easiest) < SAR↔PAN < RGB↔SAR (hardest).
• Task 4: Pan-sharpening:
  • LAGNet outperformed other methods (PSNR: 25.94, SSIM: 0.839), demonstrating the dataset's utility for high-fidelity spatial-spectral fusion in marine environments.
• Task 5: Cross-modal Change Detection:
  • SS-SegFormer (B1 backbone) achieved the best results (mF1: 71.5%, mIoU: 63.5%), validating that Transformer-based architectures are effective for aligning heterogeneous optical and SAR temporal data.

5. Significance

• Advancing Maritime Surveillance: SMART-Ship fills a critical void in the RS community by providing a dataset specifically tailored to the unique challenges of maritime environments (water reflectance, port infrastructure, ship diversity).
• Enabling Complex Research: The dataset enables research into multi-task learning and cross-modal transfer, moving beyond simple detection to complex tasks like re-identification and change monitoring.
• Benchmarking Standard: By providing rigorous benchmarks and evaluating SOTA methods, the paper establishes a new standard for evaluating multi-modal algorithms, highlighting that current fusion strategies need significant improvement to handle the RGB-SAR domain gap.
• Future Directions: The results suggest a need for modality-specific architectures and better alignment mechanisms to handle the inherent differences between optical and radar sensing, particularly for long-term maritime monitoring.

In conclusion, SMART-Ship is a foundational resource that not only provides high-quality, multi-modal data but also rigorously defines the current capabilities and limitations of multi-modal learning in the maritime domain.