A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception

Technical Summary: SOVIS – A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception

Problem Statement
Underwater robots rely on optical cameras for rich semantic details (color, texture) and acoustic sonars for robust range measurements in turbid or low-light conditions. However, neither modality alone provides a complete scene understanding. While cross-modal prediction (learning to map between visual and acoustic representations) has driven significant advances in terrestrial robotics (e.g., LiDAR-camera fusion for autonomous driving), it remains underexplored in underwater environments. The primary bottleneck is the lack of large-scale, paired sonar-visual datasets and the difficulty of synchronizing sensors with fundamentally different coordinate systems (dense Cartesian pixels vs. sparse polar beams) in complex underwater physics.

Methodology
The authors present SOVIS (SOnar-VISual), a real-world dataset and accompanying tooling designed to bridge this gap.

1. Data Collection Platform:

   • Hardware: A Blueye X3 ROV equipped with a built-in 1080p monocular camera (30 fps, wide-angle) and a Blueprint Subsea Oculus M750d multibeam sonar (750 kHz, 130° aperture, 512 beams).
   • Environment: Data was collected across 17 dives at six sites in the Trondheimfjord, Norway, ranging from 0 to 91 meters depth with water temperatures between 7°C and 18°C.
   • Sensors: The platform also logs water temperature and hydrostatic pressure at 1 Hz, essential for correcting acoustic range estimates based on sound speed variations.

2. Processing Pipeline:

   • Ingestion: Raw video, sonar files, and telemetry are converted into a unified format (ROS 2 bags).
   • Synchronization: Camera frames are anchored using OCR-based timestamp detection on video overlays. Sonar pings are matched to camera frames within a 2-second tolerance window after cross-device timestamp correction.
   • Image Generation: Raw sonar pings (1D arrays) are reshaped into 2D polar grids (446 range bins × 512 beams), cropped/padded to a uniform size, and aligned frame-by-frame with the camera stream.

3. Cross-Modal Annotation Tool:

   • To address the difficulty of labeling across coordinate systems, the authors developed an interactive tool that displays synchronized camera and sonar views side-by-side.
   • Projection: It uses a parameterized lens model to automatically project bounding boxes drawn on the camera image (Cartesian) into the sonar's polar coordinate system (angle and range).
   • Semi-Automation: The tool integrates a three-stage pipeline: (1) YOLOv8m for camera-based fish detection, (2) temporal median subtraction for sonar motion detection, and (3) cross-modal fusion based on angular overlap. This reduces labeling time by approximately an order of magnitude compared to manual annotation.

4. Proof-of-Concept Task (Cross-Modal Fish Detection):

   • Task: Predict the sonar bounding box (angle center, angle width, range center, range width) given a camera image and a fish bounding box.
   • Model (SOVISFishNet): An EfficientNet-B4 backbone extracts multi-scale visual features, fused via a Feature Pyramid Network (FPN). A 2D Gaussian spatial attention mask, generated from the input camera bounding box, gates the feature map. The attended features are concatenated with normalized bounding box coordinates and decoded by an MLP head to predict the four sonar parameters.

Key Contributions

• SOVIS Dataset: A real-world dataset comprising ~76,600 synchronized sonar-visual pairs, including environmental logs (temperature/pressure), collected in open ocean conditions.
• Annotation Tooling: An interactive interface that enables efficient cross-modal labeling by projecting annotations between visual and acoustic domains, significantly accelerating the creation of paired ground truth.
• Benchmark Task: A demonstration of cross-modal fish detection, showing that models can learn to map visual inputs to acoustic outputs.

Results
The authors evaluated SOVISFishNet on a small subset of 306 labeled fish instances (276 training, 30 validation).

• Performance: The model achieved 46.7% mAP@0.10 (mean Average Precision at IoU threshold 0.10).
• Comparison: This represents a 7× improvement over a geometric baseline that maps camera detections to sonar coordinates using only known sensor geometry and a fixed mean range.
• Error Analysis:
  • Range: The model reduced median range error from 0.644 m (baseline) to 0.102 m, demonstrating its ability to learn visual depth cues (e.g., apparent fish size, scene context).
  • Angle: The model had a slightly higher median angle error (4.8°) compared to the geometric baseline (3.3°), which is attributed to the baseline using calibrated angular mapping directly, whereas the model must learn this relationship from a limited dataset.

Significance and Claims
The paper positions SOVIS as the first step toward advancing cross-modal underwater perception research. The authors claim that the dataset and tooling enable research directions previously unexplored in the underwater domain, such as:

• Dense Sonar Prediction: Generating dense sonar intensity maps directly from monocular images.
• Sensor Fusion: Allowing robots to infer acoustic scene structure without active sonar, or to recognize objects solely from acoustic returns using visual priors.

The authors explicitly state that while the current work focuses on bounding-box-level fish detection as a proof-of-concept, the full synchronized dataset provides the foundation for broader tasks, including dense pixel-level alignment and more complex cross-modal prediction scenarios. They emphasize that the dataset serves to overcome the "lack of paired multi-modal data" that has historically bottlenecked this field.