💻 computer science

Towards Versatile Opti-Acoustic Sensor Fusion and Volumetric Mapping

This paper presents a robust volumetric mapping framework for autonomous underwater vehicles that fuses stereo sonar and monocular camera data to generate confidence-weighted 3D point clouds, effectively overcoming the limitations of individual sensors to enable safe navigation in both clear and turbid environments.

Original authors: Ivana Collado-Gonzalez, John McConnell, Brendan Englot

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivana Collado-Gonzalez, John McConnell, Brendan Englot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to navigate a boat through a dense, foggy forest at night. You have two main tools to help you: a flashlight (your camera) and a sonar (like a bat's echolocation).

Here is the problem:

The Flashlight: In clear air, it's amazing. You can see every leaf and branch in high definition. But if the fog gets thick (turbid water), the light scatters, and you see nothing but a white wall.
The Sonar: It cuts right through the fog. It tells you "there is a tree 5 meters away." But it's like a blurry, low-resolution sketch. It knows the distance, but it doesn't know if the tree is tall or short, or if it's leaning left or right. It's great at seeing how far things are, but bad at seeing how high they are.

This paper introduces a clever new system for underwater robots (AUVs) that combines these two tools to create a perfect 3D map, even in the murkiest water.

The Core Idea: The "Trust Me" Map

The researchers built a system that acts like a smart team of detectives. Instead of just blindly trusting one tool, they use a "Confidence Score" for every piece of data they collect.

Here is how the system works, step-by-step:

1. The "Two-Ears" Trick (Stereo Sonar)

The robot has two sonars mounted at right angles to each other (one looking forward, one looking sideways).

The Analogy: Think of this like having two ears. If you hear a sound, one ear might tell you it's to the left, and the other tells you it's to the right. By combining them, you can pinpoint exactly where the sound is coming from in 3D space.
The Result: Where the two sonars overlap, the robot gets a very accurate 3D point. It knows the distance, the angle, and the height. This is the most reliable data.

2. The "Spotlight" Trick (Camera + Sonar)

What about the areas where the two sonars don't overlap? Or what if the object is too close for the sonars to see clearly?

The Analogy: Imagine the sonar says, "There is a wall 2 meters away," but it doesn't know how tall the wall is. The camera (even if the water is a bit murky) can see the shape of the wall.
The Magic: The system uses a smart AI (a neural network) to find objects in the camera image. It then projects the "distance" from the sonar onto the "shape" from the camera. It's like taking a blurry sonar shadow and painting it with the high-definition details of the camera.
The Catch: This is less reliable than the "Two-Ears" trick because it relies on the camera seeing something. So, the system gives this data a medium confidence score.

3. The "Guessing Game" (Image Expansion)

Sometimes, the sonar sees a post, but the camera sees a whole pier.

The Analogy: If the sonar sees the bottom of a pier piling, and the camera sees the whole pier, the system assumes the rest of the pier follows the same distance. It "fills in the blanks" to create a complete picture.
The Catch: This is a big assumption. It might be wrong if the object is weirdly shaped. So, the system gives this data the lowest confidence score.

The "Gaussian Process" Brain

Once the robot has collected all these points (some super reliable, some okay, some just guesses), it needs to build a map.

Most old mapping systems treat every point equally. If a sonar glitch says there's a rock where there isn't one, the map gets messed up.

This new system uses something called Gaussian Process Volumetric Mapping.

The Analogy: Imagine you are painting a wall. You have a bucket of high-quality, bright blue paint (the reliable sonar data) and a bucket of slightly faded, watery blue paint (the camera guesses).
The Innovation: Instead of mixing them all together into a muddy mess, this system says, "I will use the bright blue paint for the main structure, and I will only use the watery paint to fill in the gaps where I have no other choice."
It weighs every single point based on its "Confidence Score." If a point is shaky, the map ignores it or treats it as "maybe." If a point is solid, the map locks it in as "real."

Why This Matters

The researchers tested this in two places:

A Clear Tank: They built complex shapes (like disks on poles) and showed their robot could map them perfectly, while other robots got confused or missed details.
A Murky Marina: They went to a real harbor in New York where the water was so dirty you couldn't see your hand in front of your face.
- Old methods: Either saw nothing (because the camera failed) or saw a blurry, incomplete mess (because the sonar couldn't tell height).
- This new method: Successfully mapped the wooden pilings and even spotted small metal pipes in front of them, which are critical for the robot not to crash into.

The Bottom Line

This paper is about giving underwater robots a superpower: the ability to see clearly in the dark and the fog by trusting the right tool at the right time.

It's like having a navigator who knows exactly when to trust their eyes and when to trust their ears, and who knows how to weigh the evidence to draw the most accurate map possible. This makes underwater exploration, pipeline inspection, and search-and-rescue missions much safer and more reliable.

1. Problem Statement

Autonomous Underwater Vehicles (AUVs) require accurate 3D volumetric mapping for safe navigation in obstacle-rich environments. However, existing sensing modalities face significant limitations in underwater conditions:

Optical Cameras: Provide high-resolution data but fail in turbid water, low light, or scattering media due to blurring and halo effects.
Imaging Sonars: Are robust to turbidity and lighting but suffer from low resolution, sparse data, and elevation ambiguity (they produce 2D projections of 3D scenes, lacking vertical depth information).
Current Fusion Methods: Many existing approaches rely heavily on visual features for matching, making them unreliable in turbid water. Others are limited to simple geometries or require large, scarce sonar training datasets.

The core challenge is to create a mapping framework that fuses a stereo sonar pair (two orthogonally mounted sonars) with a monocular camera to resolve elevation ambiguity and generate reliable 3D maps across varying visibility conditions, prioritizing safety-critical data.

2. Methodology

The proposed framework, GPC SS RGB (Gaussian Process Confidence mapping with Stereo Sonar and RGB), operates through a multi-stage pipeline:

A. Sensor Configuration & Data Association

Hardware: A BlueROV2 equipped with two forward-looking imaging sonars (horizontal and vertical) and one monocular camera with overlapping Fields of View (FOV).
Stereo Sonar Matching: Features detected in orthogonal sonar images are matched based on range and overlapping FOV. This resolves the elevation angle ( $\phi$ ) for points within the overlap, creating a fully defined 3D point cloud ( $P_{ss}$ ).
Handling Sparse Data: For close-range objects outside the stereo overlap or missed by stereo matching, the system retains unmatched sonar features ( $C_v, C_h$ ) to be processed later.

B. Optical Segmentation & Projection

Segmentation: Instead of point-feature matching (which fails in turbid water), the system uses a YOLO11n-seg model (trained via transfer learning on underwater imagery) to detect Regions of Interest (ROIs) in the camera image.
Projection & Masking:
1. Stereo Points: The 3D points from stereo sonar ( $P_{ss}$ ) are projected onto the camera image. Points falling outside the detected ROIs are discarded to remove erroneous matches.
2. Unmatched Sonar Expansion: For unmatched close-range sonar features, the system projects them to all possible elevations (creating vertical/horizontal lines in the image). These are then masked by the ROIs. If a pixel falls within an ROI, the system infers the elevation based on the object's position in the image, effectively "expanding" the sonar's limited vertical FOV using optical cues.

C. Confidence-Based Volumetric Mapping

Gaussian Process (GP) Regression: The framework uses GP occupancy mapping to create a continuous probabilistic map, which handles sparse data better than discrete grid methods (like OctoMaps).
Heteroscedastic Noise Modeling: A key innovation is assigning confidence values to different data sources, which translates to specific noise variances ( $\sigma^2_n$ $σ_{n}^{2}$ ) in the GP model:
- High Confidence ( $\alpha_{ss}$ ): Stereo sonar matches (no geometric assumptions, fully defined 3D).
- Medium Confidence ( $\alpha_s$ ): Sonar-to-image projections (assumes range validity within sonar FOV).
- Low Confidence ( $\alpha_e$ ): Image expansion (assumes constant standoff distance across the optical elevation span).
Fusion: The GP predictive distribution weights measurements by their reliability. High-confidence data dominates the map, while low-confidence data fills gaps without introducing significant errors.

3. Key Contributions

First Stereo Sonar + Monocular Camera Fusion: The first underwater sensing strategy to fuse a stereo sonar pair with a monocular camera to resolve elevation ambiguity and expand vertical FOV.
Confidence-Weighted Mapping: A novel volumetric mapping framework that prioritizes reliable measurements (stereo sonar) over less reliable ones (optical expansion) using heteroscedastic Gaussian Processes.
Robustness to Turbidity: The method relies primarily on sonar, using camera data only when available and valid, ensuring functionality in low-visibility conditions where pure vision fails.
Efficient Training: Uses transfer learning on visual data for segmentation, avoiding the need for scarce, manually annotated sonar datasets.
Open Source: Release of code and data to facilitate reproducibility.

4. Experimental Results

The system was evaluated in two environments: a clear-water tank with complex geometries and a turbid-water marina.

Tank Experiments (Clear Water):
- Setup: Mapped single and double disk structures with surface discontinuities.
- Metrics: The proposed GPC SS RGB method achieved a Mean Absolute Error (MAE) of 1.17 cm (Single Disk) and 1.62 cm (Double Disk), significantly outperforming baselines like OctoMap (5.98 cm) and standard GP (6.60 cm).
- Precision: It achieved the highest precision (88.18%) and effectively captured complex geometries that other methods missed or distorted.
Field Experiments (Turbid Water - Marina):
- Setup: Mapped a wooden pier with small metal pipes in highly turbid water.
- Qualitative Results: The proposed method successfully captured small, critical features (the pipes) and preserved the diagonal orientation of pilings.
- Comparison: Baseline methods either failed to capture small objects (Stereo Sonar only), overestimated occupancy (OctoMap), or distorted geometry due to single-distance assumptions (Single Sonar + RGB).
Computation Time: The confidence-based approach added negligible computational overhead compared to standard GP mapping, while being significantly faster than OctoMap.

5. Significance

This work addresses a critical gap in underwater autonomy: the ability to navigate safely in both clear and turbid waters without relying on fragile visual features. By prioritizing sonar data and using optical data only to refine elevation and expand FOV, the system ensures that safety-critical close-range information is preserved even when visibility is poor. The confidence-weighted fusion mechanism allows the robot to make informed decisions about which parts of the map are reliable, directly enhancing path planning and collision avoidance in complex, unstructured underwater environments.