Beyond Pixel Similarity: Task-Aware Evaluation of GAN-Based Synthetic Sonar Data for Robotic Perception

Underwater robots face a unique challenge: the ocean is often too dark, too murky, or too filled with floating debris for standard cameras to see clearly. To navigate these environments, engineers rely on imaging sonar, which uses sound waves to create pictures of the seabed and the objects within it. However, these sound-based images look very different from the photographs humans are used to. They are filled with grainy noise, strange shadows, and bright spots that depend heavily on the angle of view and the texture of the ocean floor. To teach a robot to recognize a shipwreck or a piece of trash in these difficult images, developers need thousands of labeled examples. Gathering and manually marking these real-world images is slow, expensive, and often dangerous. A common solution is to create synthetic data—computer-generated images that mimic the real thing—so robots can learn from a vast library of examples without needing to visit the ocean. But a critical question remains: how do we know if a fake sonar image is actually good enough to teach a robot?

For years, the standard way to judge the quality of a synthetic image has been to measure how closely it matches a real one, pixel by pixel. Researchers use mathematical tools to calculate the difference in brightness and color between the two images, giving them a score that indicates how similar they are. If the score is high, the assumption is that the synthetic image is realistic and therefore useful. A new study challenges this assumption, suggesting that for the specific task of helping robots "see" underwater, looking at the image as a whole is less important than looking at the specific details that a machine learning algorithm needs to find an object. The researchers investigated whether these traditional similarity scores truly reflect how well a synthetic image would help a robot detect objects, or if they are merely measuring visual resemblance without capturing the functional reality of the data.

To explore this, the team used a type of artificial intelligence known as a generative adversarial network, which acts like a pair of competing artists. One part of the system tries to create a realistic sonar image based on a simple outline of an object, while the other part tries to spot the difference between the fake image and a real one. The researchers tested four different versions of this "spotter," each with a different way of looking at the image. One version examined the image one tiny dot at a time, another looked at small patches, a third looked at medium-sized areas, and the fourth looked at the entire picture at once. They trained these systems using real sonar data from two different sources: one from a controlled pool environment and another from a variable river setting. Once the systems generated their synthetic images, the team evaluated them in two ways. First, they ran the traditional similarity tests to see which version produced the most visually accurate pictures. Second, they took the synthetic images and fed them into object-detection software that had been trained exclusively on real sonar data, essentially asking the software to find the objects in the fake pictures as if they were real.

The results revealed a surprising disconnect between how an image looks and how useful it is. In the controlled pool dataset, the system that examined the image one tiny dot at a time produced the highest similarity scores, making it look the most like the real reference image according to standard metrics. However, when the object-detection software tried to find items in those images, it performed significantly better with images generated by the systems that looked at small patches of the picture. Similarly, in the river dataset, the system that analyzed the entire image at once achieved the best similarity scores, yet the patch-based systems again allowed the detection software to find objects more accurately. The study found that the configurations which achieved the highest pixel-level similarity did not consistently yield the best detection performance. In fact, the systems that produced slightly blurrier or less pixel-perfect images often preserved the sharp edges and local structures that the detection software needed to identify targets.

This finding suggests that the tools we currently use to judge synthetic data might be missing the most important parts of the picture. The traditional scores reward an image for matching the overall brightness and general structure of a real photo, which can sometimes lead to images that are smooth and uniform but lack the specific, high-contrast details a robot needs to make a decision. The patch-based systems, by focusing on local regions, seemed to preserve the critical features that help a machine distinguish an object from the background, even if the overall image looked slightly less perfect to a human observer. The researchers conclude that for synthetic sonar data intended for robotic perception, relying solely on visual similarity metrics is insufficient. Instead, the true test of a synthetic image's quality should be how well it helps a robot perform its actual job. This shift in perspective implies that the future of training underwater robots may depend less on creating images that look exactly like reality and more on creating images that function effectively for the machines that need to see them.