Label Cleaning in Multimodal Public Retinal Image Datasets
This study systematically identified and corrected label noise in nine public multimodal retinal image datasets by combining automated screening with expert verification, revealing that approximately 5.39% of images contained errors—including significant proportions of out-of-distribution and multi-label cases that challenge conventional single-label frameworks—while demonstrating that tools like Cleanlab effectively reduce manual review burdens.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The human eye is a window to the body's health, and the retina at the back of the eye is a map where diseases like diabetes and macular degeneration leave their first traces. To understand these conditions, doctors take pictures of the retina using specialized cameras. In recent years, scientists have taught computers to read these pictures, hoping to build artificial intelligence that can spot sickness faster and more accurately than a tired human eye. But for a computer to learn, it needs a massive library of examples, and each example must come with a correct label, like a tag saying "healthy" or "diseased." If the tags are wrong, the computer learns the wrong lessons. This problem of incorrect tags, known as label noise, is a hidden flaw in many public collections of eye images. Even a small number of bad tags can mislead a computer, causing it to miss real diseases or invent problems that do not exist. Before these digital doctors can be trusted, the maps they study must be checked and corrected.
A team of researchers set out to clean up nine large public collections of retinal images, which included different types of photographs: standard color photos, cross-sectional scans, and ultra-wide views that capture the entire back of the eye. They gathered nearly 35,000 images from these sources and subjected them to a rigorous screening process. First, they used a computer program designed to spot patterns and inconsistencies. This software acted like a highly efficient filter, scanning the images to find duplicates, strange outliers, and pictures where the label seemed to contradict what the image actually showed. The program flagged over 4,400 images as suspicious, a task that would have taken human experts weeks to perform manually.
Once the computer did its initial sweep, the real work began. The flagged images were passed to a team of ophthalmologists, or eye doctors, who examined them one by one. This was not a casual glance; it was a structured, four-stage review process where multiple doctors independently assessed each image, often comparing notes with senior colleagues to reach a final, agreed-upon diagnosis. This human verification step was crucial because it turned the computer's guesses into confirmed facts. The doctors found that the computer was right to be suspicious. Out of the nearly 35,000 images, they confirmed that 1,884 of them had incorrect labels. This means that roughly one in every 20 images in these public datasets was tagged with the wrong disease or condition.
The nature of these errors revealed something important about the limits of current medical data. Many of the mistakes were simple swaps, where a picture of one disease was labeled as another. However, a significant portion of the errors were more complex. Some images showed patients with multiple diseases at once, which the original single-label system could not capture. Others showed rare conditions that did not fit into the standard categories at all, or images that were so blurry or poor in quality that no reliable diagnosis could be made. In fact, nearly a quarter of the confirmed errors fell into these "out-of-distribution" categories, meaning they were cases that the original dataset was not designed to handle. The researchers also found hundreds of duplicate images, where the same picture appeared multiple times under different names, which can trick a computer into thinking it has seen more evidence than it actually has.
The study demonstrated that a combination of automated screening and expert human judgment is a powerful way to fix these datasets. The computer program successfully identified the vast majority of the errors, reducing the workload for the human doctors by filtering out the clear cases and focusing their attention on the difficult ones. When the computer suggested a new label for a mislabeled image, it was correct more than half the time, and in some datasets, it was right almost every time. This suggests that such tools can serve as a reliable first line of defense, guiding experts to the images that need the most attention. The researchers concluded that while artificial intelligence holds great promise for eye care, the data it learns from must be held to a high standard. By systematically cleaning these public image libraries, the team has provided a clearer foundation for future research, ensuring that the next generation of medical AI is trained on truth rather than error.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.