🤖 AI

An automated method of identifying incorrectly labelled images based on the sequences of loss functions of deep learning networks

This paper proposes and validates an automated method for identifying incorrectly labelled medical images by analyzing deep learning loss function sequences, demonstrating that correcting these errors significantly improves dataset quality and model performance in diabetic retinopathy screening.

Original authors: Zhipeng Zhang, Wenhui Shou, Wengting Ma, Dongjia Xing, Qingqing Xu, Li-Qun Xu, Qingxia Fan, Ling Xu

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhipeng Zhang, Wenhui Shou, Wengting Ma, Dongjia Xing, Qingqing Xu, Li-Qun Xu, Qingxia Fan, Ling Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of fruit. You show it thousands of pictures, but some of the pictures have the wrong name tags stuck on them. For instance, you might accidentally label a banana as an apple. If you let the robot study these wrong tags, it will get confused and learn the wrong things.

In the medical world, this is exactly what happens with AI that analyzes eye scans. Doctors have to manually label thousands of images to train these AI systems. But doctors are human; they get tired, the images can be blurry, or the conditions can look very similar. The authors of this paper estimate that up to 10% of these medical labels might be wrong.

The Problem: The "Confused" Student
The researchers noticed something interesting about how AI "learns." When an AI looks at a picture with the correct label, it gets confident quickly. Its "mistake score" (called a loss function) goes down smoothly, like a student who understands the lesson and gets better grades every day.

However, when the AI looks at a picture with a wrong label, it gets stuck in a loop of confusion. It tries to learn, fails, tries again, and fails again. Its "mistake score" bounces around wildly or stays high, like a student who is constantly confused by the teacher's instructions.

The Solution: Listening to the AI's "Heartbeat"
The paper proposes a clever, automated way to find these bad labels without needing a human to check every single image. Here is how they did it:

The Long Run: They let the AI study the dataset many times over (like running a marathon of training sessions).
The Story of the Score: Instead of just looking at the final result, they recorded the AI's "mistake score" after every single training session. This created a unique "story" or sequence for every single image.
Finding the Patterns: They used a mathematical tool to look at these stories. They found that the stories of correctly labeled images all looked very similar (a smooth downhill slide). The stories of incorrectly labeled images looked totally different (bumpy, flat, or chaotic).
The Sort: They used a computer algorithm to group these stories into two piles: "The Confused Ones" (likely wrong labels) and "The Confident Ones" (likely correct labels).

The Experiment: Cleaning the Classroom
To test if this worked, the researchers took a perfect set of 10,788 eye images and secretly swapped the labels on 6% of them (648 images), pretending they were real-world mistakes.

The Hunt: They ran their new method on this messy dataset. It successfully found 75% of the swapped labels.
The Safety Net: Crucially, it didn't cry wolf too often. It only falsely accused about 5% of the correct labels.
The Cleanup: They took the images the computer flagged, had senior doctors review just those specific ones, and fixed the labels. This reduced the error rate in the whole dataset from 6% down to 1.5%.

The Result: A Smarter AI
Finally, they retrained the AI using this "cleaned" dataset. The result? The AI became better at its job. Its accuracy on a test set went up from 95.93% to 96.50%. While that number looks small, in the world of medical AI, getting closer to the "perfect" score (which was 96.57% when trained on the original, uncorrupted data) is a big deal.

In Summary
The paper shows that by listening to how an AI's "mistake score" changes over time, we can automatically spot the images that have the wrong names attached to them. This allows doctors to focus their time only on the images that need fixing, making the final dataset cleaner and the resulting medical AI more accurate.

Technical Summary: Automated Identification of Incorrectly Labelled Medical Images via Loss Function Sequences

Problem Statement
Deep learning models for medical image analysis rely heavily on the quality of manually labelled datasets. However, manual labelling by physicians is prone to errors due to factors such as physician fatigue, the small size of lesions (e.g., microaneurysms), ambiguous boundaries between disease categories (e.g., high myopia vs. pathologic myopia), and subjective interpretations of image gradability. It is estimated that up to 10% of manually labelled medical images may be incorrect. These errors, if present in training sets, mislead model optimization, while errors in validation or test sets prevent authentic performance evaluation. Traditional solutions involve senior physicians reviewing all initial labels, a process that is time- and energy-consuming.

Methodology
The authors propose an automated method to identify incorrectly labelled medical images by analyzing the temporal sequences of loss functions generated during the training of deep learning networks. The core hypothesis is that while correctly labelled samples exhibit a consistent, gradually decreasing loss trend across training epochs, incorrectly labelled samples (where the ground truth is wrong) cause the model to remain confused, resulting in loss sequences that maintain high levels or fluctuate significantly.

The proposed pipeline consists of three stages:

Multi-Epoch Model Training: A deep learning network (pre-trained on ImageNet) is trained on the labelled dataset for $N$ epochs ( $N > 2$ ). For each sample, the loss value (cross-entropy) is recorded at every epoch, forming a sequence of length $N$ .
Feature Extraction: Six specific features are extracted from each sample's loss sequence to characterize its trend:
- F1: The mean of the entire sequence (global loss level).
- F2: The mean of the first $P$ terms (early-stage local loss).
- F3: The position $Q$ where the largest drop in loss occurs between the $Q$ -th and $(Q+1)$ -th terms.
- F4: The mean of the first $Q$ terms (pre-significant decrease loss).
- F5: The mean of the last $N-Q$ terms (post-significant decrease loss).
- F6: The difference between F4 and F5 (extent of the significant decrease).
Clustering: Using the six extracted features, the $k$ -means clustering algorithm ( $k=2$ ) is applied to partition the dataset into two categories: one representing correctly labelled samples and the other representing incorrectly labelled samples.

Experimental Validation and Results
The method was validated using a fundus image dataset for referable diabetic retinopathy (DR) screening, comprising 10,788 samples with gold-standard labels.

Dataset Simulation: To simulate real-world noise, a synthetic dataset (A') was created by intentionally flipping the labels of 6% (648 samples) of the gold-standard dataset.
Identification Performance: In the first experiment, the method successfully identified 75.31% (488 out of 648) of the intentionally mislabelled samples. The specificity was 95.15%, meaning only 4.85% of correctly labelled samples were falsely identified as incorrect. The authors note that samples not identified often possessed poor image quality or ambiguous features that made them visually similar to the opposite class, even to human experts.
Model Performance Improvement: In the second experiment, the 980 samples identified as potentially incorrect were reviewed and corrected by physicians, creating a refined dataset (A'') with only 1.5% remaining errors. A deep learning model retrained on this refined dataset achieved a best accuracy of 96.50% on an independent test set. This represents an improvement over the model trained on the noisy dataset (95.93%) and approaches the ideal performance of 96.57% achieved by a model trained on the original, perfectly clean dataset.

Key Contributions and Significance
The paper claims that this automated approach offers a practical solution to the labor-intensive problem of data cleaning in medical AI. By leveraging the intrinsic behavior of loss functions during training, the method allows senior physicians to focus their review efforts on a small, high-probability subset of images (approximately 9.1% of the dataset in this study) rather than the entire collection. This process effectively improves the consistency and reliability of labelled medical image datasets, laying a more robust foundation for training deep learning models and ultimately enhancing their diagnostic performance. The authors conclude that while the method is effective for binary classification tasks like DR screening, further research is required to validate its utility in multi-class classification and segmentation tasks.

Technical Summary: Automated Identification of Incorrectly Labelled Medical Images via Loss Function Sequences

More like this