🤖 AI

Detecting and refurbishing ground truth errors during training of deep learning-based echocardiography segmentation models

This study demonstrates that the Variance of Gradients method is effective in detecting errors in ground truth labels for echocardiography segmentation, and that correcting these errors during the training process significantly improves model performance, particularly at high error levels.

Original authors: Iman Islam, Bram Ruijsink, Andrew J. Reader, Andrew P. King

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Iman Islam, Bram Ruijsink, Andrew J. Reader, Andrew P. King

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot with a Flawed Textbook

Imagine you are trying to teach a robot to draw a perfect map of a house. You give the robot a "Ground Truth" textbook that shows exactly where the walls, doors, and windows should be.

In the world of medical AI, this "textbook" is a set of medical images (like ultrasound pictures of a heart) that have been manually traced by human doctors. The robot learns by looking at these images and trying to copy the traces.

The Problem: Humans make mistakes. Sometimes a doctor gets tired, distracted, or just draws a line in the wrong place. If the robot learns from a textbook full of these mistakes, it might learn to draw bad maps too.

This paper asks two main questions:

How much can the robot handle before it starts drawing garbage because the textbook is wrong?
Can we build a system that spots the bad pages in the textbook while the robot is learning and fixes them on the fly?

The Experiment: Breaking the Textbook on Purpose

The researchers used a dataset of heart ultrasound images called CAMUS. To test their ideas, they didn't wait for real mistakes; they created "fake" mistakes in the data to see what happened. They introduced three types of errors, like a mischievous editor tampering with the textbook:

The "Missing Piece" (Incomplete Label): Imagine a drawing of a heart where half of the left ventricle is just blank white space. The teacher forgot to draw it.
The "Smudged Edge" (Boundary Distortion): Imagine the outline of the heart is drawn too thick or too thin, like someone erasing and redrawing the border until it's messy.
The "Wrong Room" (Merged Labels): Imagine the teacher accidentally colored the "living room" (Left Ventricle) the same color as the "kitchen" (Left Atrium), merging them into one big blob.

They tested two scenarios:

Random Errors: Like a student randomly scribbling on a few pages here and there.
Systematic Errors: Like a teacher who always forgets to draw the left side of the heart, every single time.

The Detective: How to Spot the Bad Pages

The researchers needed a way to find the bad pages while the robot was studying. They compared two detective methods:

The "Grumpy Teacher" (Loss-Based): This method looks at how much the robot struggles with a specific image. If the robot keeps getting a high "score" (loss) on a picture, the teacher assumes, "This picture must be wrong."
The "Fingerprint Scanner" (VOG - Variance of Gradients): This is the method the paper proposes. Instead of just looking at the final score, it watches the robot's brain waves (gradients) over time. If the robot's brain is constantly confused and changing its mind about a specific image from one day to the next, the VOG method flags it as "Suspicious."

The Result: The "Fingerprint Scanner" (VOG) was much better at catching the bad pages than the "Grumpy Teacher." It could tell which images were actually corrupted with high accuracy.

The Fixer: "Refurbishing" the Textbook

Once the bad pages were found, the researchers didn't just throw them away. They used a technique called Pseudo-labelling (or "Refurbishing").

Think of it like this: If the robot has been studying for a while and is pretty good at drawing hearts, but it finds a page in the textbook that looks weird, the robot says, "I think the textbook is wrong here. Based on what I've learned so far, here is how I think this heart should look."

The system then replaces the bad human drawing with the robot's own "best guess" drawing. This is like the student correcting the teacher's mistake in the textbook using their own knowledge.

What They Discovered

1. The Robot is Surprisingly Tough
Even with a textbook full of random mistakes (up to 50% of the pages!), the standard robot (a U-Net model) still did a pretty good job. It was surprisingly resilient. It's like a student who can still pass the exam even if the teacher accidentally crossed out half the answers in the study guide.

2. The Fixer Helps When Things Get Bad
When the errors were "Systematic" (the same mistake happening over and over, like the teacher always forgetting the left side), the robot struggled more. In these cases, the "Refurbishing" strategy (fixing the bad pages) helped the robot perform significantly better.

3. The Sweet Spot
The fixer worked best when there were moderate amounts of errors. If the textbook was perfect, you didn't need a fixer. If the textbook was 100% destroyed, the robot couldn't learn anything to begin with. But in the messy middle ground, the fixer saved the day.

The Bottom Line

The paper concludes that:

Deep learning models for heart imaging are very robust. They can handle a surprising amount of human error in their training data without breaking.
The VOG method is a great, lightweight tool to spot which specific images are corrupted without needing to retrain the whole system.
Refurbishing (fixing the labels) is useful, but maybe we don't need to do it for every image. It's like a quality control check: you only need to fix the pages that are clearly wrong, not the whole book.

In short, the robot is tougher than we thought, but having a smart system to catch and fix the teacher's mistakes makes the final result even better.

Technical Summary: Detecting and Refurbishing Ground Truth Errors in Echocardiography Segmentation

Problem Statement

Deep learning (DL) models for medical image segmentation, particularly in echocardiography, rely heavily on ground truth (GT) labels generated through manual annotation. While these labels are treated as a "gold standard," they are susceptible to human error, including random mistakes and systematic biases caused by inter-annotator variability. Despite the prevalence of such errors in real-world datasets, their impact on the training of state-of-the-art (SOTA) segmentation models has been largely overlooked. Existing literature on handling label errors is predominantly focused on classification tasks, leaving significant gaps regarding multi-class segmentation, the robustness of models to different error types, and the specific challenges posed by echocardiography images (e.g., anatomical variability and poor image quality).

Methodology

The study utilizes the CAMUS dataset, specifically selecting 500 apical four-chamber (A4C) view images at end-diastole with manual labels for the left ventricle (LV), left ventricular myocardium (LVM), and left atrium (LA).

1. Error Simulation

To evaluate model robustness, the authors introduced synthetic but realistic errors into a random subset of the training and validation data. Three specific error types were simulated:

Incomplete label error: Parts of a structure are left unlabelled (background).
Boundary distortion error: Labels are dilated or eroded.
Merged labels error: One structure is incorrectly merged with another.

These errors were applied in two categories:

Random errors: Arbitrary mistakes where any structure has an equal chance of being affected.
Systematic errors: Consistent mislabelling of specific structures (e.g., consistent erosion of boundaries or consistent merging of LVM into LV).

2. Baseline Model

A standard U-Net architecture was used as the baseline, trained with cross-entropy loss, the Adam optimizer, and a batch size of 4 for 100 epochs.

3. Proposed Pipeline: Detection and Refurbishment

The authors propose an iterative pipeline to detect and correct erroneous labels during training:

Detection (VOG): Instead of relying solely on training loss, the study employs the Variance of Gradients (VOG) method. This measures the gradient variability of each sample's activations over a window of previous epochs ( $t=5$ ). Samples with VOG scores exceeding a threshold defined by the upper bound of the interquartile range ( $Q3 + 1.5 \times IQR$ ) are flagged as outliers (potentially mislabelled).
Refurbishment (Pseudo-labelling): Following a 10-epoch warm-up period, the system performs label refurbishment every 5 epochs. For flagged samples, the original GT label is replaced with a pseudo-label. This pseudo-label is calculated by averaging the model's predictions over the preceding 5 epochs.

Key Contributions

Robustness Evaluation: The study systematically investigates the robustness of SOTA DL segmentation models to both random and systematic GT label errors in a multi-class echocardiography segmentation task, a scenario previously under-explored.
Novel Detection Strategy: The paper compares a loss-based error detection method with a VOG-based approach, demonstrating the superiority of VOG in identifying noisy labels.
Refurbishment Framework: It proposes and evaluates a specific pipeline combining VOG-based detection with pseudo-labelling refurbishment to correct suspected errors dynamically during training.

Experimental Results

Experiment 1: Error Detection Performance

When 25% of the data contained random label errors, the VOG-based detection method consistently outperformed the standard loss-based approach across all three error types (Incomplete, Boundary Distortion, Merged).

Accuracy: VOG achieved 0.90–0.99 accuracy compared to 0.81–0.95 for the loss-based method.
Sensitivity: VOG showed significantly higher sensitivity (e.g., 0.97 vs. 0.87 for boundary distortion; 1.0 vs. 1.0 for merged labels).

Experiment 2: Refurbishment Efficacy

The study measured the Dice score improvement of labels after refurbishment.

Random Errors: Refurbishment consistently improved Dice scores, particularly when error rates were below 50%. For example, in the "Merged labels" scenario with 12.5% random errors, the Dice score improved from 0.53 to 0.93.
Systematic Errors: Improvements were observed but were less consistent at higher error rates (e.g., 50% systematic errors showed mixed results depending on the error type).

Experiment 3: Model Robustness and Pipeline Impact

Baseline Robustness: Surprisingly, the standard U-Net maintained relatively strong performance even with increasing proportions of random GT label errors.
Systematic vs. Random: Systematic errors caused more pronounced degradation in segmentation quality than random errors.
Pipeline Benefit: The detection and refurbishment strategy provided performance gains over the baseline, particularly under high-error conditions and for systematic errors. However, at low error levels, the gains were modest.

Significance and Claims

The paper concludes with several modest but significant observations:

Unexpected Resilience: A naively trained U-Net demonstrates surprising robustness to a wide range of GT label errors, particularly random ones. This finding suggests that current models may be more tolerant of annotation noise than previously assumed.
Practical Utility of VOG: The VOG method is identified as a reliable, lightweight tool for flagging corrupted labels without requiring architectural changes or retraining from scratch.
Clinical Implications: Given that annotation errors in practice typically affect less than 10% of data, the study suggests that while error detection and refurbishment strategies are valuable for high-error or systematic error scenarios, they may be best applied selectively rather than as a routine requirement for all clinical datasets.
Contribution to the Field: The authors claim that the observed robustness of standard models under such conditions has not been previously reported, offering a valuable perspective on the reliability of DL models in real-world clinical settings where perfect annotations are unattainable.