🤖 AI

Federated Medical Image Segmentation under Real-World Label Noise: A Benchmark Suite for Noisy Label Learning Method Selection

This paper introduces a comprehensive benchmark suite designed to address the gap between synthetic and real-world evaluations in federated medical image segmentation by providing diverse noisy datasets and realistic client-noise scenarios to facilitate systematic assessment and informed selection of federated noisy label learning methods.

Original authors: Markus Bujotzek, Dimitrios Bounias, Stefan Denner, Ralf Floca, Maximilian Fischer, Peter Neher, Klaus Maier-Hein

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Markus Bujotzek, Dimitrios Bounias, Stefan Denner, Ralf Floca, Maximilian Fischer, Peter Neher, Klaus Maier-Hein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of doctors from different hospitals trying to build a super-smart AI that can automatically outline tumors and organs in medical scans. They want to do this together without sharing their patients' private data. This is called Federated Learning. Instead of sending the data to one central computer, they send the "brain" of the AI to each hospital, let it learn locally, and then send the "lessons learned" back to be combined.

However, there's a big problem: The teachers are making mistakes.

In the real world, the labels (the outlines drawn by doctors) aren't perfect. One doctor might draw a tumor slightly bigger, another might miss a tiny part, and a third might confuse a tumor with healthy tissue. If the AI learns from these messy, inconsistent drawings, it gets confused and performs poorly. This is the "Noisy Label" problem.

This paper is like a giant, realistic training camp designed to test different strategies for teaching this AI when the teachers are imperfect.

The Problem: The "Messy Classroom"

Imagine a classroom where the teacher (the AI) is trying to learn to draw a circle.

Student A draws a perfect circle.
Student B draws a circle that's a bit squashed.
Student C draws a square but calls it a circle.
Student D draws a circle but leaves a gap.

If the teacher just averages all their drawings, the result is a messy blob. In the medical world, this "mess" happens because different hospitals use different scanners, different doctors, or even different software to create the labels.

The Solution: A "Stress Test" Suite

The authors built a Benchmark Suite (a standardized test kit) to see which "teaching strategy" works best when the labels are messy. They didn't just use fake, computer-generated errors; they used six real-world datasets from actual medical studies involving CT scans, eye photos, and microscope images.

They tested four main strategies to fix the mess:

The "Group Vote" (FedAvg): The standard method. It just averages everyone's updates. It's simple but doesn't account for bad teachers.
The "Quality Control" (FedA³I): Tries to figure out which hospitals are drawing better contours and gives their answers more weight.
The "Local Specialist" (IOP-FL): Lets the AI adapt its brain specifically for each hospital's unique style of drawing, rather than forcing one size fits all.
The "Smart Filter" (FedSelect): This is the paper's star. It uses a clever trick to figure out which specific examples (images) are reliable and which are garbage, ignoring the bad ones during training.
The "Label Fixer" (FedCorr): Tries to rewrite the wrong labels based on what the global model thinks is right.

The Results: Who Won the Race?

The paper ran thousands of simulations with different types of noise (squashed circles, missing gaps, confused labels) and different scenarios (some hospitals are messy, others are clean).

The Champion: FedSelect (The Smart Filter) came out on top overall. It was the most consistent at ignoring the bad data and learning from the good data, no matter the type of error.
The Runner-Up: IOP-FL (The Local Specialist) was a strong competitor, especially in specific situations.
The Baseline: The standard "Group Vote" (FedAvg) was actually quite good and often beat the other fancy methods. This is a key finding: You don't always need a complex fix; sometimes the simple average works well enough.
The Losers: The other two methods (FedA³I and FedCorr) didn't improve much over the simple average and sometimes made things worse.

The "Cheat Sheet" (Decision Guide)

The authors didn't just say "FedSelect wins." They realized that the "best" method depends on what kind of mistake is being made.

If the problem is squashed outlines (contour errors), FedSelect is the best.
If the problem is missing or extra objects (like a tumor that isn't drawn at all), FedSelect or IOP-FL are best.
If the problem is confused labels (calling a tumor a cyst), FedSelect is still the leader, but the results are trickier.

They created a Decision Guide (like a flowchart) to help doctors and researchers pick the right strategy based on their specific data problems.

The Bottom Line

This paper is a reality check for the field of Federated Learning in medicine.

Real-world data is messy: It's not just one type of error; it's a mix of missing parts, extra parts, and wrong shapes.
Simple isn't always bad: The standard method (FedAvg) is still a very strong competitor.
The "Smart Filter" (FedSelect) is the new gold standard for handling this mess, but you have to choose your tool based on the specific type of noise you have.

The authors have made their code and test kit open-source, so anyone can use this "training camp" to test their own new ideas against these real-world challenges, ensuring that future medical AIs are built on solid, reliable foundations.

Technical Summary: Federated Medical Image Segmentation under Real-World Label Noise

Problem Statement

Federated Learning (FL) facilitates collaborative medical image segmentation without centralizing sensitive data. However, real-world deployment is hindered by label noise arising from inter-rater variability, annotation errors, and automated label generation. In segmentation, this noise manifests as contour inconsistencies, missing or additional target instances, and confused class labels. These imperfections degrade model performance and can propagate through clinical workflows.

While Federated Noisy Label Learning (FNLL) aims to mitigate these effects, its practical adoption is limited. Existing research relies heavily on synthetic noise, simplified client-noise scenarios, and narrow datasets, failing to capture the complexity of real-world federated settings where data and annotation heterogeneity are intrinsic. Furthermore, the lack of a standardized benchmark prevents systematic assessment and informed method selection for specific noise types and deployment scenarios.

Methodology

The authors introduce a comprehensive benchmark suite designed to evaluate FNLL methods under realistic conditions. The suite integrates four core components:

1. Datasets with Real-World Label Noise

The benchmark utilizes six curated medical image segmentation datasets covering diverse modalities (CT, Fundus, Microscopy, MR) and clinical domains (thoracic, retinal, prostate, mouse tumors, multi-organ).

Noise Source: Labels are derived from inherent inter-rater variability (multi-rater datasets) or automatically generated labels, rather than synthetic corruption.
Ground Truth: Clean references are established via majority voting, STAPLE fusion, or expert annotations.
Noise Types: The datasets exhibit a mix of contour disagreement, missing/extra instances, and class confusion.

2. Federated Benchmarking Framework

Architecture: Built on a federated adaptation of nnU-Net, ensuring task-adaptive preprocessing and consistent network configurations across clients.
Client-Noise Scenarios: Four scenarios are evaluated to test robustness:
- Clean: All clients use clean labels.
- Noisy: All clients use noisy labels.
- Ratio of Clients (roc): A fraction $p$ of clients are fully noisy.
- Ratio on All (roa): Each client contains a fraction $p$ of noisy samples.
Evaluation Metrics:
- General: Dice score (overlap-based).
- Noise-Specific: HD95 (contour disagreement), Instance-level F1 (missing/extra structures), and Class Confusion (label swapping).
- Statistical Rigor: Bootstrap resampling (1000 iterations) and one-sided paired Wilcoxon signed-rank tests with Holm-Bonferroni correction are used to assess significance against the baseline.

3. Benchmarked FNLL Methods

Five representative methods are compared against the standard FedAvg baseline:

FedA³I (Noise-aware aggregation): Estimates client reliability via a Gaussian Mixture Model (GMM) to reweight updates, targeting contour bias.
IOP-FL (Noise-robust personalization): Adapts models to client-specific distributions by fusing local and global gradient information.
FedCorr (Label correction): Identifies noisy clients and refines labels using global predictions and proximal regularization without requiring clean validation data.
FedSelect (Sample selection): Uses meta-learned strategies to select informative samples and weight clients based on training dynamics.

4. Experimental Setup

Datasets are partitioned into 3–5 clients based on clinically meaningful factors (e.g., scanner, center, rater). Hyperparameters are optimized via grid search to maximize average performance across datasets.

Key Results

Data Analysis

Real-world label noise is rarely isolated; it frequently occurs in combinations. GleasonHD exhibited the most severe and heterogeneous noise, while RIGA and MouseT showed predominantly contour-driven noise. MMIA was characterized primarily by missing or additional target structures.

Method Performance

Top Performer: FedSelect emerged as the strongest overall FNLL method, demonstrating high rank stability and consistent performance across datasets, noise types, and scenarios.
Competitor: IOP-FL was the main competitor, often achieving the best Dice scores in specific dataset-scenario comparisons, particularly in contour-dominated settings.
Baseline: FedAvg remained a strong, competitive baseline. In many cases, dedicated FNLL methods did not significantly outperform FedAvg, highlighting that FNLL superiority is not universal.
Underperformers: FedA³I and FedCorr showed the least consistent gains, frequently failing to improve over FedAvg in the tested settings.

Robustness and Specificity

Scenario Sensitivity: The roc scenario (partially noisy clients) caused more significant performance degradation than the roa scenario (partial noise within all clients).
Noise-Type Specificity:
- Contour Noise (HD95): FedSelect consistently matched or outperformed FedAvg.
- Instance Noise (F1): FedSelect and IOP-FL were the strongest.
- Class Confusion: FedSelect ranked best, though results were limited by the single dataset (GleasonHD) available for this metric.
Degradation vs. Absolute Performance: While FedSelect achieved the highest absolute Dice scores, FedCorr showed the smallest degradation relative to the clean baseline in highly noisy roc and fully noisy settings, largely driven by its performance on the difficult GleasonHD dataset.

Contributions and Significance

The paper claims the following contributions:

Benchmark Suite: A reusable, open-source framework (available at MIC-DKFZ/FedSegNoiseBench) combining diverse real-world noisy datasets, a federated training framework, and noise-targeted evaluation protocols.
Systematic Evaluation: The first comprehensive assessment of FNLL methods across multiple real-world noise types and client-noise scenarios, moving beyond synthetic noise limitations.
Decision Guide: An actionable guide (Table 4) linking observed label-noise characteristics and client-noise scenarios to suitable FNLL strategies, supporting informed method selection in deployment.
Community Resource: A foundation for fair benchmarking, dataset-specific noise characterization, and future method development under realistic federated settings.

Conclusion

The study concludes that while FedSelect is the most robust and consistent FNLL method overall, the choice of method is highly dependent on the specific noise type and client-noise scenario. The benchmark demonstrates that dedicated FNLL methods do not universally outperform FedAvg; rather, their utility is context-dependent. By providing a realistic and discriminative basis for evaluation, the suite enables the medical imaging community to move from synthetic assumptions to data-driven decisions for federated segmentation in noisy, real-world environments.