🤖 machine learning

Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels

This paper proposes Federated Global Reviser (FedGR), a novel method that leverages the global model's inherent slow memorization of noisy labels to collaboratively rectify label noise and regularize local training, thereby achieving superior robustness against heterogeneous label noise in federated learning compared to state-of-the-art baselines.

Original authors: Yuxin Tian, Mouxing Yang, Yuhao Zhou, Jian Wang, Qing Ye, Tongliang Liu, Gang Niu, Jiancheng Lv

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxin Tian, Mouxing Yang, Yuhao Zhou, Jian Wang, Qing Ye, Tongliang Liu, Gang Niu, Jiancheng Lv

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: A Classroom with Bad Notes

Imagine a group of students (called clients) who are trying to learn a subject together, but they are in different rooms and cannot share their notebooks directly to protect their privacy. This is Federated Learning (FL).

Usually, they rely on a teacher (the server) to combine their notes into one "Master Textbook" (the Global Model).

However, in the real world, some students have bad notes (noisy labels). Maybe a student wrote "Cat" next to a picture of a "Dog" by mistake. If the teacher just averages everyone's notes, the Master Textbook gets confused and starts learning the wrong things. This is the Federated Label-Noise (F-LN) problem.

Previous methods tried to fix this by asking each student to check their own notes for errors. But because every student has a different style of writing and different types of mistakes (some mix up cats and dogs, others mix up cars and trucks), these local checks often fail.

The Big Discovery: The "Slow Learner" Advantage

The authors of this paper noticed something surprising while watching the students learn.

In a traditional classroom (Centralized Learning), if you give a student bad notes, they eventually memorize the mistakes perfectly. They get too confident in the wrong answers.

But in this Federated setup, the Master Textbook (the Global Model) acts like a slow learner. It takes a long time to memorize the bad notes. While the individual students are frantically memorizing their specific errors, the Master Textbook is still holding onto the correct, general patterns. It's like a wise old librarian who ignores the scribbles in the margins and focuses on the main story.

The paper calls this the "Intrinsic Label-Noise Robustness" of Federated Learning. The global model naturally resists learning from mistakes better than a single model trained on all the data at once.

The Solution: FedGR (The Global Editor)

The authors built a new system called FedGR (Federated Global Reviser) to use this "slow learner" superpower. Instead of asking students to fix their own notes, the Master Textbook acts as a Global Editor to help them.

FedGR works in three steps, like a three-part editing process:

1. The Sieve and Refine (The Detective)

The Analogy: Imagine the teacher collects a sample of every student's homework (just the score, not the actual answers, to keep privacy). The teacher looks at the class as a whole to figure out which students are likely making mistakes and which specific homework problems are probably wrong.
How it works: The server uses a statistical tool (a "Gaussian Mixture Model") to separate "clean" data from "noisy" data for each student. It tells the student, "Hey, these 10 problems look suspicious; let's fix them."
The Fix: For the suspicious problems, the teacher uses the Master Textbook to guess the correct answer (a "pseudo-label") and updates the student's notes.

2. The Global Revision (The Mentor)

The Analogy: Sometimes, a student's notes are so messy that even their own "study buddy" (a local model) gets confused. The teacher steps in and says, "Don't just trust your own notes; look at what the Master Textbook says."
How it works: The system creates a "safe" version of the student's model (an Exponential Moving Average or EMA). Before the student learns from this safe version, the teacher "revises" it by mixing in the Master Textbook's knowledge. This ensures the student doesn't learn from their own accumulated mistakes.

3. The Representation Guard (The Bodyguard)

The Analogy: Even with a revised model, a student might still get too attached to their specific, weird way of seeing things. The teacher puts a bodyguard on them to make sure they don't drift too far from the group's general understanding.
How it works: The system forces the student's model to keep its "internal view" of the data consistent with the Master Textbook's view. If the student starts seeing a "Dog" as a "Cat" because of bad notes, the bodyguard pushes them back toward the correct understanding.

The Results: Why It Matters

The authors tested this system on three different "classrooms" (datasets: CIFAR-10, CIFAR-100, and Clothing1M) with various levels of bad notes and different student groupings.

The Outcome: FedGR consistently outperformed seven other top methods.
The Key Win: Even when the noise was severe (lots of bad notes) and the students were very different from each other (high heterogeneity), FedGR kept learning effectively. In some cases, it even performed better than a standard system trained on perfectly clean data, simply because its "Global Editor" strategy was so good at filtering out the noise.

Summary

The paper argues that instead of fighting against the noise, we should lean into the unique way Federated Learning works. By realizing that the Global Model is naturally slower to learn mistakes, we can use it as a powerful tool to clean up the data and guide the local students, creating a system that is both privacy-preserving and highly robust against errors.

1. Problem Definition

The paper addresses the Federated Label-Noise (F-LN) problem. While Federated Learning (FL) enables privacy-preserving collaborative training, it relies heavily on high-quality labels, which are often unavailable in real-world decentralized scenarios (e.g., healthcare, recommendation systems).

The F-LN problem is significantly more challenging than centralized label-noise learning due to dual heterogeneity:

Label-Noise Heterogeneity: Different clients experience varying noise types (symmetric, asymmetric, mixed), noise ratios, and patterns.
Data Heterogeneity: Clients possess non-I.I.D. (Independent and Identically Distributed) data distributions, leading to class imbalances.

Existing methods often fail because they either rely on centralized assumptions (like class balance) or require transmitting sensitive local statistics to build consensus, posing privacy risks.

2. Key Observation: Intrinsic Label-Noise Robustness

The authors identify a critical, previously under-explored phenomenon: The global FL model exhibits a slower memorization of noisy labels compared to centralized training.

Observation: In centralized training, models quickly overfit to noisy labels, degrading test performance. In contrast, the global FL model (aggregated via FedAvg) tends to memorize clean patterns first and delays fitting to noise.
Implication: This "intrinsic robustness" allows the global model to maintain reliable predictions and robust representations even when local data is noisy. This phenomenon is attributed to the aggregation process, where coherent signals (clean patterns) reinforce each other, while incoherent noise gradients tend to cancel out.

3. Methodology: FedGR (Federated Global Reviser)

Motivated by the observation above, the authors propose FedGR, a method that leverages the robust global model to correct local training without compromising privacy. FedGR consists of three collaborative modules:

A. Sieving-and-Refining Module

This module partitions data into clean and noisy subsets and corrects labels.

Federated Sieving (FS): Instead of each client modeling noise locally (which is unreliable under heterogeneity), the server aggregates instance-level data proxies (specifically, the mean inference loss of samples over previous rounds) from all clients.
- The server fits a Gaussian Mixture Model (GMM) to these global loss distributions to estimate the probability of a sample being "clean."
- This allows the server to partition client data into clean and noisy subsets and estimate the noise ratio ( $r_k$ ) for each client.
- Privacy: Only scalar loss values and opaque identifiers are transmitted, preserving data distribution privacy.
Label Refining (LR): Based on the server's partition and estimated noise ratio, clients refine their labels:
- For clients with low noise ratios, noisy labels are corrected using a weighted combination of the original label and a pseudo-label generated by the robust global model.
- For clients with high noise ratios (or complex noise types), all labels are treated as untrustworthy, and only pseudo-labels from the global model are used.

B. Globally Revised EMA Distillation Module

To address the conflict where the global model is robust but may not fit local distributions well, and local models fit local distributions but overfit noise:

Clients maintain a local Exponential Moving Average (EMA) model.
Before distillation, the local EMA model is revised by the global model parameters. This ensures the "teacher" model used for knowledge distillation is robust to noise.
The local student model learns from this revised teacher via Kullback-Leibler (KL) divergence on weakly augmented data. This regularizes the local training and prevents overfitting to cumulative errors.

C. Global Representation Regularization Module

To further prevent local models from overfitting to noisy labels:

A consistency constraint is enforced between the global representation (from the global model on weakly augmented data) and the local representation (from the local model on strongly augmented data).
This ensures the local model learns robust feature representations aligned with the global consensus, rather than memorizing specific noisy label artifacts.

4. Key Contributions

Novel Observation: The paper provides empirical evidence and theoretical intuition that the global FL model has an intrinsic robustness to label noise due to the aggregation mechanism, a property not fully exploited in prior work.
FedGR Framework: A self-contained, privacy-preserving method comprising three modules (Sieving-and-Refining, Globally Revised EMA Distillation, and Global Representation Regularization) that effectively handles dual heterogeneity.
Theoretical Guarantee: The authors provide a convergence analysis proving that FedGR achieves the same non-convex convergence rate ( $O(1/\sqrt{T})$ ) as standard FedAvg.
Privacy Preservation: Unlike methods requiring local statistics exchange, FedGR only transmits loss values and identifiers, ensuring no leakage of raw data distributions.

5. Experimental Results

The authors evaluated FedGR on three benchmarks: CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset, under various noise types (Symmetric, Asymmetric, Mixed) and noise ratios (up to 100% noise on some clients).

Performance: FedGR consistently outperformed seven state-of-the-art baselines (including FedAvg, FedProx, FL-Coteaching, FL-DivideMix, FedCorr, FedNoRo, FedFixer, and FedDiv).
Robustness: In severe dual-heterogeneity settings (e.g., Non-I.I.D. data with high noise ratios), FedGR maintained high accuracy where other methods failed or collapsed.
Ablation Studies: Removing any of the three modules (Sieving, Distillation, or Regularization) resulted in significant performance drops, confirming the necessity of the full framework.
Efficiency: While FedGR introduces some communication overhead (transmitting loss proxies), it is significantly more efficient than methods requiring multiple peer networks or complex consensus mechanisms.

6. Significance

This work is significant because it shifts the paradigm for handling noisy labels in FL. Instead of treating FL as a distributed extension of centralized noisy label learning (which often fails due to heterogeneity), FedGR leverages the unique dynamics of the FL aggregation process itself as a defense mechanism. By utilizing the global model's inherent resistance to noise, FedGR offers a robust, privacy-preserving solution for real-world applications where data quality cannot be guaranteed, such as medical imaging and personalized recommendation systems.