🤖 machine learning

Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning

This paper investigates how imperfect datasets (non-IID and noisy) and biased client selection affect federated learning performance, and proposes a privacy-preserving scoring method to effectively assess client contributions and mitigate these issues.

Original authors: Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei Chen

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your smartphone, your smartwatch, and your neighbor's laptop all want to learn how to recognize a cat, but they can't share their photos. Maybe the photos are too private, or maybe the internet connection is too slow to upload them all to a giant central computer. This is the problem Federated Learning tries to solve. Instead of gathering all the data in one place, the "teacher" (a central server) sends a basic lesson plan to everyone. Each student (a client device) practices on their own private photos, figures out what they learned, and sends just the lesson notes back to the teacher. The teacher mixes all the notes together to create a smarter lesson plan for the next round. It's like a massive, global study group where everyone keeps their homework hidden but still learns together.

However, this study group has a few messy problems. First, not everyone has the same amount of homework (some have 10 photos, others have 10,000). Second, some students only have photos of orange cats, while others only have black cats (this is called "label skew"). Third, some students might have accidentally scribbled "dog" on a picture of a cat (mislabeled data). If the teacher picks students randomly to share their notes, the class might learn weird things or get stuck. The big question is: How does the teacher decide who to pick to help the class learn best, without peeking at anyone's private homework?

This paper, titled "Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning," dives into exactly that mess. The authors, researchers from National Yang Ming Chiao Tung University, ran a series of experiments to see how different types of "bad" data affect the group's learning speed and final grade. They discovered that the solution isn't one-size-fits-all. Sometimes, you need to be fair and pick everyone equally; other times, you need to be strict and avoid students with messy data. To solve this, they invented a new "report card" system that lets the teacher rate how helpful each student is without ever seeing their private photos.

The Great Study Group Experiment

The researchers set up a virtual classroom with 100 clients (students) and a central server (the teacher). They used two classic datasets for their experiments: MNIST (handwritten numbers) and CIFAR-10 (pictures of animals and objects). In their simulation, they tested what happened when they changed the rules of the game.

The "Bad Data" Scenarios
They created three main types of trouble to see how the class reacted:

Quantity Skew: Some students had tiny notebooks, others had huge libraries. They found that bigger notebooks generally led to better grades, but if a notebook was too small, it couldn't help much.
Label Skew: Imagine a student who only has pictures of "3s" and another who only has "7s." If the class only listens to these two, they won't learn what a "1" or a "2" looks like. The researchers tested everything from "everyone has a mix" to "everyone has only one specific number." They found that when the data was very unbalanced (like one student having only "3s"), the class struggled to learn, especially with the harder CIFAR-10 pictures.
Mislabeled Data: This was the real troublemaker. They simulated students who had photos of cats labeled as "dogs." They tested three ways this could happen: random mistakes, sequential mistakes (where "1" becomes "2," "2" becomes "3," etc.), and cyclic mistakes. The results were stark: mislabeled data was much worse than unbalanced data. In fact, when they simulated a high level of sequential errors (where labels were shifted by 7 spots), the learning process completely failed. The class couldn't learn anything useful from students who were confidently wrong.

The "Fairness" Dilemma
Next, they asked: "Should the teacher pick students randomly (Fair) or avoid the ones with messy data (Unfair)?"

When the problem was Label Skew (unbalanced data): Being Fair was the winner. Even if a student only had pictures of "3s," letting them participate helped the class learn that specific number. If the teacher excluded them, the class missed out. The results showed that fair selection improved accuracy, though it sometimes took a few more rounds to finish.
When the problem was Mislabeled Data: Being Fair was a disaster. Including students with wrong labels dragged the whole class down. In these cases, the "Unfair" strategy (excluding the messy students) actually worked better. The paper suggests that when data is corrupted, you must prioritize quality over fairness.

The New "Report Card" System

Since the teacher can't peek at the students' private notebooks, how do they know who is helpful and who is messy? The authors proposed a three-part scoring system that runs entirely on the server side:

Datasize Score: This is simple. It just checks how big the student's notebook is. A bigger notebook gets a higher score (normalized between 0 and 1).
Quality Score: This is the clever part. After a student sends their lesson notes back, the teacher tests those notes on a secret "practice test" the teacher already knows the answers to. If the student's notes lead to a high score on the practice test, they get a high Quality Score. If they fail the practice test, their score drops. This tells the teacher, "This student's data is useful," or "This student's data is noisy," without ever seeing the actual photos.
Fairness Score: This is a "patience meter." Every round, every student's score goes up by a small amount. If a student gets picked to participate, their score resets to zero. This ensures that students who haven't been picked in a while eventually get a chance, preventing the teacher from only listening to the same few "smart" kids.

The Smart Balancer
The real magic is how they combine these scores. The system looks at the variance (the spread) of the Quality Scores.

If the Quality Scores are all similar (low variance), it means the data is mostly clean. The system then leans on the Fairness Score, making sure everyone gets a turn.
If the Quality Scores are all over the place (high variance), it means some students have terrible, noisy data. The system immediately switches gears, ignoring the Fairness Score and picking only the students with the highest Quality Scores.

What They Found (and What They Didn't)

The paper concludes that there is no single "best" way to pick students. It depends entirely on why the data is imperfect.

If the data is just unbalanced (some students have rare labels), the paper suggests that fairness is crucial. Excluding these students hurts the model.
If the data is corrupted (mislabeled), the paper suggests that fairness should be sacrificed to protect the model from bad information.

The authors are careful to note that their "Quality Score" is a great tool, but it has a blind spot. It can tell you that a student's data is "bad," but it can't tell you why. It doesn't know if the data is bad because the student only has "3s" (label skew) or because they wrote "dog" on a cat picture (mislabeled). Because of this, the system sometimes has to guess. If the variance is high, the system assumes the worst (mislabeled data) and prioritizes quality, which is a safe bet. However, the authors admit that if the high variance is actually just due to label skew, this conservative approach might miss out on some good learning opportunities.

In short, the paper doesn't claim to have solved the problem forever. Instead, it provides a measured, experimental guide showing that a smart, adaptive system—one that can switch between "be fair" and "be strict" depending on the situation—is the best path forward for Federated Learning. They suggest that future work should focus on building a system that can tell the difference between "unbalanced" and "corrupted" data even better, so the teacher can make the perfect choice every time.

Technical Summary: Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning

Problem Statement

Federated Learning (FL) enables decentralized model training while preserving client privacy, yet it faces significant challenges when clients possess imperfect datasets. These imperfections include non-independent and identically distributed (non-IID) data (manifesting as quantity skew, label skew, and feature skew) and noisy data (mislabeled samples). Such conditions can degrade model accuracy and increase convergence latency.

While client selection strategies are often employed to mitigate these issues by excluding problematic clients, an overly biased selection process can inadvertently harm learning performance. Specifically, there is an unresolved tension between fairness (ensuring balanced participation to prevent convergence to local optima) and quality (excluding clients with severe data issues). Furthermore, existing literature lacks a comprehensive, privacy-preserving method to predict a client's contribution to the global model without accessing their local data distributions.

Methodology

1. Experimental Analysis of Imperfect Data

The authors conducted extensive experiments using the MNIST and CIFAR-10 datasets with 100 clients and the FedAvg algorithm to quantify the impacts of various data imperfections:

Dataset Size: Verified that larger datasets correlate with improved accuracy and faster convergence.
Label Skew: Tested six scenarios ranging from IID to extreme single-label distributions. Results indicated that while label skew generally degrades performance, the severity varies by task (e.g., MNIST showed resilience compared to CIFAR-10).
Mislabeled Samples: Evaluated random, sequential, and cyclic mislabeling. The study found that mislabeling causes a more pronounced performance drop than label skew, with sequential mislabeling (where labels are shifted) causing learning failure at high degrees.
Fairness vs. Unfair Selection:
- In label-skewed environments, "Fair" selection (random, uniform participation) significantly improved accuracy compared to "Unfair" selection (excluding skewed clients), though it required more rounds to converge.
- In mislabeled environments, "Unfair" selection (excluding clients with mislabeled data) outperformed fair selection, as mislabeled samples provided no valuable learning information.

2. Proposed Scoring Mechanism

To address the need for privacy-preserving client assessment, the paper proposes a composite scoring system that evaluates clients based on data size, quality, and participation history without accessing local data distributions.

Datasize Score ( $d_i$ ): A normalized metric (0 to 1) based on the client's dataset size relative to the minimum and maximum sizes in the client set.
Quality Score ( $q_i$ ): A server-based assessment calculated at the end of each training round. The server evaluates the client's uploaded model on a global test dataset. The score is normalized based on the client's test accuracy relative to the minimum and maximum accuracies observed among all clients. This captures the joint impact of label skew and mislabeling.
Fairness Score ( $f_i^t$ ): A dynamic score that increases by a constant $\beta$ for every round a client is not selected and resets/decreases if selected. This ensures clients with long periods of inactivity are prioritized.

3. Adaptive Selection Mechanism

The paper introduces an adaptive mechanism to balance fairness and quality based on the variance of Quality Scores among available clients:

Low Variance: Indicates homogeneous data quality (neither severe skew nor mislabeling). The mechanism prioritizes fairness, selecting clients with high Fairness Scores.
High Variance: Indicates significant heterogeneity (likely severe mislabeling or skew). The mechanism prioritizes quality, expanding the candidate pool to ensure high-quality clients are selected regardless of their fairness score.
The number of clients selected ( $L_{fair}^t$ ) is dynamically adjusted using a formula involving the variance of Quality Scores, a sensitivity parameter ( $\alpha_1$ ), and a minimum proportion parameter ( $\alpha_2$ ).

Key Results

Impact of Mislabeling: Sequential mislabeling was identified as the most detrimental factor, causing learning failure in extreme cases. This suggests that clients with severe mislabeling should be excluded.
Context-Dependent Fairness: The experiments demonstrated that the optimal selection strategy is context-dependent. Fairness is beneficial when data heterogeneity stems from label skew but detrimental when it stems from mislabeling.
Scoring Effectiveness: The proposed Quality Score successfully correlated with data quality. In label-skew experiments, scores increased with the diversity of labels (up to a point where sample-per-label counts became too low). In mislabeling experiments, scores decreased as the mislabeling rate increased.
Adaptive Performance: The adaptive mechanism successfully navigated the trade-off, prioritizing fairness when data was relatively clean and prioritizing quality when data heterogeneity was high.

Significance and Claims

The paper claims to provide a comprehensive analysis of how different types of imperfect data (quantity skew, label skew, and mislabeling) affect FL performance. Its primary contribution is the demonstration that a single client selection strategy is insufficient; the strategy must adapt to the dominant source of heterogeneity.

The authors propose a privacy-preserving scoring framework that allows the server to assess client contributions using only global test results and dataset sizes, avoiding the need to inspect local data distributions. The significance lies in the adaptive mechanism that dynamically balances the competing goals of fairness and data quality, acknowledging that fairness can be harmful when clients possess severely mislabeled data.

The authors modestly note a limitation: the current Quality Score captures the joint impact of label skew and mislabeling but cannot distinguish between the two. Consequently, the adaptive mechanism treats high variance conservatively (prioritizing quality), which may be suboptimal if the variance is caused solely by label skew rather than mislabeling. Future work is suggested to develop separate metrics to quantify these factors independently.