🤖 machine learning

Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data

This paper proposes EdgeFD, a resource-efficient federated distillation method that utilizes a KMeans-based density ratio estimator for client-side filtering of proxy data, thereby eliminating the need for complex server-side filtering and outperforming state-of-the-art methods in non-IID scenarios with reduced computational overhead.

Original authors: Ahmed Mujtaba, Gleb Radchenko, Radu Prodan, Marc Masana

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Mujtaba, Gleb Radchenko, Radu Prodan, Marc Masana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of neighbors who all want to learn how to bake the perfect cake, but they are too shy to let anyone into their kitchens to see their secret recipes or ingredients. This is the problem of Federated Learning: everyone wants to learn together without sharing their private data.

However, traditional methods are like neighbors shouting their entire recipe (which is huge and slow to send) over the fence. Federated Distillation is a smarter approach: instead of sharing the whole recipe, neighbors just share a small "taste test" (a prediction) of how their cake turned out.

But there's a catch: if a neighbor tries to bake a cake using a recipe meant for a completely different type of dessert (like trying to bake a pizza using a cake recipe), the advice they give is bad. In the world of AI, this is called Non-IID data (data that doesn't look the same for everyone). If the neighbors don't filter out these bad "taste tests," the whole group learns the wrong lessons.

The Problem with Old Methods

Previous attempts to fix this involved using a super-complex, heavy-duty calculator (called a "statistical density ratio estimator") to figure out which taste tests were good and which were bad.

The Analogy: Imagine trying to sort a pile of mixed-up mail using a supercomputer that weighs every single envelope to see if it belongs in your house. It's accurate, but it's too slow and uses too much electricity for a regular person to do on their phone or a small edge device.
The Bottleneck: These old methods were so heavy that they couldn't run on small, battery-powered devices (like smart cameras or medical sensors). They also required a "manager" (the server) to double-check the mail, which slowed everything down.

The Solution: EdgeFD

The authors of this paper propose a new method called EdgeFD. Think of it as giving every neighbor a simple, lightweight KMeans filter.

Instead of using a super-computer to weigh every envelope, this new filter works like a sorting bin with a few marked zones:

The Map: Each neighbor quickly maps out where their own "good" data lives (like drawing a circle around their favorite ingredients).
The Check: When a new "taste test" (prediction) arrives from a neighbor, the filter just checks: "Is this taste test close to my circle?"
- Yes? It's a good match (In-Distribution). Keep it.
- No? It's too far away (Out-of-Distribution). Throw it away.
The Result: This process is incredibly fast and uses very little battery power. It's like using a simple ruler instead of a supercomputer.

How It Works in Practice

The paper describes a workflow where:

No "Teacher" Needed: Unlike other methods that need a pre-trained "master chef" (a teacher model) on the central server to guide them, EdgeFD lets the neighbors learn directly from each other's filtered advice.
Client-Side Filtering: The neighbors do the filtering themselves before sending anything to the central server. This means the server doesn't have to waste time sorting through bad data.
Robustness: Even if the neighbors have very different types of data (some only have chocolate cakes, others only have fruit tarts), EdgeFD successfully filters out the confusing advice and keeps the learning on track.

The Results

The researchers tested this method on three different "baking challenges" (datasets called MNIST, FashionMNIST, and CIFAR10):

Speed and Efficiency: The new "ruler" method (KMeans) was much faster and used less memory than the old "supercomputer" method. It scales perfectly for small devices.
Accuracy: In difficult scenarios where everyone's data was very different (Strong Non-IID), EdgeFD achieved 98.92% accuracy on simple tasks and 86.37% on complex ones. This was better than all the other top methods tested.
Minimal Data Sharing: The method works well even if neighbors only share a tiny slice (20%) of their private data to create the "proxy" (the taste test) for the group.

Real-World Scenarios Mentioned

The paper explicitly suggests three places where this could be used immediately:

Hospitals: Different hospitals could collaborate to train a medical AI without sending patient X-rays to a central server, only sharing filtered predictions.
Robot Fleets: A group of robots with cameras could learn together over slow wireless connections by sharing only the most relevant "snapshots" of what they see.
Car Safety: Cars could collaboratively learn to detect driver fatigue by sharing data about faces from different ethnicities and nationalities, without compromising individual privacy.

Summary

EdgeFD is like giving every small device a simple, efficient tool to sort its own data. It stops the group from learning from bad advice, runs fast on battery-powered devices, and allows everyone to learn together without needing a heavy central manager or sharing their entire private data. It makes collaborative AI possible even on the smallest, most resource-limited gadgets.

Technical Summary: EdgeFD – Efficient Client-Side Filtering for Non-IID Data in Federated Distillation

Problem Statement

Federated Distillation (FD) offers a promising alternative to traditional Federated Learning (FL) by exchanging model outputs (soft logits) on a shared proxy dataset rather than full model parameters, thereby reducing communication costs and enhancing privacy. However, existing feature-based FD methods face significant hurdles when deployed on resource-constrained edge devices, particularly under Non-Independent and Identically Distributed (Non-IID) data conditions.

Current state-of-the-art approaches, such as Selective-FD, rely on complex statistical density ratio estimators (e.g., KuLSIF-DRE) to filter proxy data. These methods require clients to identify in-distribution (ID) versus out-of-distribution (OOD) proxy data using computationally expensive matrix operations and often necessitate synthetic auxiliary data. This creates a high computational and memory overhead, limiting scalability on edge devices. Furthermore, server-side filtering of ambiguous knowledge introduces additional latency. The challenge is to develop a lightweight, robust FD method that effectively filters proxy data on the client side without relying on heavy statistical estimators or server-side intervention.

Methodology: EdgeFD and KMeans-DRE

The authors propose EdgeFD, a resource-efficient federated distillation framework designed to operate on heterogeneous edge devices with customized ML models. The core innovation is a two-stage client-side filtering strategy powered by a KMeans-based Density Ratio Estimator (KMeans-DRE).

1. Workflow Overview:

Initialization: Clients train a local KMeans model on their private data to compute centroid positions representing the local data distribution. They also generate a proxy dataset by sharing a small subset (e.g., 10–20%) of their private data with the server, which aggregates and redistributes a global proxy dataset to all clients.
Training Rounds:
- The server selects random indices from the global proxy dataset and broadcasts them to clients.
- Client-Side Filtering: Each client computes predictions for the selected proxy samples using its local model. Crucially, the client applies the KMeans-DRE filter:
  - It calculates the Euclidean distance between each proxy sample and the nearest centroid derived from the client's private data.
  - Samples with distances below a predefined threshold ( $T_{ID}$ ) are classified as In-Distribution (ID).
  - Samples exceeding the threshold are classified as Out-of-Distribution (OOD) and discarded.
- Only the predictions corresponding to ID samples are uploaded to the server.
- The server aggregates these filtered predictions to create an ensemble teacher model, which is then used by clients to update their local models via knowledge distillation.

2. Technical Distinctions:
Unlike statistical DREs that rely on complex kernel density estimation and auxiliary data, EdgeFD captures data distribution through centroid positions. This eliminates the need for synthetic auxiliary data and reduces the filtering process to simple distance calculations, significantly lowering computational complexity.

Key Contributions

Resource-Efficient KMeans-DRE: The paper introduces a novel density ratio estimator that replaces computationally intensive statistical methods with KMeans clustering. This approach reduces time complexity from exponential (in the case of KuLSIF-DRE with respect to sample size) to linear, making it suitable for edge devices.
Client-Side Filtering without Server Overhead: EdgeFD removes the need for server-side filtering of ambiguous knowledge. By effectively distinguishing ID and OOD proxy data locally, the method prevents negative knowledge transfer and reduces communication latency.
Robustness to Non-IID Data: The method is designed to handle strong and weak Non-IID scenarios where clients possess heterogeneous data distributions and customized model architectures, a common constraint in real-world edge deployments.
Minimal Proxy Data Requirement: The framework demonstrates effective knowledge distillation using only a small fraction (20%) of client private data as proxy data, reducing the privacy risk associated with data sharing.

Experimental Results

The authors evaluated EdgeFD on three benchmark datasets (MNIST, FashionMNIST, CIFAR-10) across Strong Non-IID, Weak Non-IID, and IID scenarios, comparing it against six state-of-the-art methods (including FedMD, FedED, Selective-FD, and data-free approaches).

Accuracy Performance: EdgeFD consistently outperformed existing methods, particularly in challenging Non-IID settings.
- Strong Non-IID: Achieved 98.92% on MNIST, 88.55% on FashionMNIST, and 82.57% on CIFAR-10, surpassing the previous best (Selective-FD) by significant margins (e.g., +4.24% on MNIST).
- Weak Non-IID: Achieved 98.88% (MNIST), 88.74% (FashionMNIST), and 84.88% (CIFAR-10).
- IID: Maintained competitive performance, achieving 99.08% on MNIST, 89.90% on FashionMNIST, and 86.37% on CIFAR-10.
Computational Efficiency: Comparative analysis of time and space complexity revealed that KMeans-DRE exhibits linear growth in both learning and estimation phases, whereas KuLSIF-DRE shows exponential growth in time complexity and higher memory consumption.
Threshold Sensitivity: Experiments indicated that while higher thresholds increase OOD inclusion (degrading accuracy), the method remains robust with optimized thresholds, even on complex datasets like CIFAR-10 when using feature extraction (ResNet-18).

Significance and Claims

The paper claims that EdgeFD addresses the critical scalability bottleneck in federated distillation for edge computing. By replacing heavy statistical estimators with a lightweight KMeans-based approach, the method enables the deployment of collaborative learning on devices with limited processing capabilities and bandwidth.

The authors emphasize that EdgeFD bridges the gap between centralized AI and decentralized applications by:

Enabling high-accuracy model training under strong data heterogeneity without requiring a pre-trained teacher model on the server.
Reducing the computational overhead to a level suitable for resource-constrained edge devices.
Providing a practical solution for scenarios involving heterogeneous models and strict bandwidth limits, such as medical consortia, robot fleets, and driver monitoring systems.

The work concludes that EdgeFD significantly enhances the real-world applicability of federated distillation, offering a scalable path toward privacy-preserving collaborative learning in heterogeneous edge environments.