🤖 machine learning

Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

This paper proposes ASCEND, an adaptive heterogeneous compression framework for Federated Knowledge Distillation that formulates strategy selection as a non-stationary multi-armed bandit problem to dynamically optimize communication efficiency and training time across diverse client resources while maintaining model accuracy.

Original authors: Chenwang Liu, Yijun Liu, Chang Liu, Xu Zhang, Pengchao Han

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenwang Liu, Yijun Liu, Chang Liu, Xu Zhang, Pengchao Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital world, our phones and sensors are constantly generating data, from health metrics to photos of our daily lives. A powerful idea called federated learning allows these devices to learn together to build smarter artificial intelligence without ever sending their private data to a central server. Instead, the devices train their own small models locally and only share the mathematical lessons they learned. However, this process faces a significant hurdle: the devices are not all the same. Some are powerful smartphones, while others are tiny, battery-powered sensors with limited memory and slow internet connections. Furthermore, the models they run often have different shapes and sizes to fit their specific hardware. When these diverse devices try to learn together, the constant exchange of information can clog the network, slowing everything down or causing the learning to fail.

Researchers have developed a method called federated knowledge distillation to help these mismatched devices collaborate. Instead of trying to force every device to use the exact same model, this approach lets them share the "essence" of what they are learning, allowing a small sensor to learn from a powerful phone even if their internal structures differ. Yet, a new problem emerged: even with this smarter sharing method, the devices still need to send large amounts of mathematical data back and forth, which consumes too much time and energy on the edge of the network. The traditional solution was to compress this data, but existing methods treated every device the same way, ignoring that a strategy that works for a fast computer might overwhelm a slow one.

To solve this, a team of researchers led by Chenwang Liu and his colleagues at Guangdong University of Technology and Chongqing University of Posts and Telecommunications proposed a new system where each device gets to choose its own compression strategy. They created a framework where devices can pick from a set of different ways to shrink their data, such as keeping only the most important numbers, picking numbers at random, or rotating through the data in a set pattern. The challenge was figuring out which method was best for which device at any given moment, since the best choice changes as the training progresses and network conditions shift.

The researchers treated this selection process like a game of chance where a player must decide which lever to pull to get the best reward. In their system, each compression method is a lever, and the reward is a score that balances how much the model improved against how long the process took. They developed an algorithm named ASCEND that allows each device to learn from its own experience. At first, the devices try different methods to see what works. Over time, they start to favor the methods that give them the best results for their specific hardware and current network speed. If a device notices that its learning is suddenly going backward or becoming unstable, the system has a safety mechanism that instantly reverts to a stable, conservative setting to prevent errors.

The team tested this approach on a real-world platform using ten Raspberry Pi devices acting as edge clients, as well as in simulations with standard image datasets like MNIST and CIFAR-10. They found that their adaptive system consistently outperformed methods that forced all devices to use the same compression rule. In scenarios where devices had different computing power or were connected to networks with varying speeds, ASCEND successfully adapted. For example, on smaller, simpler models, the devices tended to choose a method that kept the most significant data points, while on larger, more complex models, they often switched to a method that was faster to compute, even if it was slightly less precise. The results showed that this flexible approach reduced the total time needed to train the models and lowered the communication burden without sacrificing the final accuracy of the artificial intelligence.

The study confirms that there is no single "best" way to compress data for all devices in a federated network. Instead, the optimal strategy depends on the specific mix of the device's hardware, the size of the model it is running, and the current state of the training. By allowing each participant to dynamically select its own path, the system achieves a balance between speed and intelligence that rigid, one-size-fits-all solutions cannot match. This work suggests that the future of distributed learning lies not in forcing uniformity, but in building systems smart enough to adapt to the unique constraints of every device involved.

Technical Summary: Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

Problem Statement
Federated Learning (FL) enables privacy-preserving distributed training but faces significant challenges due to heterogeneous model architectures and limited communication resources at the network edge. Federated Knowledge Distillation (FedKD) addresses model heterogeneity by combining prototype-wise parameter aggregation with logit-based knowledge transfer across different architectures. However, transmitting full gradients in FedKD introduces substantial communication overhead. Existing gradient compression methods (e.g., Top-K, Random-K, Periodic-K) typically apply a uniform strategy across all clients and training rounds. This approach neglects the diverse computational capabilities, memory constraints, and model characteristics of individual clients. For instance, computationally intensive strategies like Top-K may overload resource-constrained devices, while lightweight methods like Random-K may degrade performance on specific model architectures. The core problem is how to select compression strategies for clients with diverse model architectures and resource capacities to effectively balance training efficiency and learning performance.

Methodology
The authors propose a heterogeneous compression framework where each client independently selects a compression strategy from a candidate set $\mathcal{S} = \{\text{Top-K, Random-K, Periodic-K}\}$ based on its local model characteristics and resource capacities.

System Model: The framework operates within a standard FedKD setup. Clients perform local training, accumulate gradient residuals to compensate for compression loss, and transmit compressed gradients to the server. The server reconstructs gradients, performs prototype-wise parameter averaging, and conducts global knowledge distillation using a public dataset to align heterogeneous models.
Strategy Selection as MAB: The selection of the optimal compression strategy is formulated as a non-stationary stochastic Multi-Armed Bandit (MAB) problem. Each compression strategy corresponds to an "arm." The reward is defined as an efficiency-aware performance gain, calculated by jointly considering local optimization improvement (loss reduction), global knowledge alignment (KL divergence reduction), and execution time (computation and communication overhead).
ASCEND Algorithm: To solve the MAB problem, the authors develop the Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND). Key components include:
- EMA-enhanced $\epsilon$ -greedy Policy: Clients use an Exponential Moving Average (EMA) to update their utility estimates of each strategy, allowing them to adapt to non-stationary reward distributions caused by dynamic training states. An $\epsilon$ -greedy policy balances exploration (trying new strategies) and exploitation (using the best-known strategy).
- Warm-up Phase: A fixed number of initial trials are conducted for each strategy to establish baseline utility estimates.
- Stability Safeguard: A rollback mechanism is implemented. If a sudden surge in global loss is detected (exceeding a threshold $\gamma$ ), the system reverts to the previous round's model parameters and temporarily enforces a unified Top-K strategy to ensure stability.

Key Contributions

Heterogeneous Compression Framework: The paper introduces a framework enabling clients to adaptively select customized compression strategies under heterogeneous model and resource conditions. The authors provide a convergence analysis for this framework under non-convex loss functions, yielding an $O(1/\sqrt{T})$ convergence rate.
Optimization Formulation: The strategy selection problem is formulated to maximize cumulative efficiency-aware performance, explicitly capturing the trade-off between compression cost, communication cost, and learning performance.
ASCEND Algorithm with Theoretical Guarantee: The authors model the selection process as a non-stationary stochastic MAB. Under a piecewise-stationary approximation, they derive a sublinear regret bound of $O(\sqrt{MT})$ , where $M$ is the number of stationary segments and $T$ is the number of rounds, providing a theoretical guarantee for long-term performance.
Experimental Validation: Extensive experiments on real-world platforms (Raspberry Pi) and simulations using MNIST and CIFAR-10 datasets demonstrate the algorithm's effectiveness.

Results
Experimental results indicate that ASCEND effectively adapts to diverse model and resource settings:

Performance vs. Time: ASCEND consistently achieves competitive or superior model accuracy compared to uniform compression baselines (Top-K, Random-K, Periodic-K) and an EXP3-based adaptive method, while significantly reducing training time and communication overhead.
Adaptivity: The algorithm dynamically shifts strategy preferences based on conditions. For example, on smaller models (LeNet5Half) or aggressive compression levels, clients predominantly select Top-K. Conversely, for larger models (LeNet5, ResNet18) or larger communication budgets, clients shift toward Random-K to reduce selection overhead.
Robustness: The method maintains robust performance across varying degrees of data heterogeneity (controlled by Dirichlet parameter $\alpha$ ) and different communication bandwidths (5 Mbps to 500 Mbps).
Real-World Validation: Deployed on a real-world platform with 10 Raspberry Pi devices, ASCEND demonstrated faster convergence in early training stages and better overall efficiency compared to fixed strategies.

Significance and Claims
The paper claims that enforcing a uniform compression strategy in FedKD leads to suboptimal performance due to the inherent heterogeneity of edge devices and models. By framing strategy selection as an online learning problem, ASCEND allows the system to autonomously identify the most effective compression strategy for each client at each training round. The authors assert that this approach achieves a favorable balance between model accuracy and training latency without compromising the theoretical convergence guarantees of the underlying federated optimization. The work highlights that adaptive selection among multiple heterogeneous strategies is an open problem that, when solved, significantly enhances the practicality of FedKD in resource-constrained environments. Future work is suggested to investigate the joint optimization of strategy selection and dynamic adjustment of the compression level (retained gradient budget).

More like this