🤖 machine learning

Towards Interpretable Federated Learning

This paper presents the first comprehensive survey on Interpretable Federated Learning (IFL), introducing a novel taxonomy and analyzing existing approaches, evaluation metrics, and future directions to address the critical need for balancing performance, privacy, and interpretability in collaborative machine learning.

Original authors: Anran Li, Rui Liu, Ming Hu, Yuanyuan Chen, Shipeng Wang, Lizhen Cui, Han Yu

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anran Li, Rui Liu, Ming Hu, Yuanyuan Chen, Shipeng Wang, Lizhen Cui, Han Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of neighbors who all want to build a super-smart weather prediction machine. However, they live in different houses and are very protective of their private journals (their local data). They don't want to share their journals with each other or with a central builder.

Federated Learning (FL) is the solution: everyone keeps their journal at home, but they send only the "lessons learned" (model updates) to a central builder, who combines them to make a better global machine.

The problem? The resulting machine is a "black box." It's so complex that no one knows why it predicts rain or sunshine. In high-stakes situations like banking or healthcare, people need to know the "why" to trust it. This is where Interpretable Federated Learning (IFL) comes in.

This paper is essentially a guidebook and map for researchers trying to make these secret-keeping machines understandable. Here is a breakdown of what the paper covers, using simple analogies:

1. The Goal: Making the Black Box Transparent

The authors argue that for this technology to be trusted (especially in finance and health), we need to explain three things:

Why did the model make this prediction? (e.g., "It flagged this transaction as fraud because of these specific patterns.")
Why did we pick this person's data? (e.g., "We selected this bank's data because their records were high quality.")
Who deserves a reward? (e.g., "This client contributed the most to the final model, so they get the biggest bonus.")

2. The Map: A New Taxonomy (Classification System)

The paper creates a unique "map" to organize all the different ways researchers are trying to solve this. They break the process down into four main stages, like stops on a train journey:

Stop 1: Choosing the Passengers (Client Selection)
- The Problem: Not all neighbors have good journals. Some might have messy or fake data.
- The Solution: The paper reviews methods to figure out which neighbors are "good" without seeing their actual journals.
- Analogy: Imagine a teacher trying to pick the best students for a group project without looking at their report cards. They might look at how much the student's answers "wiggle" away from the group average (Model Deviation) or how much the student's answers improve the final grade (Influence).
Stop 2: Choosing the Pages (Sample Selection)
- The Problem: Even a good neighbor might have some useless or confusing pages in their journal.
- The Solution: Methods to identify which specific data points are helpful and which are noise.
- Analogy: It's like a chef tasting a soup and deciding, "This pinch of salt is great, but this rock is bad." The paper looks at techniques to filter out the "rocks" (noisy data) while keeping the "salt" (important data).
Stop 3: Choosing the Ingredients (Feature Selection)
- The Problem: Sometimes the data includes irrelevant details (like the color of the car instead of the speed).
- The Solution: Figuring out which specific variables actually matter for the prediction.
- Analogy: If you are baking a cake, you need to know that flour and eggs are crucial, but the color of the mixing bowl doesn't matter. The paper reviews ways to identify the "flour and eggs" without the baker revealing their secret recipe.
Stop 4: Building the Machine (Model Optimization & Contribution)
- The Problem: How do we build the model so it's naturally easy to understand, and how do we fairly pay everyone who helped?
- The Solution:
  - Inherently Interpretable Models: Using simpler structures (like decision trees) that are easy to read, rather than complex deep neural networks.
  - Fair Rewards: Using math (like Shapley Values) to calculate exactly how much each neighbor contributed to the final result, ensuring everyone gets a fair share of the reward.

3. The Security Guard: Privacy Protection

A major theme is that you can't explain the machine without accidentally peeking at the private journals.

The Threat: Some neighbors might be "semi-honest" (they follow the rules but try to guess others' secrets) or "malicious" (they try to break the system).
The Shield: The paper reviews tools like Differential Privacy (adding static noise to hide details), Homomorphic Encryption (doing math on locked boxes without opening them), and Secure Multiparty Computation (a group of people solving a puzzle together without anyone seeing the others' pieces).

4. How Do We Know It Works? (Evaluation)

The paper explains how to test these methods:

Faithfulness: If we remove the "important" data the model identified, does the model's performance crash? If yes, the explanation was truthful.
Efficiency: Did the explanation cost too much time or battery power? (Remember, these devices are often small, like phones or sensors).

5. Where Do We Go From Here?

The authors point out that current methods are still a bit rough. They suggest future research should focus on:

Mimicry: Taking a complex, hard-to-understand model and training a simple, easy-to-read model to act exactly like it.
Noise Handling: Better ways to tell the difference between "hard" data (that needs to be learned) and "bad" data (that should be ignored).
Big Language Models: Adapting these ideas for massive AI chatbots, which is currently very difficult.
Real-World Security: Moving beyond "polite" participants to handle truly malicious attackers.

In summary: This paper is the first comprehensive "user manual" for making Federated Learning transparent. It organizes the chaos of current research, explains how to keep secrets while explaining decisions, and charts a course for making these systems trustworthy enough for the real world.

Technical Summary: Towards Interpretable Federated Learning

Problem Statement

Federated Learning (FL) enables collaborative model training across multiple data owners without exposing local private data. However, the widespread adoption of FL, particularly in mission-critical domains like finance and healthcare, is hindered by a lack of interpretability. Modern FL models often rely on complex, non-linear architectures (e.g., Deep Neural Networks) with millions of parameters, making their internal mechanisms and decision-making processes opaque to stakeholders. This opacity erodes trust and complicates compliance with regulations such as the GDPR, which mandates "meaningful explanations" for automated decisions.

While interpretable AI exists for centralized learning, applying these concepts to FL is significantly more challenging due to:

Data Invisibility: Local data remains hidden from the server and other clients.
Resource Constraints: Clients often operate on devices with limited computation and communication power.
Interdisciplinary Complexity: Solutions require integrating machine learning, optimization, cryptography, and human factors.

Currently, there is no comprehensive survey addressing these specific challenges in the context of Interpretable Federated Learning (IFL).

Methodology: Proposed Taxonomy

The paper proposes the first comprehensive survey and a unique taxonomy for IFL. The framework is structured around the FL training process, stakeholder analysis, and privacy protection requirements.

1. Definition and Scope

The authors extend the definition of interpretability beyond client selection to include:

Definition: The ability of any party (server or client) to explain or present behaviors related to models and data in understandable terms to other parties under an FL protocol.
Stakeholders:
- Direct: FL Server (needs to understand client data quality, model behavior, and reward allocation) and FL Clients (need to understand selection criteria and reward rationale).
- Indirect: Researchers, regulators, and policymakers (need debugging insights and regulatory compliance verification).
Privacy Targets:
- To Server: Protect raw data and data distributions.
- To Clients (in VFL): Protect labels from exposure to other clients.
Threat Models: Semi-honest participants (follow protocol but infer data) and Malicious participants (compromise models).
Privacy Techniques: Differential Privacy (DP), Homomorphic Encryption (HE), and Secure Multi-Party Computation (MPC).

2. Taxonomy of IFL Approaches

The survey categorizes existing works into five stages of the FL lifecycle:

A. Interpretable Client Selection

Methods to identify high-quality clients for aggregation.

Importance-based:
- Model Deviation: Uses deviations of local updates from global updates (e.g., Krum, Bulyan) to detect noisy or malicious clients.
- Loss-based: Quantifies importance via local loss or loss-based utilities.
- Gradient Norm-based: Uses gradient norms (or upper bounds) for more accurate measurement than loss alone.
Influence-based:
- Retraining: Perturbs/removes clients to measure performance impact (computationally expensive).
- Influence Functions: Uses second-order optimization (Hessian) to approximate influence without full retraining. Recent works utilize Hessian Vector Products (HVP) to reduce complexity from $O(p^3n)$ to $O(np)$.

B. Interpretable Sample Selection

Methods to select specific data samples within a client's local dataset.

Logic-based: Uses logical reasoning or rule learning to aggregate updates or select prompts (e.g., for LLMs).
Importance-based: Selects samples based on loss thresholds or gradient norm upper bounds, often filtering outliers.
Influence-based: Approximates the impact of individual samples on model predictions using HVP or sign-based influence values to avoid high computational costs.

C. Interpretable Feature Selection

Methods to identify relevant features and discard noise/redundancy.

Model-Agnostic: Treats the model as a black box.
- Supervised: Uses statistical measures (Mutual Information, Gini impurity, F-statistics) via MPC or distributed aggregation.
- Unsupervised: Uses methods like One-Class SVM to detect outlier features in non-IID settings.
Model-Specific: Treats the model as a white box.
- Utilizes constraints (L1, L2, L0) and architectures (Autoencoders, Stochastic Dual-Gates) to select features directly during training.

D. Interpretable Model Optimization

Inherently Interpretable Models: Replaces complex DNNs with self-explanatory models like Decision Trees (e.g., FedForest) or intermediate representations.
Interpretable Robust Aggregation:
- Semi-honest: Uses incentive mechanisms (e.g., Stackelberg games) to reward high-quality updates.
- Malicious: Assigns lower weights to outlier clients based on influence values to mitigate Byzantine attacks.

E. Interpretable Contribution Evaluation

Methods to quantify the value of clients or features for fair reward allocation.

Client Contribution:
- Utility Games: Measures marginal utility changes.
- Shapley Value (SV): The gold standard for fair allocation. Works focus on accelerating SV calculation (e.g., gradient-based, local embedding) to avoid exponential complexity. Secure SV calculation is explored for malicious settings using HE or blockchain.
Feature Contribution:
- SV-based: Estimates feature importance (often requiring assumptions that violate privacy in VFL).
- Attention-based: Uses hierarchical attention mechanisms to visualize feature correlations.
- Activation-based: Extracts input features that strongly activate neurons (e.g., Flames2Graph for time series).

Key Contributions

First Comprehensive Survey: The paper provides the first dedicated survey on IFL, bridging the gap between FL and interpretability.
Novel Taxonomy: It introduces a structured taxonomy covering client/sample/feature selection, model optimization, and contribution evaluation, while explicitly mapping these to stakeholder needs, threat models, and privacy techniques.
Systematic Analysis: The authors analyze representative approaches, their underlying mechanisms (e.g., HVP approximations, MPC protocols), and their trade-offs regarding accuracy, efficiency, and privacy.
Evaluation Metrics Review: The paper categorizes evaluation metrics into Effectiveness (Post-interpretation performance, Faithfulness via leave-some-out retraining) and Efficiency (Computation and communication costs).
Future Directions: The authors outline specific research gaps, including:
- Interpretable Model Approximation: Using mimic learning to approximate complex FL models with interpretable ones without significant accuracy loss.
- Hard Sample-Aware Noise-Robust IFL: Developing methods that distinguish between noisy samples and positively influential hard samples without prior knowledge of thresholds.
- Interpretability for LLMs: Adapting IFL for Federated Large Language Models (FedLLM) where inputs are combinatorial objects.
- Complex Threat Models: Moving beyond semi-honest assumptions to handle malicious/colluding actors.
- Privacy-Efficiency Trade-offs: Reducing the high costs of privacy-preserving techniques (DP, HE, MPC) for resource-constrained devices.
- Standardized Evaluation: Developing metrics that measure the privacy exposure level relative to the achieved interpretability.

Results and Claims

The paper does not present new experimental results or a novel algorithm. Instead, its "results" are the synthesis of the current state-of-the-art.

Findings: Current IFL research is fragmented, often focusing on single stages (e.g., only client selection) or specific threat models (semi-honest). Many methods rely on expensive retraining or strong assumptions (e.g., server access to test data) that may not hold in practice.
Significance: The paper claims that by providing this taxonomy and analysis, it lowers the barrier to entry for new researchers and guides the development of versatile IFL techniques. It emphasizes that achieving interpretability is crucial for building trust, ensuring regulatory compliance, and enabling fair incentive mechanisms in FL ecosystems.

The authors maintain a modest tone, acknowledging that while IFL is an emerging field, significant challenges remain in balancing interpretability, privacy, and efficiency, particularly in complex, real-world scenarios involving malicious actors and resource-constrained devices.