🤖 machine learning

Robust Federated Inference

This paper addresses the critical vulnerability of federated inference to attacks by formalizing the problem, analyzing the limitations of averaging-based aggregators, and proposing a novel DeepSet-based framework that combines adversarial training with test-time robust aggregation to significantly improve accuracy over existing methods.

Original authors: Akash Dhasade, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Maxime Jacovella, Anne-Marie Kermarrec, Rafael Pinot

Published 2026-04-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Akash Dhasade, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Maxime Jacovella, Anne-Marie Kermarrec, Rafael Pinot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

This paper primarily addresses the question of "how to safely aggregate opinions from multiple parties."

Imagine you are a principal (server) with a group of students (clients/models). These students are learning locally, each possessing their own knowledge (private data), but their proficiency levels vary, and some may even be troublemakers (malicious attackers) mixed in.

Now, you have a question to ask everyone, hoping to get the most accurate answer. This is "Federated Inference": the principal does not collect the students' homework (data) but asks them directly for their answers and then aggregates everyone's responses.

The core of this paper is: When troublemakers intentionally write wrong answers, how should the principal aggregate them intelligently to avoid being misled?

Below is a breakdown of the paper's content in plain, colloquial language:

1. Current Status: How was this done before? (Simple Averaging vs. Being Misled)

Previous Approach (Simple Averaging):
The principal adds up the answers of 10 students and divides by 10.
- Problem: If 3 troublemakers intentionally write "The answer is 999," while good students write "The answer is 1," the average might become "300," which is completely wrong.
Previous Improvement (Robust Averaging):
The principal is smart enough to know to remove the highest and lowest scores (e.g., removing the extreme answers of 3 troublemakers) before calculating the average. This is like the "trimmed mean" method.
- Problem: The paper finds that even after removing the troublemakers, if the good students' answers themselves show some "disagreement" (e.g., some think it's 1, others think it's 2), or if the troublemakers are cunning and tweak their answers to be very close to the correct one but not quite right, simple "trimmed averaging" can still lead to the wrong choice. It's like removing the most outrageous scores, but among the remaining scores, the correct answer and the wrong answer are so close that the principal is still likely to make a mistake.

2. Core Finding: Why is simple "trimming" insufficient?

The paper's authors conducted a mathematical analysis and found that the principal's errors depend mainly on three factors:

The proportion of troublemakers: The more bad students, the harder it is.
Disagreement among good students: If the good students themselves are in fierce disagreement (large variance in answers), it is hard for the principal to judge who is right.
The "confidence" of the correct answer: If the correct answer is significantly stronger than others (e.g., the correct answer has a 90% probability while others have only 5%), it is less likely to be wrong; but if the correct answer is only slightly better than the second best (e.g., 40% vs. 39%), it is very easy to be misled by even minor interference.

Conclusion: Simply kicking out the troublemakers (trimmed averaging) is not enough, because the good students' answers themselves might be ambiguous, or the troublemakers might be playing a "fine-tuning" tactic.

3. Innovative Solution: DeepSet + Adversarial Training (Equipping the Principal with a "Super Brain")

To solve this problem, the authors propose a "combination strategy", like equipping the principal with an AI assistant system:

Step 1: Equip the Principal with a "Super Brain" (DeepSet Model)

The principal no longer relies on simple addition, subtraction, multiplication, or division but uses a neural network (DeepSet).

Analogy: This brain is very smart; it does not care about the order in which students answer (the result is the same regardless of who is asked first or last). It can learn patterns from complex response patterns.
Challenge: How to train this brain? If the identity of the troublemakers changes every time during training (today it might be Zhang San, tomorrow Li Si), the training volume would be too large to compute.
Clever Trick: The authors discovered that by randomly sampling a small subset of troublemaker combinations for training, the brain can learn to recognize "features of troublemakers" without needing to exhaustively enumerate all possibilities. This is like "responding to万变 (ten thousand changes) with one unchanging principle."

Step 2: Dual Insurance (Adversarial Training + Trimming During Inference)

The authors combined two techniques:

Adversarial Training: When training the principal's brain, various "troublemakers" are deliberately generated to attack it, teaching it to remain clear-headed amidst chaos. This is like "leveling up in simulated battles."
Trimming During Inference (Robust Averaging): When the actual exam (inference) takes place, the principal's brain passes the final answer through a "trimmed averaging" filter before outputting it, filtering out the most outrageous outliers one more time.

Analogy: This is like a "referee specially trained by a special forces unit" (adversarial training), who, before blowing the whistle, will "first eliminate a few obviously cheating contestants" (trimmed averaging).

4. How Effective Is It? (A Resounding Victory)

The authors tested this method on various exams (datasets: CIFAR-10, CIFAR-100, AG-News):

Opponents: Traditional "trimmed averaging," simple "median voting," and other previous defense methods.
Attackers: Various high-intelligence troublemakers (including a newly invented "strongest reverse attack SIA").
Results: The authors' "Super Brain + Dual Insurance" combination performed best in all tests.
- In extremely difficult scenarios, the accuracy was 4.7% to 22.2% higher than the second-best method.
- It's like in a noisy room full of interference, only this principal can hear the correct answer clearly, while everyone else is guessing wildly.

5. Summary: What Does This Paper Actually Say?

Simply put, this paper tells us:
In scenarios of multi-party collaboration (Federated Learning/Inference), simple "minority obeys majority" or "removing extreme values" is no longer sufficient. Facing cunning interference, we need:

A smarter aggregator (using neural networks instead of simple mathematical averaging).
Stricter training (simulating various attacks during training to strengthen the model).
Dual defense (leveling up during training, filtering again during usage).

One-sentence analogy:
Previously, we thought that kicking out the troublemakers and averaging the remaining good students' answers would yield a high score; now the authors tell us that good students might also disagree, and troublemakers are very cunning. Therefore, we need a super referee trained through "devilish training" that can understand the complex opinions of good students, instantly recognize the disguises of troublemakers, and make the most accurate judgment amidst chaos.

Technical Summary: Robust Federated Inference

Problem Formulation

The paper addresses the critical gap in robustness within Federated Inference, a paradigm where a central server aggregates predictions from multiple client-held models (e.g., via one-shot federated learning, edge ensembles, or federated ensembles) without accessing local data or model weights. While existing literature focuses on the accuracy of such ensembles, it largely neglects their vulnerability to Byzantine failures and poisoned outputs.

The authors formalize the problem of Robust Federated Inference as follows:

Setup: A system of $n$ clients, each possessing a local probit-valued classifier $h_i$ mapping inputs to a probability simplex $\Delta_K$ .
Threat Model: Up to $f < n/2$ clients (with hidden identities) may return arbitrarily corrupted probit vectors at inference time.
Goal: Design an aggregation scheme $\psi$ that minimizes the robust federated inference risk ( $R_{adv}$ ), defined as the worst-case prediction error over all possible corruption patterns of size $f$ , while maintaining accuracy on the global data distribution.

The paper distinguishes between two types of aggregators:

Linear (Averaging-based): The server computes a robust average of client probits (e.g., Coordinate-wise Trimmed Mean - CWTM) before taking the argmax.
Non-linear (Learned): The server uses a trained neural network aggregator to combine client probits, which has shown superior performance in uncorrupted settings but lacks robustness guarantees.

Methodology

1. Analysis of Averaging-Based Aggregators

The authors first analyze robust averaging schemes (like CWTM) as a baseline. They demonstrate that while robust averaging ensures the output is close to the average of honest probits in $\ell_2$ -norm, this proximity does not guarantee correct classification due to the non-continuity of the argmax operator.

Theoretical Insight: They derive a robustness certificate showing that the error of the aggregator depends on three factors:
1. The fraction of corruptions ( $f/n$ ).
2. The dissimilarity between honest client outputs ( $\sigma_x$ ).
3. The margin between the top two predicted classes.
Conclusion: Robust averaging is insufficient when the margin is small or client models are highly dissimilar, even if the aggregation is robust in a statistical sense.

2. Robust Inference as an Adversarial ML Problem

For non-linear aggregators, the authors cast the problem as an adversarial learning problem over the space of probit vectors.

Challenge: Standard adversarial training is computationally intractable because the set of possible adversarial subsets (choosing $f$ malicious clients out of $n$ ) has a high cardinality ( $\binom{n}{f} \times f!$ ).
Solution (DeepSet): The authors propose using a DeepSet architecture (Zaheer et al., 2017), which is permutation-invariant. This property allows the search space to be reduced from permutations to combinations ( $\binom{n}{f}$ ), making the problem more tractable.
Approximation: Instead of enumerating all adversarial subsets, they sample a small number $N$ of random adversary configurations during training ( $N \ll \binom{n}{f}$ ) and apply multi-step Fast Gradient Sign Method (FGSM) to perturb the probits of the selected adversaries.

3. The DeepSet-TM Composition

To further enhance robustness, the authors introduce a novel composition of Adversarial Training and Robust Averaging at inference time:

Architecture: A DeepSet model where the input probits are first processed by a robust averaging operator (CWTM) before being fed into the neural network aggregator.
Training Strategy: The model is trained adversarially on the raw probits (without CWTM) to learn robust features.
Inference Strategy: At test time, the robust averaging (CWTM) is applied to the client inputs before the DeepSet inference. This decouples the training cost from the inference-time robustness mechanism, preventing the training process from becoming prohibitively expensive.

Key Contributions

Formalization: The first formal definition and robustness analysis of the Federated Inference problem under Byzantine corruptions.
Theoretical Bounds: Derivation of formal robustness certificates for both linear (averaging) and non-linear (DeepSet) aggregators, linking error rates to corruption fractions, model dissimilarity, and prediction margins.
Algorithmic Innovation: The proposal of DeepSet-TM, a method combining permutation-invariant neural networks with adversarial training and test-time robust averaging. This approach addresses the computational intractability of training against arbitrary subsets of adversaries.
New Attack: Introduction of the Strongest Inverted Attack (SIA), a new white-box and black-box attack that exploits the second-most probable class to flip aggregation decisions, used to rigorously evaluate defenses.

Experimental Results

The authors evaluate their approach on CIFAR-10, CIFAR-100, and AG-News using diverse model families (ResNet-8, ViT-B/32, DistilBERT) and varying levels of data heterogeneity ( $\alpha \in \{0.3, 0.5, 0.8\}$ ).

Performance: DeepSet-TM consistently outperforms existing robust aggregation baselines (CWTM, CWMed, Geometric Median) and state-of-the-art defenses (COPUR, Manifold Projection).
Improvement: The method achieves a 4.7% to 22.2% improvement in worst-case accuracy points across diverse benchmarks and attack types.
Robustness to Strong Attacks: Under the proposed SIA and PGD-cw attacks, baseline methods often suffer accuracy drops of 35–40 points, whereas DeepSet-TM limits drops to 20–30 points. On AG-News, DeepSet-TM retains 83.2% accuracy under PGD-cw compared to ~53% for baselines.
Scalability: The approach scales effectively with the number of clients ( $n=10, 17, 25$ ) and adversaries ( $f=3, 4, 5$ ), maintaining superior performance as the system grows.
Efficiency: While adversarial training incurs a one-time cost (approx. 1.5 hours), the inference latency remains low (approx. 0.3ms per batch), making it suitable for cross-silo settings.

Significance and Claims

The paper claims that Robust Federated Inference is a distinct and critical problem that cannot be solved by simply applying existing Byzantine-robust training techniques or purification methods (like COPUR), which often fail in the probit space or under strong attacks.

The significance of the work lies in:

Bridging the Gap: It provides the first comprehensive framework for securing federated inference, a growing paradigm for leveraging proprietary models (including LLMs) without centralizing data.
Novel Composition: By combining adversarial training (from the adversarial ML literature) with robust averaging (from the robust statistics/Byzantine ML literature), the authors demonstrate that these two fields can be synergistically combined to achieve state-of-the-art robustness.
Practicality: The proposed DeepSet-TM offers a practical solution that balances high robustness against sophisticated attacks with acceptable computational overhead, specifically tailored for cross-silo scenarios where clients are static but inference requests are dynamic.

The authors conclude that their method sets a new state-of-the-art for federated inference, effectively preventing technological advantages from becoming significant vulnerabilities in the presence of malicious actors.