🤖 machine learning

FAIRVAR: Fair Federated Learning via Variance Regularization

This paper introduces FairGrad and FairGrad*, two gradient-variance-regularized methods that effectively minimize performance disparities across clients in heterogeneous federated learning settings while maintaining competitive global model accuracy.

Original authors: Zahra Kharaghani, Ali Dadras, Tommy Löfstedt

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zahra Kharaghani, Ali Dadras, Tommy Löfstedt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of neighbors trying to build a single, shared recipe book. Each neighbor brings their own collection of ingredients and cooking notes (their local data) to contribute to this global book. The goal is to create one "Master Recipe" that works well for everyone.

This is the basic idea of Federated Learning (FL). However, there's a problem: some neighbors have huge, diverse pantries (lots of data), while others have very small, specific collections (little data). Some neighbors only cook spicy food, while others only bake sweets.

If the group just averages everyone's notes blindly, the final Master Recipe might be great for the neighbors with big pantries but terrible for the ones with small or unique collections. The "spicy" neighbors might end up with a recipe that tastes like bland oatmeal, and the "bakers" might get a recipe that burns their oven. This is unfairness: the global model works well for some but fails for others.

The Paper's Solution: "FairVar"

The authors of this paper, Zahra Kharaghani, Ali Dadras, and Tommy Löfstedt, propose a new way to mix these recipes called FairVar. Their goal is Performance Equitable Fairness: making sure the final recipe works equally well for every neighbor, not just on average.

To do this, they introduce a concept called Variance Regularization. Think of "variance" as the measure of how much the neighbors' results differ from each other. The paper suggests two main ways to fix the recipe book:

1. The "Score Variance" Approach (FairLoss)

Imagine the neighbors are grading the Master Recipe after trying it.

The Problem: If the spicy neighbors give it a 2/10 and the bakers give it a 9/10, the average is a decent 5.5/10. But the spicy neighbors are unhappy.
The Fix: The FairLoss method adds a "penalty" to the group's goal. If the scores are too spread out (high variance), the group is told, "Hey, you need to adjust the recipe to make the low scores go up, even if it means the high scores drop a tiny bit." This forces the recipe to find a middle ground that satisfies everyone.

2. The "Direction Variance" Approach (FairGrad) — The New Star

This is the paper's main innovation. Instead of just looking at the final scores, they look at the direction the neighbors are trying to push the recipe.

The Metaphor: Imagine the neighbors are all pulling a giant rope to move a heavy cart (the model).
- The "bakers" are pulling hard to the North.
- The "spicy cooks" are pulling hard to the East.
- The "average" recipe just pulls North-East, which might not move the cart very far for anyone.
The Fix: The FairGrad method looks at how much the neighbors are pulling in different directions. If the "East" pullers are being ignored because the "North" pullers are stronger, FairGrad adds a "tension" to the rope. It says, "We need to align our pulls more closely." It penalizes the group if the directions are too scattered. By forcing the neighbors to agree more on how to change the recipe, the final result ends up being much fairer for the minority groups.

How They Tested It

The researchers tested these ideas on a "digital kitchen" with four famous datasets (MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet). They simulated different levels of chaos:

Low Chaos: Everyone has similar ingredients (Homogeneous data).
High Chaos: Everyone has totally different ingredients (Heterogeneous data).

They compared their new methods (FairGrad and FairGrad*) against old methods like FedAvg (the standard "average everything" approach) and q-FFL (a previous fairness attempt).

What They Found

When everyone is similar: If all neighbors have similar pantries, the fancy new methods don't really change anything. The standard "average" approach works fine. The new methods don't break anything, but they don't need to do much extra work.
When everyone is different (The Real Test): This is where the magic happened.
- The standard methods (FedAvg) produced a recipe that was good on average but had huge gaps: some neighbors loved it, others hated it.
- The FairGrad methods significantly reduced these gaps. They made the "worst-case" neighbor much happier without ruining the experience for the others.
- In many tests, FairGrad didn't just fix fairness; it actually made the overall recipe better too. It was like finding a secret ingredient that made the dish taste good for everyone, not just the majority.

The Two Versions of FairGrad

The paper offers two ways to run this "FairGrad" method:

FairGrad (Approximate): This is the "lazy" but efficient version. Neighbors look at what the group did last time to decide what to do now. It saves time and communication (less talking between neighbors).
FairGrad* (Exact): This is the "precise" version. Neighbors look at what the group is doing right now. It requires more talking and is slower, but it's more accurate. The paper suggests using this when the task is very hard or the neighbors are very different.

The Bottom Line

The paper concludes that by adding a "variance penalty" (specifically looking at how much the neighbors' directions differ), we can build machine learning models that are fairer. Instead of letting the loudest voices (the biggest data owners) dictate the outcome, the model is nudged to listen to the quieter voices, ensuring that the final result works well for everyone in the room, regardless of how much data they brought to the table.

Technical Summary: FAIRVAR - Fair Federated Learning via Variance Regularization

Problem Statement

Federated Learning (FL) enables collaborative model training across multiple clients without sharing raw data. However, data heterogeneity often leads to significant performance disparities among clients, where the global model performs well on average but poorly for specific clients with distinct data distributions or smaller sample sizes. This lack of performance equitable fairness—defined as minimizing performance disparities across clients while maintaining high overall model performance—is a critical challenge. Existing approaches to address this, such as loss-variance regularization and $q$ -Fair Federated Learning ( $q$ -FFL), often lack a solid theoretical foundation regarding their behavior under heterogeneity, and it remains unclear which methods are most effective in practical settings.

Methodology

The paper proposes a new approach to performance equitable fairness based on gradient-variance regularization. The authors analyze and compare several existing methods and introduce new algorithms derived from two primary objective functions:

Loss-Variance Regularization: Building on prior work, this approach penalizes the variance of client losses:
$L_\lambda(x) = F(x) + \frac{\lambda}{2n} \sum_{i=1}^n (f_i(x) - F(x))^2$
where $F(x)$ is the global average loss and $f_i(x)$ is the local loss for client $i$ .
Gradient-Variance Regularization (Proposed): The authors introduce a novel objective that minimizes the variance of client gradients to promote fairness:
$J_\gamma(x) = F(x) + \frac{\gamma}{2n} \sum_{i=1}^n \|\nabla f_i(x) - \nabla F(x)\|^2$
Here, $\nabla F(x)$ is the average gradient across clients.

To solve these non-separable objectives in a federated setting, the authors propose four algorithms:

FairLoss & FairLoss*: Approximate and exact variants for minimizing the loss-variance objective ( $L_\lambda$ ).
FairGrad & FairGrad*: Approximate and exact variants for minimizing the gradient-variance objective ( $J_\gamma$ $J_{γ}$ ).
- Approximate variants (FairLoss, FairGrad): Use information from the previous iteration (e.g., previous average loss or gradient) to reduce communication overhead.
- Exact variants (FairLoss*, FairGrad*): Use current iteration information for more accurate updates at the cost of increased communication.

These are compared against baselines including FedAvg, Agnostic Federated Learning (AFL), $q$ -FFL, and AAggFF.

Theoretical Analysis

The paper provides a theoretical characterization of the relationships between these methods under both homogeneous and heterogeneous data settings:

Homogeneous Data: In settings where all clients share the same data distribution, the regularization terms (loss or gradient variance) do not alter the location of the stationary points on average. Thus, the optimal solutions remain consistent with the standard FedAvg objective, and regularization is theoretically unnecessary.
Heterogeneous Data: Under heterogeneity, the regularization terms actively shift the optimization landscape. The authors prove that if client losses or gradients are not aligned at the optimum of the unregularized objective, the regularized objectives ( $L_\lambda$ , $J_\gamma$ , and $H_q$ ) will have different stationary points. Specifically, variance regularization can move the solution away from an average-loss optimum toward a point with more balanced performance across clients.
Connections: The analysis establishes that $q$ -FFL (with $q=1$ ) can be upper-bounded by the loss-variance regularization objective under specific conditions, linking these previously distinct approaches.

Experimental Results

The methods were evaluated on four image classification datasets (MNIST, CIFAR-10, CIFAR-100, Tiny ImageNet) with varying levels of data heterogeneity controlled by Dirichlet concentration parameters ( $\alpha \in \{0.05, 0.1, 0.5\}$ ).

Performance: The proposed variance-regularized methods, particularly FairGrad and FairGrad*, consistently outperformed baselines in heterogeneous settings.
Fairness: The methods significantly reduced the variance of client accuracies (the fairness metric) compared to FedAvg, AFL, and $q$ $q$ -FFL.
- On CIFAR-100, FairGrad* achieved the best accuracy and lowest variance at high heterogeneity ( $\alpha=0.05, 0.1$ ).
- On CIFAR-10, FairGrad consistently achieved the lowest variance across all heterogeneity levels and the highest accuracy at $\alpha \in \{0.1, 0.5\}$ .
- On Tiny ImageNet, FairGrad* achieved the lowest variance at $\alpha \in \{0.05, 0.5\}$ .
Homogeneous Settings: On MNIST (where data is relatively homogeneous), all methods performed similarly, with regularized variants offering only marginal improvements in fairness, aligning with the theoretical prediction that regularization is less critical in homogeneous settings.

Significance and Claims

The paper claims that FairGrad and FairGrad* represent a principled and effective approach to improving fairness in Federated Learning. Key contributions include:

Novelty: The introduction of gradient-variance regularization specifically for performance equitable fairness, a direction not previously explored in this context.
Theoretical Clarity: Providing a theoretical framework that explains the relationship between existing fairness methods (like $q$ -FFL and loss-variance regularization) and demonstrates how variance regularization alters optimization behavior under heterogeneity.
Empirical Validation: Demonstrating that reducing gradient variance leads to more equitable performance across clients without sacrificing overall model accuracy, often outperforming state-of-the-art fairness-aware baselines.

The authors conclude that while variance-based regularization is theoretically redundant in homogeneous settings, it is a crucial mechanism for achieving fairness in the heterogeneous scenarios typical of real-world FL deployments. The work highlights the gradient alignment regularizer as a robust method for balancing client performance.