🤖 machine learning

Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularization

This paper proposes an end-to-end Fairness-Aware Mixture-of-Experts framework that mitigates routing-induced bias and performance disparities across demographic groups by combining subgroup reweighting with gate entropy regularization to ensure balanced expert utilization while maintaining competitive predictive performance.

Original authors: Sunhee Hwang

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sunhee Hwang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computers are learning to see the world with a clarity that rivals human vision. They can identify faces, sort medical scans, and recognize objects in a split second. Yet, as these systems move from research labs into real-world applications, a quiet flaw has emerged. Because the data used to teach them often reflects the unevenness of human society, the resulting models frequently perform better for some groups of people than for others. A system might recognize a young man's face with high accuracy but struggle significantly with an older woman's, simply because the training data contained far more pictures of the former. This is not a bug in the code, but a reflection of the data itself: when a computer learns from an unbalanced diet of information, it develops a skewed understanding of the world.

For years, researchers have tried to fix this by teaching the computer to ignore sensitive details like age or gender, or by manually adjusting the data to make it more even. However, a new study suggests that these traditional fixes miss a deeper problem, particularly when using a specific type of advanced architecture known as a mixture of experts. This approach is designed to be flexible, allowing a model to route different inputs to different specialized sub-networks, much like a hospital directing patients to the most appropriate specialist. The researchers found that when the data is unbalanced, this routing system can accidentally learn to send specific groups of people to specific specialists, effectively segregating the model's internal logic and reinforcing the very bias it was meant to solve.

Sunhee Hwang, a researcher at Dongyang Mirae University, set out to diagnose and correct this hidden failure. The study focuses on a framework called FAMoE, which stands for Fairness-Aware Mixture-of-Experts. To understand the problem, one must first understand how these models work. Instead of forcing every image through a single, rigid path, a mixture-of-experts model uses a "gate" to decide which of several internal experts should handle a specific input. In a well-functioning system, this gate should route images based on their visual content. However, Hwang discovered that when the training data is unbalanced—for instance, if there are far more pictures of young men than older women—the gate learns a shortcut. It begins to route inputs based on the sensitive attributes of the people in the photos rather than the actual task at hand. This phenomenon, which the author terms "routing-induced bias," causes certain experts to become overloaded with specific demographic groups while others sit idle, creating a new layer of unfairness that standard fairness methods cannot see.

To solve this, the team proposed a two-part strategy that works directly within the training process. First, they adjusted the importance of the data samples themselves. By giving more weight to the rare combinations of attributes—such as an older woman with a specific facial feature—the model is forced to pay attention to the groups it would otherwise ignore. This ensures that the learning signal is strong enough for everyone, not just the majority. Second, and perhaps more critically, they added a rule to the gate that prevents it from becoming too predictable. In a standard setup, the gate might learn to always send one type of person to one specific expert. The new method introduces a penalty that encourages the gate to spread its decisions more evenly, ensuring that no single expert is monopolized by a specific demographic group. This keeps the internal routing of the model balanced and, importantly, interpretable.

The results of this approach were tested on a large dataset of facial images, where the goal was to predict certain attributes while ensuring fairness across gender and age groups. The study found that the new method significantly reduced the performance gap between different groups without sacrificing the overall accuracy of the system. In fact, the model achieved a better balance between being fair and being correct than many existing techniques, including those that rely on complex, multi-step training processes. Perhaps the most striking finding was not just the improvement in numbers, but the transparency of the solution. Because the routing mechanism is now balanced, researchers can actually look at the gate's decisions to see how the model is treating different groups. In traditional models, fairness is often a black box; here, the routing distribution acts as a window, showing exactly how the model allocates its resources.

This work highlights a crucial insight: fixing bias in artificial intelligence often requires looking at how the model makes decisions, not just what it learns. By addressing the way a model routes information, the researchers were able to prevent the system from developing hidden preferences for certain groups. The study demonstrates that when you correct the data imbalance and ensure the internal pathways remain open to all, the model naturally becomes fairer. It is a reminder that in the pursuit of intelligent systems, the path the data takes through the machine is just as important as the data itself.

Technical Summary: Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularization

Problem Statement

Deep learning models frequently exhibit performance disparities across demographic groups defined by sensitive attributes (e.g., gender, age). This unfairness often stems from training data imbalances where specific combinations of target labels and sensitive attributes are underrepresented. Consequently, models optimize for majority subgroups, leading to underfitting in minority intersections.

While existing fairness methods generally fall into two categories, they face distinct limitations:

Two-stage methods learn representations separately from the prediction task, potentially causing misalignment between fairness objectives and downstream predictions.
Single-stage methods typically rely on a shared representation space. While they jointly optimize for fairness and accuracy, a shared space struggles to capture the heterogeneous characteristics of different subgroups, often forcing a trade-off between bias mitigation and predictive performance.

Furthermore, the author identifies a previously underexplored failure mode specific to Mixture-of-Experts (MoE) models, termed routing-induced bias. In standard MoE architectures, subgroup imbalance can cause the gating network to correlate routing decisions with sensitive attributes. This results in specific subgroups being routed almost exclusively to a few experts, leaving other experts underutilized. This bias is invisible to traditional fairness methods that assume a monolithic shared representation.

Methodology: FAMoE Framework

The author proposes FAMoE (Fairness-Aware Mixture-of-Experts), an end-to-end framework designed to diagnose and correct routing-induced bias. The architecture consists of a shared backbone encoder, $N$ expert networks, and a gating network.

1. Architecture

Shared Backbone: Extracts a feature representation $z$ from the input image. This ensures that performance differences are attributable to the MoE components rather than feature extraction variations.
Expert Networks: Each expert $E_i$ processes $z$ independently to produce an expert-specific logit $o_i$ .
Gating Network: Computes a routing vector $g$ based on $z$ , determining the weight for each expert. The final prediction is a weighted aggregation of expert outputs.

2. Key Components

To address the identified biases, FAMoE integrates two specific mechanisms:

Subgroup Reweighting: To counteract data imbalance, the training process assigns weights to samples inversely proportional to the size of their joint subgroup $(y, s)$ , where $y$ is the target label and $s$ is the sensitive attribute. This is implemented via a weighted binary cross-entropy loss ( $L_{cls}$ ), ensuring underrepresented subgroups contribute more significantly to the gradient updates.
Gate Entropy Regularization: To prevent the gating network from collapsing routing decisions onto specific subgroups (which leads to expert monopolization), the author introduces an entropy regularization term ( $H(g)$ ) to the loss function.
- Unlike conventional load balancing which focuses on computational efficiency, this regularization specifically aims to keep the routing distribution interpretable.
- By encouraging higher entropy, the method prevents any single expert from being monopolized by a specific subgroup, ensuring that the routing weights reflect a balanced allocation of expert capacity across the population.

The total loss function is defined as:
$L = L_{cls} - \lambda_{ent}H(g)$
where $\lambda_{ent}$ controls the strength of the regularization.

Key Contributions

Identification of Routing-Induced Bias: The paper identifies a specific failure mode in MoE models where subgroup imbalance drives the gating network to correlate routing with sensitive attributes, leading to skewed expert utilization.
FAMoE Framework: The proposal of an end-to-end solution combining subgroup reweighting with gate entropy regularization. This approach corrects routing bias while maintaining the flexibility of conditional computation.
Interpretability via Routing: The author demonstrates that the routing distribution serves as an interpretable diagnostic tool. Unlike monolithic models where fairness is achieved in a "black box" shared space, FAMoE allows for the inspection of how subgroups are allocated across experts.
Empirical Validation: The method is shown to improve the fairness-accuracy trade-off across multiple settings on the CelebA dataset.

Experimental Results

The proposed method was evaluated on the CelebA dataset using binary classification tasks with various target and sensitive attributes (e.g., Attractiveness, Big Nose vs. Male, Young).

Metrics: Performance was measured using Equalized Odds (EO), Accuracy, and the Fairness-Accuracy Trade-off Score (FATS).
Comparisons: FAMoE was compared against baseline models (ResNet-18), standard MoE, and state-of-the-art fairness methods (both single-stage like GRL, MFD, and two-stage like FD-VAE, FairCL).
Findings:
- Fairness: FAMoE achieved the lowest FATS scores (indicating a better balance) across most target/sensitive attribute combinations. For instance, in the "Attractiveness" task with "Male" as the sensitive attribute, FAMoE achieved an EO of 4.1 compared to 13.3 for standard MoE and 7.4 for MFD.
- Accuracy: The method maintained competitive predictive accuracy, avoiding the significant drops often seen in fairness-constrained models.
- Ablation: Removing either subgroup reweighting or gate entropy regularization resulted in higher EO scores, confirming that both components are necessary to fully mitigate bias and recover accuracy.
- Expert Utilization: Analysis of gate weights revealed that standard MoE exhibited pronounced skewness (certain experts dominated by specific subgroups), whereas FAMoE produced a more balanced distribution of routing weights.

Significance and Claims

The paper argues that addressing fairness in MoE requires more than just standard representation learning; it necessitates explicit management of expert utilization. The significance of this work lies in:

Correcting a Specific Failure Mode: It addresses routing-induced bias, a problem inherent to conditional routing that existing fairness methods (built on shared representations) cannot detect or fix.
Enhanced Interpretability: By regularizing the routing distribution, the model provides a transparent view of subgroup-level behavior. This "routing-level interpretability" offers insights into fairness mechanisms that are absent in monolithic models.
Holistic Optimization: The framework demonstrates that jointly addressing data imbalance and routing dynamics leads to superior fairness-accuracy trade-offs compared to decoupled or single-stage approaches.

The author concludes that explicitly accounting for expert utilization is critical not only for improving fairness metrics but also for rendering the model's decision-making process more transparent and interpretable.