🤖 machine learning

Training Fair Tabular Foundation Models

This paper introduces FairTFM, a scalable training strategy that incorporates fairness constraints directly into Tabular Foundation Models via synthetic fairness tasks and a gradient reversal layer, enabling fair, in-context predictions without sensitive attribute access while maintaining competitive accuracy.

Original authors: Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

Published 2026-08-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a super-smart, pre-trained AI assistant. This assistant has read millions of case files and learned how to spot patterns in data, like a seasoned detective who can look at a few clues and instantly guess the culprit. This is the world of Tabular Foundation Models. Think of them as "universal detectives" that can jump into any new case—whether it's predicting loan approvals, diagnosing illnesses, or deciding who gets a scholarship—without needing to be retrained from scratch. They use a trick called In-Context Learning, which is like giving the detective a few examples of the current case in a notebook and saying, "Here's how we usually solve these; now you solve this one."

But here's the catch: even the smartest detectives can have hidden biases. If the detective learned from old case files where certain groups of people were treated unfairly, they might accidentally treat new people from those same groups unfairly, too. This is the problem of algorithmic fairness. In the real world, we have strict rules (like anti-discrimination laws) that say decisions shouldn't depend on sensitive traits like gender, race, or age. The big question scientists are asking is: Can we teach these "universal detectives" to be fair before they ever see a real case, so they don't need to be fixed later?

This paper introduces a new method called FairTFM, which tries to bake fairness directly into the detective's brain during its training. Instead of just teaching the model to be accurate, the researchers teach it to ignore the "sensitive" clues that shouldn't matter. They found that by using a clever training trick, they can create a model that makes fairer predictions in a single glance, without needing to be retrained for every new job. While the model sometimes has to make a tiny trade-off in how perfectly accurate it is, the results suggest it can significantly reduce unfairness—sometimes by over 70%—while still doing a great job at its main task.

The Problem: The "Universal Detective" with a Blind Spot

Tabular data is just information organized in rows and columns, like a giant spreadsheet. For decades, the best way to make predictions from these spreadsheets was to use tree-based models (think of them as flowcharts that ask a series of yes/no questions). But recently, a new type of AI called a Tabular Foundation Model (TFM) has emerged. These are like the "super-detectives" of the data world. They are trained on massive collections of data so they can adapt to new tasks instantly, using only a few examples provided at the moment of prediction.

The problem is that these super-detectives are learning from the past, and the past is often full of unfairness. If a bank's historical data shows that women were denied loans more often than men for no good reason, a standard TFM might learn that pattern and continue denying loans to women, even if it's illegal and wrong.

Usually, to fix this, data scientists try to "clean" the data before training, or they tweak the model after it's already trained. But with TFMs, this is tricky. Because these models are designed to be "frozen" (meaning we don't change their internal weights for every new task), we can't easily go back and retrain them for every specific bank or hospital. We need a way to make them fair before they are deployed, so they are ready to be fair the moment they are asked to solve a new problem.

The Solution: Training the Detective to Ignore the "Wrong" Clues

The authors of this paper, Patrik Kenfack and his team, propose a new way to train these models called FairTFM. Their idea is simple but powerful: during the training phase, they teach the model to solve a game where it has to predict the outcome without using the sensitive information.

Here is how they do it, using a fun analogy:

Imagine you are training a detective to solve a mystery about who stole a cookie. You have a list of clues: the time of day, the type of shoe print, and the suspect's favorite color. But you also have a "sensitive" clue: the suspect's gender. You want the detective to solve the crime based on the shoe print and time, but not based on gender.

In the past, you might have tried to just hide the gender clue. But the detective is smart; they might still guess the gender based on other clues (like the shoe print) and use that to gain an unfair advantage.

FairTFM uses a two-part training game:

The Main Game: The detective tries to guess who stole the cookie (the main prediction).
The Anti-Cheat Game: At the same time, the detective is forced to guess the suspect's gender based only on the clues they used to solve the crime.

Here is the magic trick: The researchers use a special tool called a Gradient Reversal Layer. Think of this as a "reverse gear" for the detective's brain. When the detective tries to guess the gender, the tool flips the feedback. If the detective gets the gender right, the tool says, "No, that's bad! You're relying too much on gender!" and pushes the detective to forget that connection.

By doing this over and over again with thousands of different fake "cookie theft" scenarios (where the sensitive clue changes from gender to age to race randomly), the detective learns to solve the crime using only the relevant clues. They learn to build a mental map of the world where the gender clue is invisible.

The Results: Fairness Without Losing the Plot

The team tested their new "Fair Detective" (FairTFM) on 132 different tasks, ranging from predicting income to checking if someone will be re-arrested. They compared it against:

Standard Detectives: The usual, super-accurate TFMs (like TabPFN and TabICL) that don't care about fairness.
Old-School Detectives: Traditional models like Random Forests and Logistic Regression.
The "Fair" Competitor: Another model called FairPFN, which tries to be fair but uses a different, more complex definition of fairness.

The results were promising. The standard, super-accurate TFMs were indeed the most accurate, but they were also the most unfair. They made big mistakes in how they treated different groups of people.

FairTFM, however, found a sweet spot. By adjusting a "knob" (called $\lambda$ ) that controls how hard the model tries to be fair, the researchers could dial in the perfect balance.

The Trade-off: To get fairer predictions, the model had to give up a little bit of raw accuracy. The paper found that for a small drop in accuracy (between 2% and 11%), the model became 32% to 75% fairer than the standard models.
The "One-Shot" Win: The best part is that FairTFM does this in a single forward pass. You don't need to retrain it for every new bank or hospital. You just give it the data, and it produces a fair prediction instantly.

When compared to the other "fair" model (FairPFN), FairTFM came out ahead. FairPFN was fair, but it was much less accurate. FairTFM managed to be both fair and accurate, beating FairPFN on almost every metric.

What This Means for the Future

The paper suggests that we don't have to choose between a smart AI and a fair AI. By training the model to ignore sensitive attributes during its "childhood" (pre-training), we can create a tool that is ready to be fair the moment it enters the real world.

Of course, the authors are careful to note that this isn't a magic wand that solves all discrimination. The model still needs to be tested on real-world data, and the definition of "fairness" can be complex. But this work shows a clear path forward: we can build foundation models that are not just powerful, but also principled, making them safer and more trustworthy for high-stakes decisions in finance, healthcare, and justice.

In short, the authors have shown that with the right training, we can teach our AI detectives to solve crimes without letting their biases get in the way. And that's a win for everyone.

Technical Summary: Training Fair Tabular Foundation Models

Problem Statement

Tabular Foundation Models (TFMs), such as TabPFN and TabDPT, have emerged as state-of-the-art methods for tabular predictive tasks, leveraging in-context learning (ICL) to adapt to new tasks without task-specific training or hyperparameter tuning. However, as these models are increasingly deployed in high-stakes domains like healthcare and finance, their fairness properties remain largely unexplored. Existing fairness-aware training techniques, which typically intervene at pre-processing, during training, or post-processing, are incompatible with the ICL paradigm. In ICL, predictions must be generated in a single forward pass using a frozen model, precluding the application of task-specific fairness constraints or retraining. Furthermore, existing TFM fairness approaches, such as FairPFN, focus on causal fairness notions (counterfactual fairness) rather than the statistical group fairness metrics (Demographic Parity, Equalized Odds, Equal Opportunity) commonly used in tabular benchmarks. This creates a gap: there is no scalable method to ensure TFMs produce fair predictions directly through ICL without access to sensitive attributes at inference time or requiring task-specific optimization.

Methodology

The authors propose FairTFM, a framework that incorporates fairness constraints directly into the pretraining of tabular foundation models. The approach consists of two primary components:

1. Synthetic Fairness Task Generation

To enable scalable fairness-aware pretraining without relying on labeled sensitive attributes in every dataset, the authors introduce a self-supervised task sampling strategy.

Process: From a sampled dataset, one input feature is randomly designated as the sensitive attribute ( $s$ ) and removed from the feature set.
Discretization: If the selected feature is continuous, it is converted into a categorical variable using a "stick-breaking" discretization scheme. This involves sampling proportions from Beta distributions to construct mixture weights via a Dirichlet process, creating data-dependent, non-uniform bins.
Task Formation: The dataset is split into context (training) and query (inference) sets. The sensitive attribute is masked in the query set and replaced with a learnable parameter during both pretraining and inference, ensuring the model does not rely on direct access to $s$ at inference time.

2. Fairness-Aware Architecture and Training

The model extends the nanoTabPFN transformer architecture with a dual-head prediction mechanism and adversarial learning.

Architecture: The transformer processes triplets $(x_i, y_i, s_i)$ $(x_{i}, y_{i}, s_{i})$ , where $s_i$ $s_{i}$ is the tokenized sensitive attribute. It utilizes a shared backbone and two Multi-Layer Perceptron (MLP) heads:
1. Label Prediction Head: Predicts the target label $\hat{y}_{qy}$ .
2. Sensitive Attribute Head: Predicts the sensitive attribute $\hat{s}_{qy}$ .
Gradient Reversal Layer (GRL): A GRL is placed before the sensitive attribute head. While this head is trained to minimize prediction error for $s$ , the GRL multiplies the gradient signal by $-1$ before backpropagating to the shared transformer encoder. This adversarial objective forces the encoder to learn representations that are invariant to the sensitive attribute, effectively suppressing information about $s$ while preserving task-relevant signal.
Training Objective: The model is optimized using a joint cross-entropy loss:
$L = CE(\hat{y}_{qy}, y_{qy}) + \lambda \cdot CE(\hat{s}_{qy}, s_{qy})$
The hyperparameter $\lambda$ controls the trade-off between predictive accuracy and fairness. A higher $\lambda$ increases the penalty for encoding sensitive information, pushing the model toward the fairer region of the Pareto frontier.

Key Contributions

Scalable Fairness Task Synthesis: A strategy for generating diverse fairness tasks from generic data priors using stick-breaking group construction, requiring no pre-labeled sensitive attributes.
FairTFM Architecture: A transformer-based architecture utilizing adversarial learning (via GRL) that enables fair predictions in a single forward pass. It handles sensitive attributes that change identity across tasks and are masked at inference.
Comprehensive Empirical Evaluation: Extensive testing on 132 fairness tasks (120 derived from ACS PUMS and 12 from other benchmarks) demonstrating that FairTFM improves statistical group fairness metrics while maintaining competitive accuracy.

Experimental Results

The authors evaluated FairTFM against standard machine learning baselines (LR, RF, XGBoost, KNN), recent TFMs (TabPFNv2.5, TabICLv2), and the causal-fairness baseline FairPFN.

Fairness-Accuracy Trade-off: On 120 ACS-based tasks, FairTFM variants consistently improved fairness metrics (Demographic Parity, Equalized Odds, Equal Opportunity) by 32–75% relative to the strongest TFM baseline (TabPFNv2.5), with accuracy costs of only 2–11%.
Comparison with Classical Methods: FairTFM achieved competitive or superior performance compared to classical baselines. Specifically, at $\lambda=0.7$ , FairTFM dominated Logistic Regression on all metrics, offering higher accuracy and better fairness.
Comparison with FairPFN: While FairPFN (causal fairness) achieved lower unfairness than unconstrained models, it suffered a significant predictive cost (lowest AUCROC). FairTFM provided operating points that Pareto-dominated FairPFN, offering both higher accuracy and better fairness.
Generalization: The model generalized well to 12 non-ACS benchmarks (e.g., Adult, COMPAS, German Credit), showing that fairness learned from synthetic pretraining transfers across data provenances.
Representation Analysis: Probing experiments confirmed that FairTFM embeddings selectively remove sensitive information (reducing sensitive attribute prediction AUC to near-random guessing) while preserving label predictability.
Efficiency: Unlike classical fairness-aware methods (e.g., Exponentiated Gradient) that require retraining for every new task and constraint, FairTFM achieves these results in a single forward pass without task-specific optimization.

Significance and Claims

The paper claims to present the first pretraining framework for tabular foundation models that accounts for statistical group fairness notions (Demographic Parity, Equal Opportunity, Equalized Odds) during pretraining.

The significance of this work lies in its ability to integrate fairness directly into the foundation model's pretraining phase, enabling fair predictions in a single forward pass without the need for post-hoc correction, task-specific retraining, or access to sensitive attributes at inference time. The authors argue that this approach bridges the gap between the efficiency of in-context learning and the ethical requirements of high-stakes decision-making. They position FairTFM not as a replacement for specialized fairness methods in every scenario, but as a scalable foundation that offers competitive Pareto-efficient solutions across heterogeneous tasks, paving the way for future work on better fairness-task priors and adaptive fairness behaviors in foundation models.

The authors modestly acknowledge limitations, noting that their pretraining relies on synthetic priors which may not capture all semantic structures of real-world sensitive attributes, and that FairTFM does not dominate specialized fairness baselines on every single metric or task configuration, particularly in stricter settings involving age as a sensitive attribute.