📊 statistics

A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

This paper proposes a novel data preprocessing algorithm using quantile distribution mapping to achieve counterfactual fairness in reinforcement learning by estimating counterfactual states and rewards without relying on strict additivity assumptions, while providing theoretical bounds on unfairness and suboptimality and validating the approach through numerical and real-world digital health experiments.

Original authors: Jianhan Zhang, Jitao Wang, John D. Piette, Donglin Zeng, Chengchun Shi, Zhenke Wu

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianhan Zhang, Jitao Wang, John D. Piette, Donglin Zeng, Chengchun Shi, Zhenke Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a series of decisions, like a doctor choosing treatments for a patient over many weeks or a self-driving car navigating a busy city. This field is called Reinforcement Learning (RL). Think of it as a video game where the robot learns by trial and error: it takes an action, sees what happens, gets a "score" (reward), and tries to figure out how to get the highest total score over time. The goal is usually to help the most people possible.

However, there's a tricky problem. Sometimes, the robot learns to be unfair. It might start treating people differently based on things like their race, gender, or background, not because it's "evil," but because it noticed patterns in the data that link those traits to certain outcomes. This is like a robot learning that "people from Group A usually get sick less often," so it decides to give them fewer check-ups, accidentally ignoring the fact that Group A might have had better access to food or medicine in the past. This is called unfairness.

To fix this, scientists use a concept called Counterfactual Fairness. Imagine you could hit "undo" on a person's life and ask: "If this person had been born a different race or gender, but everything else about their life history stayed exactly the same, would the robot still make the same decision?" If the answer is "yes," the robot is being counterfactually fair. It's not just about looking at groups of people; it's about ensuring the robot treats every single individual fairly, regardless of their sensitive traits.

The Paper's Big Idea: A Time-Traveling Map for Robots

This paper introduces a clever new tool called CFSMDM (Counterfactually Fair Sequential Marginal Distribution Mapping). Think of it as a special "data translator" or a "time-traveling map" that helps the robot learn to be fair before it even starts making decisions.

The authors realized that previous methods for making robots fair were a bit too rigid. They assumed that the world works in a simple, straight-line way (like adding numbers together). But real life is messy and curved. Sometimes, a person's background changes how they experience the world in complex, non-linear ways. The old tools would break or get confused in these situations.

What CFSMDM Does:
Instead of trying to guess the exact math of how the world works, CFSMDM uses a technique called Quantile Distribution Mapping. Here is a simple way to visualize it:

Imagine you have a giant jar of marbles representing all the possible outcomes for a patient. Some marbles are red (bad outcomes), some are blue (good outcomes). The robot needs to know which marble a specific patient will get.

The Old Way: The robot tries to predict the average color of the marble.
The CFSMDM Way: The robot looks at where the patient's marble sits in the jar. Is it in the top 10%? The bottom 50%? This is the "quantile."
The Magic Step: The algorithm asks, "If this patient had a different background (say, a different gender), but kept the exact same life story, where would their marble sit in that new jar?"
The Translation: It then finds the marble in the "new jar" that sits in the exact same spot (the same quantile) as the original marble.

By doing this step-by-step for every week of the patient's journey, CFSMDM creates a new, "fair" version of the training data. It strips away the unfair influence of sensitive traits while keeping the rest of the story intact. The robot then learns its policy (its decision rules) using this fair, translated data.

What They Found:
The authors tested this idea in two main ways:

Simulations: They created a fake world with a computer where they knew exactly how the unfairness worked. They tested CFSMDM against other methods. The results showed that CFSMDM was much better at keeping the robot fair, especially when the world was messy and non-linear (where the old, rigid methods failed). It managed to keep the "unfairness score" very low while still making good decisions.
Real-World Data: They applied their method to a real dataset from a study called PowerED, which involved helping patients manage opioid pain medication over 12 weeks. They looked at sensitive traits like race, gender, and education.
- The Result: When they used CFSMDM, the robot's decisions became much fairer across almost all these groups. For example, it reduced the unfairness related to "education" and "ethnicity" more effectively than other methods.
- The Trade-off: There was a tiny cost. The robot's overall "score" (how well it helped patients in general) was slightly lower than if it had ignored fairness completely. This is expected: making a system fair often requires giving up a tiny bit of raw efficiency to ensure everyone gets a fair shot.

What They Argue Against:
The paper explicitly argues against the idea that you can just "add up" the effects of background traits to fix unfairness (the "additive noise" assumption used by previous methods). They showed through math and simulations that the real world is often more complex than simple addition, and methods that assume it is simple will fail to catch all the unfairness.

How Sure Are They?
The authors are very confident in their mathematical proof that their method works under certain reasonable conditions. They proved that the "unfairness gap" and the "loss in performance" are bounded, meaning they can't get infinitely bad. In their simulations and real-world tests, the method consistently performed well, showing that as they gave the robot more data, the unfairness went down. However, they note that in the real-world study, the improvement wasn't perfect for every single group (for instance, "sex" showed a tiny bit of unfairness remaining), likely because the real data was small and the math is hard to get exactly right every time.

In short, this paper offers a robust, flexible new way to teach robots to make fair decisions in complex, real-world scenarios, ensuring that the "time-traveling map" they use to learn doesn't leave anyone behind.

Technical Summary: A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) is increasingly used for sequential decision-making in high-stakes domains like healthcare. However, standard RL agents may inadvertently learn policies that systematically disadvantage specific subpopulations defined by sensitive attributes (e.g., race, gender). This unfairness often arises not because the sensitive attribute is explicitly used as a feature, but because state and reward variables encode implicit information correlated with these attributes.

Existing fairness criteria, such as demographic parity, operate at the group level and rely on statistical associations rather than causal mechanisms. Counterfactual Fairness (CF), introduced by Kusner et al. [19] and extended to RL by Wang et al. [31], offers a more rigorous framework. CF requires that an individual's decision distribution remains unchanged if their sensitive attribute were hypothetically switched to a different value, holding all historical exogenous variables constant.

While Wang et al. [31] proposed a data preprocessing algorithm to enforce CF in offline RL, their method relies on a strict additive noise assumption (i.e., the state and reward are linear functions of the sensitive attribute plus independent noise). This assumption is often restrictive in real-world scenarios where the relationship between variables is non-linear or non-additive.

Methodology: Counterfactually Fair Sequential Marginal Distribution Mapping (CFSMDM)

This paper proposes CFSMDM, a novel data preprocessing algorithm that enables counterfactually fair policy learning without requiring the additive noise assumption. The method operates in two stages:

1. Theoretical Foundation

The approach relies on Structural Causal Models (SCMs) and the concept of counterfactuals. The authors introduce Assumption 1 (strict monotonicity of the state in the noise variable) and Assumption 2 (strict monotonicity of the reward in the noise variable). These assumptions are strictly weaker than the additive noise assumption used in prior work [31].

Under these assumptions, the authors prove (Theorems 1 and 2) that the quantile level of an observed state (or reward) with respect to its conditional distribution remains invariant across different values of the sensitive attribute, provided the historical exogenous variables are held constant. This allows for the recovery of counterfactual states and rewards via quantile matching.

2. The Algorithm

CFSMDM follows a two-stage procedure:

Data Preprocessing (SMDM):
- The algorithm estimates conditional quantile functions for states and rewards given the sensitive attribute and historical context.
- For each observed trajectory, it calculates the quantile level ( $\tau$ ) of the observed state/reward under the actual sensitive attribute.
- It then maps this quantile level to the conditional distributions under counterfactual sensitive attribute values to estimate the counterfactual states and rewards.
- These estimates are aggregated to construct augmented states ( $\tilde{S}_t$ ) and augmented rewards ( $\tilde{R}_t$ ), which are invariant to the observed sensitive attribute.
Policy Learning:
- Standard offline RL algorithms (e.g., Fitted Q Iteration) are applied to the preprocessed dataset containing the augmented states and rewards.
- Since the augmented data contains no information about the sensitive attribute (except for residual estimation errors), the resulting policy is theoretically counterfactually fair.

Key Contributions

Generalized Algorithm: The paper introduces CFSMDM, which extends the scope of counterfactually fair RL to a strictly larger class of contextual Markov Decision Processes (CMDPs) by relaxing the additive noise assumption to a strict monotonicity assumption.
Theoretical Guarantees: The authors derive bounds for both the suboptimality gap (the difference between the learned policy and the optimal policy) and the level of counterfactual unfairness. These bounds demonstrate that the unfairness and suboptimality are controlled by the estimation errors of the conditional quantile functions and the number of Fitted Q Iteration (FQI) steps.
Empirical Validation: The method is evaluated through numerical experiments on synthetic data with non-additive noise and applied to a real-world dataset from the PowerED study (an intervention to reduce opioid misuse).

Results

Numerical Experiments

Simulations were conducted on CMDPs with non-additive noise, comparing CFSMDM against several baselines:

Full: Uses sensitive attributes (unfair).
Unaware: Excludes sensitive attributes (often insufficient for fairness).
Random: Baseline for fairness (perfectly fair but low utility).
FLAP_M, ECOCF_M: Adaptations of single-stage fairness methods.
CFSDP: The prior state-of-the-art method [31] requiring additive noise.

Findings:

Fairness: CFSMDM achieved significantly lower counterfactual unfairness metrics than "Full," "Unaware," "FLAP_M," and "ECOCF_M." It performed comparably to or better than CFSDP, particularly in non-additive noise settings where CFSDP's performance degraded due to assumption violations.
Utility: As expected, enforcing fairness resulted in a trade-off, with CFSMDM achieving lower policy values than the "Full" and "Unaware" methods. However, it maintained competitive utility compared to other fair methods.
Robustness: CFSMDM remained robust as the strength of the sensitive attribute's impact on the environment increased.

Real-World Application (PowerED Study)

The algorithm was applied to a dataset of 207 patients in a 12-week opioid intervention study.

Metrics: CFSMDM achieved the best counterfactual fairness across most sensitive attributes (Education, Ethnicity, Age) compared to baselines.
Nuance: For the "Sex" attribute, CFSMDM was slightly less fair than CFSDP and "Unaware." The authors attribute this to the fact that excluding the sensitive attribute from decision-making already removed most unfairness in this specific case, meaning the finite-sample estimation errors introduced by CFSMDM slightly outweighed the marginal fairness gains.

Significance and Claims

The paper claims that CFSMDM provides a more general and robust tool for ensuring counterfactual fairness in RL compared to existing methods like CFSDP. By relying on quantile matching rather than additive noise, it addresses a broader class of real-world environments where causal relationships are complex and non-linear.

The authors emphasize that while their method introduces a fairness-utility trade-off, it offers a principled way to mitigate systematic discrimination in sequential decision-making. They acknowledge that the performance of CFSMDM depends on the accuracy of the conditional quantile estimation; if the quantile models are misspecified, the fairness guarantees may be compromised. The work positions itself as a step toward deploying RL agents in sensitive domains where causal fairness is a prerequisite for ethical deployment.