🤖 machine learning

Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing

This paper re-examines the role of the comparator in discrimination testing by distinguishing between the standard "ceteris paribus" approach and a novel "mutatis mutandis" framework that accounts for how protected attributes causally influence non-protected features, thereby offering a more complex but impactful alternative for machine learning-based discrimination detection.

Original authors: Jose M. Alvarez, Salvatore Ruggieri

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jose M. Alvarez, Salvatore Ruggieri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide if a hiring manager discriminated against a job applicant named Clara. Clara is a woman, and she was rejected. She claims she was rejected because she is a woman.

To prove this, you need to compare her to someone else. But who? This is the central puzzle the paper solves.

The authors, José M. Álvarez and Salvatore Ruggieri, argue that how we choose this "comparison person" (called a comparator) changes the entire outcome of the test. They propose two different ways to pick this person: the "All Else Equal" method and the "Make Adjustments" method.

Here is a simple breakdown of their argument using everyday analogies.

1. The Standard Method: "All Else Equal" (Ceteris Paribus)

This is the traditional way discrimination testing works. Imagine you want to see if a race car is faster than a truck. You put them on the exact same track, with the exact same fuel, and the exact same weather. The only thing you change is the vehicle type.

In the paper's terms, this is the CP Comparator.

The Scenario: Clara (female) applied for a job. To test her claim, we find a Mike (male) who has the exact same resume, the exact same number of years of experience, and the exact same grades.
The Logic: If Mike gets the job and Clara doesn't, despite having identical resumes, then Clara was discriminated against.
The Problem: The authors argue this is an "idealized" fantasy. In the real world, being a woman (or a member of a protected group) often changes your life before you even apply for the job. Maybe Clara took time off to raise children, or maybe she was steered away from certain classes because of societal expectations. If you find a Mike with the exact same resume, you are ignoring the fact that society treated them differently before the resume was written. You are pretending that the past didn't exist.

2. The New Method: "Make Adjustments" (Mutatis Mutandis)

The authors propose a smarter, more complex way to choose the comparison person. This is the MM Comparator.

Think of this like a time-travel simulation.

The Scenario: We still have Clara. But instead of finding a Mike with the exact same resume, we ask: "If Clara had been born a man, how would her life have unfolded differently?"
The Adjustment: If being a woman meant Clara had less time to study or fewer networking opportunities, the MM method says: "Let's adjust Mike's resume to match what Clara's life would have been if she were a man."
The Result: Maybe Mike ends up with more experience or better grades than the Clara we see today, because he didn't face the same societal hurdles.
The Logic: We compare the real Clara to this adjusted Mike. If the adjusted Mike gets the job and the real Clara doesn't, we have strong evidence of discrimination. This method acknowledges that the "playing field" wasn't level to begin with.

The "Tenure" Example

The paper uses a story about a university professor named Clara to illustrate this:

Clara (female) has 12 publications and was denied tenure.
Mike (male) has 12 publications and was denied tenure.
Vincent (male) has 18 publications and got tenure.

Using the "All Else Equal" (CP) method:
You compare Clara to Mike. They both have 12 papers. Since Mike was also rejected, you might say, "See? It wasn't because she's a woman; she just didn't have enough papers." You might dismiss her claim.

Using the "Make Adjustments" (MM) method:
You ask: "If Clara were a man, would she have had 12 papers, or would she have had 18?" The authors argue that because women often face extra burdens (like childcare) that men don't, a man in Clara's position might have had more time to write papers. So, the "adjusted" version of Clara (let's call her "Male-Clara") might have 18 papers.
You then compare the real Clara (12 papers) to Vincent (18 papers, who got tenure). If the university rejects the 12-paper applicant but accepts the 18-paper applicant, and we know the 12-paper applicant was held back by gendered societal roles, the discrimination is clearer.

Why Does This Matter?

The authors say that most current tools for finding discrimination use the CP (All Else Equal) method. They are like a camera that only takes photos of the present moment, ignoring the history that got us there.

The MM (Make Adjustments) method is like a camera with a "rewind and edit" button. It tries to reconstruct what the world would look like if the protected attribute (like race or gender) hadn't influenced the person's life path.

The Catch:
The authors admit the MM method is much harder to build. It requires complex machine learning models to simulate these "what if" scenarios. It's like trying to write a perfect alternate history novel. The CP method is easy (just find a twin), but it might miss the real injustice. The MM method is complex and requires more imagination, but it might tell the truth about how the system actually works.

The Bottom Line

The paper doesn't say the old way is "wrong" or the new way is "perfect." Instead, it says: "We need to stop pretending there is only one way to compare people."

When we test for discrimination, we are making a choice about what "fairness" looks like.

If we want to see if two people with the same resume are treated differently, use the old way.
If we want to see if two people who faced different life paths due to their identity are treated fairly, we need the new way.

The authors believe that because we now have powerful AI tools, we should start using the "Make Adjustments" method to get a clearer, more honest picture of discrimination.

1. Problem Statement

The paper addresses a fundamental limitation in current discrimination testing methodologies, specifically regarding the construction of the comparator.

Context: Discrimination testing relies on comparing a "complainant" (an individual claiming discrimination based on a protected attribute $A$ , e.g., race or gender) with a "comparator" (a similar individual from a non-protected group).
The Standard Approach (Ceteris Paribus - CP): Most existing tools (both legal and algorithmic) use a ceteris paribus ("all else equal") comparator. This assumes that the comparator is identical to the complainant in all non-protected attributes ( $X$ ), differing only in the protected attribute ( $A$ ).
The Flaw: The authors argue that the CP approach is an idealized comparison that fails to account for the causal reality that protected attributes often influence non-protected attributes (e.g., systemic racism affecting education levels or income). By forcing $X$ to remain identical when $A$ changes, the CP comparator ignores the downstream effects of discrimination, potentially masking indirect discrimination or failing to capture the true "what would have been" scenario.
Core Question: How can we derive a comparator that accounts for the causal influence of the protected attribute on other features, rather than assuming a static, idealized world where only the protected attribute changes?

2. Methodology

The authors propose a causal modeling framework to redefine the comparator, moving from a static distance metric to a generative, counterfactual approach.

A. Causal Framework

The paper utilizes Structural Causal Models (SCM) (Pearl's framework) to formalize discrimination testing.

Variables: $A$ (Protected Attribute), $X$ (Non-Protected Attributes), $Y$ (Decision Outcome).
Counterfactual Reasoning: The derivation of a comparator is framed as answering a counterfactual question: "What would the outcome be if the complainant had a different protected attribute?"
Procedure: The authors adopt the standard Abduction-Action-Prediction steps:
1. Abduction: Infer the latent exogenous variables ( $U$ ) from the observed data.
2. Action: Intervene on $A$ (set $A := \alpha$ ).
3. Prediction: Propagate the change through the structural equations to generate the counterfactual distribution of $X$ and $Y$ .

B. The Two Comparator Types

The paper introduces a formal classification of comparators:

Ceteris Paribus (CP) Comparator:
- Definition: Finds an individual $i'$ such that $d(x_i, x_{i'}) \leq \epsilon$ and $a_i \neq a_{i'}$ .
- Mechanism: Minimizes distance on non-protected attributes $X$ while flipping $A$ .
- Limitation: Assumes $X$ is independent of $A$ in the counterfactual world, ignoring historical or systemic causal links.
Mutatis Mutandis (MM) Comparator:
- Definition: Finds an individual $i'$ such that $\tilde{d}((x_i, a_i), (x_{i'}, a_{i'})) \leq \epsilon$ and $a_i \neq a_{i'}$ .
- Mechanism: Uses a protected-aware distance function $\tilde{d}$ . Crucially, it allows $x_i \neq x_{i'}$ .
- Implementation: The MM comparator requires generating a counterfactual representation $\tilde{X}$ of the complainant's non-protected attributes. This $\tilde{X}$ represents what the complainant's attributes would have been had they belonged to the non-protected group, effectively "adjusting" for the downstream effects of $A$ on $X$ .
- Mathematical Formulation: The authors propose using a generative model $g(\cdot)$ (e.g., based on SCM) to learn $\tilde{X} = g(X, A)$ such that $\tilde{X} \perp A$ (fair representation). The comparator is then found by minimizing distance between $\tilde{X}$ and the actual $X$ of the non-protected group.

C. Experimental Setup

To validate the distinction, the authors compare two tools on the Law School Admissions dataset:

Situation Testing (ST): Implements the CP comparator. It finds the nearest neighbors in the dataset based on $X$ (GPA, LSAT) after flipping the protected attribute.
Counterfactual Situation Testing (CST): Implements the MM comparator. It uses an SCM to generate counterfactual profiles (adjusting GPA/LSAT based on the causal influence of race/gender) before finding nearest neighbors.

3. Key Contributions

Causal Reframing: The authors argue that deriving a comparator is inherently a causal modeling problem, not just a similarity search. The comparator is a counterfactual representation of the factual complainant.
The MM Comparator Concept: They introduce the Mutatis Mutandis comparator as a viable alternative to the standard CP comparator. This shifts the goal from "treating similar people similarly" (formal equality) to "achieving substantive equality" by accounting for systemic disadvantages.
Formal Distinction: The paper provides a rigorous mathematical distinction between CP and MM comparators, highlighting that MM allows for dissimilarity in non-protected attributes if those differences are caused by the protected attribute.
Integration of ML: The authors position Machine Learning (specifically generative models and SCMs) as essential for implementing the MM comparator, contrasting with views that reject causal modeling in discrimination testing.

4. Results

The illustrative experiment on the Law School Admissions dataset yielded significant findings:

Detection Rates: The CST (MM) tool detected significantly more prima facie discrimination cases than the ST (CP) tool.
- Race: CST detected ~3x more cases than ST (e.g., 17.26% vs 3.08% at $k=500$ ).
- Gender: CST detected slightly more cases, but the gap was smaller, suggesting the causal impact of gender on the specific features (GPA/LSAT) in this dataset was less pronounced than race.
Distribution Shifts: Visual analysis (Box plots) showed that under ST, the test group (comparators) had distributions of GPA/LSAT nearly identical to the control group. Under CST, the test group distributions were shifted, reflecting the counterfactual adjustment where the protected attribute's influence on the features was removed.
Implication: The CP approach (ST) often fails to detect indirect discrimination because it assumes that if a minority applicant has the same GPA/LSAT as a white applicant, they are "similar." The MM approach (CST) recognizes that achieving that GPA/LSAT might have required overcoming systemic barriers, and thus the "fair" comparator should have different feature values.

5. Significance and Conclusion

Substantive vs. Formal Equality: The paper argues that relying solely on CP comparators enforces formal equality (treating people with identical current stats the same), which may perpetuate existing inequalities. The MM comparator aims for substantive equality by envisioning a world where systemic barriers are removed.
Normative Impact: The choice of comparator is not just a technical parameter; it is a normative statement about what constitutes a "fair" comparison. The MM comparator forces stakeholders to confront the causal mechanisms of discrimination.
Future Directions: The authors suggest that future discrimination testing tools should move beyond simple distance metrics and incorporate generative models (like SCMs or Fair Representation Learning) to derive MM comparators. This allows the field to move from detecting direct discrimination to uncovering complex, indirect discrimination patterns rooted in systemic bias.

In summary, the paper posits that to truly test for discrimination in a complex, biased society, we must stop asking "Who is exactly like this person but with a different race/gender?" and start asking "Who would this person be if the systemic effects of their race/gender were removed?" The Mutatis Mutandis comparator provides the technical framework to answer the latter.