📊 statistics

Neyman-Pearson and equal opportunity: when efficiency meets fairness in classification

This paper introduces a Neyman-Pearson classification framework constrained by equal opportunity, deriving an oracle classifier and proposing finite-sample algorithms that simultaneously satisfy population-level fairness and efficiency guarantees with high probability.

Original authors: Jianqing Fan, Xin Tong, Yanhui Wu, Lucy Xia, Shunan Yao

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianqing Fan, Xin Tong, Yanhui Wu, Lucy Xia, Shunan Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a bank manager trying to decide who gets a loan. You have two main goals, but they often fight each other:

The Money Goal (Efficiency): You need to make sure you don't lend money to people who won't pay it back. If you lend to a "deadbeat," you lose cash. This is your Type I error (mistaking a bad borrower for a good one). You want to keep this risk very low.
The Fairness Goal (Society): You also want to be fair. You don't want to reject qualified applicants just because they are a woman, a person of a certain race, or from a specific neighborhood. This is about Equal Opportunity. You want the rejection rate for qualified people to be the same across all groups.

The Problem:
Usually, if you try to be super strict about not losing money (Goal 1), you might accidentally become unfair and reject too many qualified people from specific groups. If you try to force perfect fairness (Goal 2), you might have to lend to some risky people, losing money. It's a tug-of-war.

The Paper's Solution: The "NP-EO" Framework
The authors (a team of statisticians from Princeton, USC, HKU, etc.) propose a new way to play this game called the NP-EO framework. Think of it as a new set of rules for your loan algorithm.

The Analogy: The "Guardian" and the "Equalizer"

Imagine your loan algorithm is a security guard at a club.

The "NP" Part (The Guardian): This is the Neyman-Pearson rule. The bank says, "Guardian, your most important job is to keep the club safe. You must never let more than 10% of the bad guys (people who will default) inside." The Guardian is given a hard limit: Do not let the "bad guy" rate exceed 10%. Once that safety line is drawn, the Guardian tries to let in as many good people as possible.
The "EO" Part (The Equalizer): This is the Equal Opportunity rule. The bank says, "Guardian, you must also be fair. If a qualified person from Group A is rejected, a qualified person from Group B should have the exact same chance of being rejected." The Equalizer checks the numbers: Are the rejection rates for qualified people equal across groups?

The Innovation:
Most old methods tried to balance these by just "averaging" the errors or guessing how much money a mistake costs. This paper says: "No, let's set hard boundaries first."

They create a system where:

Safety is Non-Negotiable: The algorithm must keep the risk of bad loans below a specific number (e.g., 10%).
Fairness is a Hard Constraint: The algorithm must keep the difference in rejection rates between groups below a tiny number (e.g., 5%).
The Trade-off: If you can't meet both, the algorithm accepts that it might have to be a little less efficient at spotting good borrowers (letting a few more qualified people slip through the cracks) to ensure it doesn't break the safety or fairness rules.

How They Do It (The "Umbrella" Method)

The authors didn't just invent a new math formula; they built a tool they call an "Umbrella Algorithm."

Imagine you have a standard, off-the-shelf classifier (like a Logistic Regression or a Random Forest). It's like a generic umbrella that works okay in the rain.

The Problem: A generic umbrella doesn't know about your specific "Safety" and "Fairness" rules.
The Fix: The authors take that generic umbrella and add a special "handle" and "rim" to it. They use a technique called Order Statistics (basically, looking at the data from worst to best) to find the perfect "cut-off" points.

They split the data into two groups:

Group A (The "Bad" Borrowers): They look at the scores of people who did default. They find a threshold that ensures they catch 90% of them (keeping the risk low).
Group B (The "Good" Borrowers): They look at the scores of people who didn't default. They adjust the thresholds for different groups (e.g., men vs. women) until the rejection rates are equal, without breaking the safety rule from Group A.

What They Found (The Results)

They tested this on:

Simulated Data: Fake loan data created by computers.
Real Data: A real dataset of 30,000 credit card holders from Taiwan (checking for gender bias) and the famous "Adult" dataset (checking for income prediction bias).

The Verdict:

Old Methods (Fairness-only or Efficiency-only): When they tried to be fair, they often broke the safety rules (letting too many bad loans through). When they tried to be efficient, they were unfair.
The NP-EO Method: It was the only method that successfully kept both the safety risk and the fairness gap within the strict limits they set.
The Cost: To achieve this double-win, the algorithm did have to reject a few more qualified people than a "purely greedy" algorithm would have. But the paper argues this is a small price to pay to avoid the massive cost of financial ruin or legal discrimination.

In a Nutshell

This paper gives organizations a "rulebook" for building AI that doesn't have to choose between making money and being fair. It says: "Set a hard limit on how much risk you can take, set a hard limit on how unfair you can be, and then build the smartest system possible that stays inside those two lines."

It's like saying, "Drive as fast as you want, but you must never exceed the speed limit, and you must never drive on the wrong side of the road." The paper provides the GPS and the brakes to make sure you can do exactly that.

Technical Summary: Neyman-Pearson and Equal Opportunity

1. Problem Statement

The paper addresses the fundamental tension between economic efficiency and societal fairness in automated decision-making systems. While organizations increasingly rely on statistical algorithms for critical tasks (e.g., loan approvals, hiring), these systems often exhibit systematic biases against disadvantaged social groups. Existing approaches to algorithmic fairness frequently prioritize fairness constraints at the cost of predictive accuracy or economic efficiency. Conversely, standard efficiency-focused models often ignore disparate impacts on sensitive groups.

The authors identify a specific gap: current frameworks struggle to simultaneously satisfy asymmetric efficiency priorities (where one type of error is more costly than the other, e.g., financial risk in lending) and fairness constraints (specifically, Equal Opportunity) with rigorous statistical guarantees. The core challenge is a three-way trade-off between Type I error (false positives), Type II error (false negatives), and fairness constraints (disparity in error rates across groups).

2. Methodology: The NP-EO Paradigm

The paper proposes a novel classification framework called Neyman-Pearson Equal Opportunity (NP-EO). This framework integrates two distinct constraints:

A. The Efficiency Component (Neyman-Pearson Constraint)

Instead of minimizing overall error or using cost-sensitive learning (which requires specifying arbitrary misclassification costs), the authors adopt the Neyman-Pearson (NP) paradigm.

Goal: Control the Type I error (probability of misclassifying a negative instance as positive) to be below a user-specified level $\alpha$ with high probability.
Objective: Minimize the Type II error (probability of misclassifying a positive instance as negative) subject to the Type I constraint.
Rationale: This allows organizations to explicitly bound the risk of the most consequential error (e.g., approving a defaulting loan) while maximizing business opportunities (minimizing rejected good applicants).

B. The Fairness Component (Equal Opportunity Constraint)

The framework incorporates a relaxed version of Equal Opportunity (EO).

Goal: Ensure that the Type II error rates are approximately equal across different sensitive groups (e.g., race, gender).
Constraint: $L_1(\phi) = |R^a_1(\phi) - R^b_1(\phi)| \le \varepsilon$ , where $\varepsilon$ is a pre-specified tolerance.
Rationale: This ensures that qualified applicants from different groups have equal opportunities to receive the favored outcome (e.g., loan approval), without requiring absolute equality which is statistically infeasible in finite samples.

C. Theoretical Foundation: The NP-EO Oracle

The authors derive an NP-EO oracle classifier, which is the optimal solution to the constrained optimization problem:
$\min_{\phi} R_1(\phi) \quad \text{subject to} \quad R_0(\phi) \le \alpha \quad \text{and} \quad L_1(\phi) \le \varepsilon$
Using the Generalized Neyman-Pearson Lemma, they prove that the oracle classifier takes the form of a likelihood ratio test with two distinct thresholds ( $c_a$ and $c_b$ ) for the two sensitive groups:
$\phi^*(X, S) = \mathbb{I}\left\{ \frac{f_{1,a}(X)}{f_{0,a}(X)} > c_a \right\}\mathbb{I}\{S=a\} + \mathbb{I}\left\{ \frac{f_{1,b}(X)}{f_{0,b}(X)} > c_b \right\}\mathbb{I}\{S=b\}$
The existence of such thresholds is guaranteed under mild regularity conditions, ensuring that the NP and EO constraints are simultaneously feasible.

D. Algorithmic Implementation: Umbrella Algorithms

Since the true densities are unknown, the authors propose two finite-sample-based "umbrella" algorithms that work with any base scoring classifier (e.g., logistic regression, random forests, neural networks) without requiring distributional assumptions. These algorithms use order statistics to determine thresholds.

NP-EOOP (One Pair of Pivots):
- A conservative approach that selects a single pair of pivots (thresholds) to ensure high-probability control of Type I error for each group individually.
- It then adjusts these thresholds to satisfy the EO constraint.
- Guarantee: Provides strict high-probability control for both NP and EO constraints.
NP-EOMP (Multiple Pairs of Pivots):
- A more flexible approach that relaxes the requirement to control group-wise Type I errors individually. Instead, it controls the aggregate Type I error.
- It searches over multiple pairs of pivots to find a classifier that minimizes Type II error while satisfying the constraints.
- Advantage: Generally achieves lower Type II error (higher power) than NP-EOOP, though with slightly relaxed theoretical guarantees on the exact probability of Type I error control (approximate control).

Both algorithms utilize the Bernstein-von Mises theorem to approximate the distribution of error rates, allowing for the selection of thresholds that satisfy the constraints with high probability ( $1-\delta$ for NP, $1-\gamma$ for EO).

3. Key Contributions

Novel Framework: The introduction of the NP-EO paradigm, which explicitly decouples economic efficiency (via NP constraints) from social fairness (via EO constraints), moving beyond the traditional accuracy-fairness trade-off.
Oracle Derivation: Theoretical derivation of the NP-EO oracle classifier, proving that simultaneous feasibility of NP and EO constraints is possible and characterizing the form of the optimal solution.
Finite-Sample Algorithms: Development of two practical, non-parametric algorithms (NP-EOOP and NP-EOMP) that provide high-probability guarantees for both constraints in finite samples. Unlike plug-in methods, these do not rely on asymptotic convergence of the base classifier.
Theoretical Guarantees: Establishment of rigorous bounds showing that the proposed classifiers satisfy the NP and EO constraints with pre-specified high probabilities, a feature often lacking in existing fairness methods.
Empirical Validation: Demonstration through simulations and real-world data (credit card default prediction) that NP-EO classifiers are the only methods capable of simultaneously satisfying both constraints with high probability, whereas existing methods (FairBayes, Cost-Sensitive Learning) fail to maintain the required Type I error control or fairness levels.

4. Results

Simulations: In controlled Gaussian settings, NP-EOOP and NP-EOMP successfully maintained Type I error violation rates near 0 and EO violation rates near the target $\gamma$ (e.g., 0.05). In contrast, FairBayes failed to control Type I error (violation rate $\approx 1$ ), and Cost-Sensitive Learning (CSL) exhibited high violation rates for both constraints despite minimizing empirical risk.
Real-World Application (Credit Card Data): Applied to a Taiwanese credit card dataset, the NP-EO algorithms successfully controlled the default risk (Type I error) to the target level ( $\alpha=0.1$ ) while maintaining gender fairness (EO violation rate $< 0.05$ ). Classical classifiers failed to control default risk (violation rate = 1), and NP-only classifiers failed to ensure fairness (EO violation rate $\approx 0.46$ ).
Trade-off Analysis: The results illustrate that satisfying both constraints incurs a cost in the form of increased Type II error (more missed business opportunities) compared to unconstrained or single-constraint models. However, this cost is presented as a necessary trade-off for achieving regulatory compliance and risk management simultaneously.

5. Significance and Claims

The paper positions itself as a response to the practical need for algorithms that can navigate the conflict between private organizational goals (profit, risk control) and social regulatory requirements (fairness).

Regulatory Perspective: The authors argue that their approach aligns with a regulatory perspective where decision-makers must pursue objectives after obeying specific constraints, rather than modeling a complex social welfare function.
Statistical Control: The significance lies in providing a "principled way to redistribute specific errors" to achieve high-probability statistical guarantees. This addresses the uncertainty inherent in predictive errors and finite sample sizes.
Modesty: The authors explicitly state they do not claim NP-EO is superior to all other paradigms in every context. Instead, it is offered as an alternative framework specifically designed for scenarios where asymmetric efficiency priorities and fairness constraints must be met simultaneously with rigorous statistical confidence.
Future Directions: The paper suggests potential extensions to multiple sensitive attributes, parametric models for small sample sizes, and other efficiency/fairness metrics, but notes these as future work rather than current contributions.

In summary, the paper provides a mathematically rigorous and practically implementable solution for organizations that must balance the risk of financial loss with the imperative of non-discriminatory treatment, offering a framework where both goals can be met with quantifiable statistical certainty.