Reliable fairness auditing with semi-supervised inference

Technical Summary: Reliable Fairness Auditing with Semi-Supervised Inference

Problem Statement
Machine learning models deployed in biomedical applications often exhibit bias that exacerbates inequities across subpopulations defined by protected attributes (e.g., race, sex). Identifying and mitigating these biases requires "group fairness auditing," a process that evaluates model performance metrics (such as True Positive Rate, False Positive Rate, or calibration) across different groups. However, reliable auditing typically demands large volumes of labeled data to detect subtle performance disparities. In biomedical contexts, obtaining such labels is costly and labor-intensive, often requiring specialized clinical expertise (e.g., annotating medical images or discharge summaries). Consequently, fairness audits frequently operate in a semi-supervised (SS) setting: a small labeled dataset is available, accompanied by a much larger unlabeled dataset. Traditional supervised auditing methods ignore the unlabeled data, leading to high variance and reduced power to detect disparities. Existing SS methods for fairness auditing are either limited to specific metrics, rely on parametric assumptions that may lead to biased inference if violated, or fail to leverage auxiliary covariates available during the audit.

Methodology: The Infairness Framework
The authors propose Infairness, a unified semi-supervised inference framework designed to audit a wide range of fairness criteria by leveraging both labeled and unlabeled data. The core of the method involves imputing missing outcomes ($Y$) in the unlabeled data using a regression model that conditions on the model's predictions ($S$) and auxiliary covariates ($W$).

1. Problem Setup: The framework assumes a validation dataset consisting of $n$ labeled samples $\{(Y_i, X_i, W_i, A_i)\}$ and $N$ unlabeled samples $\{(X_i, W_i, A_i)\}$, where $A$ is a categorical protected attribute. The goal is to estimate group-specific performance metrics ($M_a$) and their differences ($\Delta M$) with higher efficiency than supervised estimation.
2. Imputation Strategy: For metrics dependent on the outcome $Y$ (e.g., TPR, FPR), the method estimates the conditional expectation $E(Y | S, W, A=a)$ using a working model. This model utilizes flexible nonlinear basis functions (specifically natural cubic splines) of the prediction $S$ and auxiliary covariates $W$. The parameters are estimated via a penalized estimating equation on the labeled data.
3. Estimation: The missing outcomes in the unlabeled data are imputed using the fitted model. The group-specific metrics are then calculated by averaging these imputations over the large unlabeled dataset within each group. For metrics independent of $Y$ (e.g., those depending only on $S$ or $D$), the method simply averages over the unlabeled data, which inherently reduces variance due to the larger sample size.
4. Robustness: The framework is designed to be robust to the specification of the imputation model. By including an intercept, the prediction $S$, and the classification $D$ in the basis functions, the estimator remains consistent even if the functional form of the imputation model is misspecified, provided the basis includes these key terms.

Key Contributions and Theoretical Results

• Unified Framework: Infairness provides a general approach applicable to a broad spectrum of fairness metrics (accuracy, discrimination, calibration) and does not rely on the restrictive parametric assumptions of previous SS fairness methods (e.g., Beta Calibration).
• Consistency and Validity: The authors prove that the Infairness estimator is consistent for the target fairness metrics and that the inference procedures (confidence intervals) are valid under standard regularity conditions, even when the imputation model is not perfectly specified.
• Efficiency Gains: Theoretical analysis demonstrates that the Infairness estimator has a smaller asymptotic variance than the supervised estimator. Specifically, if the imputation model captures information about $Y$ beyond what is contained in $S$ alone (i.e., $E(Y|S, W, A) \neq E(Y|A)$), the variance is strictly reduced.
• Robustness to Misspecification: Unlike methods that assume a specific parametric form for the relationship between predictions and outcomes, Infairness uses flexible basis expansions. Theoretical results show that the estimator remains consistent even if the working model is misspecified, as long as the basis functions include necessary terms (intercept, $S$, $D$).

Empirical Results
The paper evaluates Infairness through extensive simulation studies and two real-world biomedical audits:

1. Simulation Studies:

   • In stylized settings, Infairness consistently provided unbiased estimates with nominal coverage probabilities, whereas parametric methods (Beta Calibration) suffered from significant bias and under-coverage when their assumptions were violated.
   • In traditional settings with complex data generating mechanisms (non-linearities, interactions, tree-based structures), Infairness maintained low bias and near-nominal coverage.
   • Efficiency: Infairness consistently outperformed supervised estimation and nonparametric baselines. When auxiliary covariates ($W$) contained information about $Y$, models incorporating $W$ (e.g., Infairness($S \times W$)) achieved relative efficiency gains of 10–37% over methods ignoring $W$.

2. Real-World Applications:

   • EHR Phenotyping (Depression): Auditing a model for depression detection across race groups using MIMIC-III data ($n=872$ labeled, $N=28,766$ unlabeled). Infairness reduced the variance of disparity estimates by approximately 43.5% compared to supervised estimation, equivalent to requiring 43.5% fewer labeled samples to achieve the same precision.
   • Medical Imaging (Chest X-ray): Auditing a cardiomegaly classifier across sex groups ($n=500$ labeled, $N=64,740$ unlabeled). Infairness reduced variance by 60.9% for TPR disparity and 56.3% for FPR disparity compared to supervised estimation.

Significance and Claims
The paper claims that Infairness offers a practical and statistically rigorous solution for the "label scarcity" bottleneck in biomedical fairness auditing. By effectively utilizing abundant unlabeled data and auxiliary covariates, the method enables more powerful detection of subgroup performance disparities without increasing the cost of data annotation. The authors emphasize that the method does not sacrifice validity for efficiency; it guarantees consistent estimation and valid inference without relying on strong parametric assumptions that could lead to overconfident, distorted conclusions about model fairness. The work underscores the value of semi-supervised inference in ensuring that ML models in healthcare are deployed responsibly and equitably.