Technical Summary: Unequal Uncertainty: Rethinking Algorithmic Interventions for Mitigating Discrimination from AI

1. Problem Statement

The paper addresses the legal and ethical risks associated with predictive uncertainty in Artificial Intelligence (AI) systems used for high-stakes decision-making (e.g., consumer credit and criminal justice). While machine learning (ML) models often exhibit similar aggregate performance, predictive uncertainty is frequently unevenly distributed across demographic groups, often concentrated in under-represented populations due to data sparsity or historical bias.

The core problem is how to manage this uncertainty without exacerbating discrimination. The authors analyze two common algorithmic interventions designed to act as "guardrails" for human-AI interaction:

1. Selective Abstention: Withholding model predictions from human decision-makers when uncertainty exceeds a threshold, forcing a decision based solely on unaided human judgment.
2. Selective Friction: Presenting the model prediction to the human decision-maker but flagging it with a warning (e.g., a "yellow flag") when uncertainty is high, prompting reflection without removing the prediction.

The paper argues that while these interventions aim to mitigate errors, they may inadvertently generate unlawful discrimination under United Kingdom (UK) law, particularly by creating procedural asymmetries where under-represented groups are disproportionately subjected to unaided human judgment or biased overrides.

2. Methodology

The authors employ a hybrid methodology combining formal technical definitions with doctrinal legal analysis and case study application.

2.1 Formal Definitions and Assumptions

• Task: Binary classification ($\hat{y} \in \{0, 1\}$) where a model $f$ predicts labels for inputs $x$.
• Uncertainty Metric: The authors use Shannon entropy ($H(\hat{p})$) to quantify predictive uncertainty, where $\hat{p}$ is the model's predicted probability. Entropy is 0 for certain predictions and 1 for maximum uncertainty ($\hat{p}=0.5$).
• Intervention Threshold: A threshold $\tau$ is set such that if $H(\hat{y}) \geq \tau$, an intervention is triggered.
• Calibration Assumption: The model is assumed to be well-calibrated at the population level but potentially poorly calibrated within specific demographic subgroups (within-group calibration failure). This leads to systematic differences in entropy values across groups.
• Human Decision-Maker: Modeled as a function $r(x)$ (unaided) or $r(x; f(x))$ (aided). The paper assumes decision-makers may exhibit cognitive biases, such as automation bias (over-reliance) or inappropriate overrides.

2.2 Case Studies

The interventions are applied to two specific domains to analyze consequences for affected groups:

1. Consumer Credit: Predicting loan default risk. The study examines how abstention or friction affects loan approval/denial rates and processing times for under-represented groups.
2. Risk of Reoffending: Predicting the likelihood of recidivism in criminal justice. The study analyzes sentencing outcomes (strict vs. light) and the impact of the Public Sector Equality Duty (PSED).

2.3 Legal Framework

The analysis is grounded in UK law, specifically:

• The Equality Act 2010: Analyzing direct and indirect discrimination, focusing on whether interventions constitute a "Provision, Criterion, or Practice" (PCP) that places protected groups at a particular disadvantage.
• Public Sector Equality Duty (PSED): Section 149 obligations for public authorities.
• Data Protection Law: The UK GDPR and the newly proposed Data (Use and Access) Act 2025 (DUAA), specifically regarding "meaningful human involvement" and solely automated decision-making.

3. Key Contributions

The paper makes two primary contributions:

1. Doctrinal Legal Analysis: It provides the first doctrinal analysis of uncertainty-based algorithmic interventions under UK law. It demonstrates that formally neutral uncertainty thresholds can generate discriminatory effects, particularly under the framework of indirect discrimination.
2. Comparative Evaluation of Interventions: It argues that selective friction is legally preferable to selective abstention. While both carry risks, friction preserves access to the model's prediction and is more likely to satisfy the proportionality test required under the Equality Act 2010.

4. Results and Analysis

4.1 Analysis of Selective Abstention

• Mechanism: When uncertainty is high, the model prediction is hidden, and the human makes a decision without AI assistance.
• Discriminatory Risk: Because under-represented groups often have higher predictive uncertainty (due to data bias), they are disproportionately diverted into "unaided" decision tracks.
• Legal Finding: This creates a procedural asymmetry. Applicants from protected groups face longer processing times and the variability of individual human judgment, while others receive standardized AI-assisted decisions. The authors argue this constitutes indirect discrimination under the Equality Act 2010. It is difficult to justify as "proportionate" because less discriminatory alternatives (like recalibration or friction) exist that do not discard the model's information.
• Outcome: Abstention does not remove the risk of discrimination; it shifts the burden of uncertainty onto specific groups.

4.2 Analysis of Selective Friction

• Mechanism: The model prediction is always shown, but high-uncertainty predictions are accompanied by a warning flag.
• Discriminatory Risk: The risk lies in how humans interpret the flag. If decision-makers (e.g., loan officers or judges) systematically override uncertain predictions in a way that correlates with protected characteristics (e.g., denying loans to flagged applicants from minority groups), discrimination occurs.
• Legal Finding: Selective friction is less likely to be found unlawful than abstention. It preserves the "AI-assisted" nature of the decision, avoiding the two-track system. It is more likely to satisfy the proportionality requirement because it is a less restrictive means of achieving the legitimate aim of error reduction. However, its lawfulness is contingent on empirical evidence that it does not systematically worsen outcomes for protected groups in practice.
• Human Involvement: Under the proposed DUAA 2025, friction may satisfy the requirement for "meaningful human involvement," potentially exempting decisions from stricter automated decision-making safeguards. However, the authors warn this could lead to "tokenistic" compliance if humans rubber-stamp the model despite the flag.

4.3 Case Study Scenarios

The paper outlines four scenarios for both credit and sentencing:

1. Agreement (Uncertain $\to$ Adverse): Human agrees with uncertain negative prediction. Risk of over-reliance on a biased model.
2. Override (Uncertain $\to$ Adverse): Human overrides uncertain positive prediction to give an adverse outcome. Identified as the most troubling scenario for discrimination, as human bias may override the model's (potentially more neutral) suggestion.
3. Agreement (Uncertain $\to$ Favorable): Human agrees with uncertain positive prediction.
4. Override (Uncertain $\to$ Favorable): Human overrides uncertain negative prediction to give a favorable outcome. While theoretically beneficial, empirical evidence suggests overrides do not systematically benefit under-represented groups.

5. Significance and Claims

The paper claims significance as the first to rigorously apply UK legal doctrine to uncertainty-based interventions. Its central thesis is that technical neutrality (using entropy thresholds) does not guarantee legal neutrality.

• Legal Preference: The authors conclude that selective friction is the superior design choice from a legal compliance perspective, as it maintains the integrity of the human-AI collaboration and avoids the procedural disadvantages inherent in abstention.
• Empirical Uncertainty: The paper explicitly states that whether selective friction improves decision quality in practice is an open empirical question. It depends on the training of decision-makers, the calibration of the uncertainty estimates, and the institutional culture.
• Future Work: The authors identify the need for empirical studies on whether uncertainty flags trigger different override rates across demographic groups and technical research into whether group-specific calibration can mitigate the disproportionate impact of these interventions.

The paper does not claim that uncertainty-based interventions are inherently unlawful, but rather that their implementation must be carefully scrutinized to ensure they do not violate the Equality Act 2010 by disproportionately burdening protected groups.