Fairness under uncertainty in sequential decisions

1. Problem Statement

The paper addresses a critical gap in Fair Machine Learning (Fair ML): while fairness in static, supervised learning is well-studied, sequential and online decision-making systems (e.g., lending, hiring, insurance) face unique challenges due to uncertainty and selective feedback.

• The Core Issue: In sequential settings, decisions are made over time, and the outcome of a decision (e.g., granting a loan) determines whether data is observed. If a loan is denied, the bank never observes whether the applicant would have repaid (the counterfactual outcome).
• The Inequality of Uncertainty: Historically marginalized groups often suffer from representation bias (less historical data) and selective feedback (systematically denied opportunities). This leads to higher prediction uncertainty for these groups.
• The Consequence: Naïve policies that ignore this uncertainty tend to exploit known safe patterns, systematically excluding underrepresented groups to minimize immediate risk. This creates a feedback loop where the lack of data reinforces the lack of opportunities, compounding disparities over time.
• Legal & Practical Risks: The authors highlight that Reinforcement Learning (RL) introduces randomness (exploration) which can conflict with non-discrimination laws if not managed carefully, yet failing to explore prevents the system from learning the true potential of marginalized groups.

2. Methodology

A. Taxonomy of Uncertainty

The authors propose a structured taxonomy of uncertainty across the ML lifecycle to diagnose where disparities arise. They categorize uncertainty into six types, distinguishing between "global" (systemic) and "local" (individual) sources:

1. World: Desired state uncertainty (subjective definition of acceptable inequality).
2. Data Collection: Representation uncertainty (skewed sampling).
3. Feature Engineering: Measurement uncertainty (proxy variables).
4. Model Training: Model uncertainty (sub-optimal parameters).
5. Predictions: Prediction uncertainty (variance in prediction confidence for specific individuals).
6. Deployment: Feedback uncertainty (unobserved counterfactual outcomes).

Focus: The paper specifically targets Model, Prediction, and Feedback uncertainties, as these directly influence the sequential decision loop.

B. Formalization

The problem is modeled as a Contextual Bandit (a simplified RL setting) where:

• State ($s$): Applicant profile $x$, sensitive attribute $A$, and constraints.
• Action ($d$): Grant ($1$) or Deny ($0$) a loan.
• Outcome ($Y_1$): Repayment ($1$) or Default ($0$).
• Utility ($u$): The authors define a utility function that accounts for both observed outcomes (realized gains/losses) and unobserved counterfactual outcomes (unrealized gains/losses).
  $$u = d(y_1 g - (1-y_1)l) + (1-d)((1-y_1)g - y_1 l)$$
  Where $g$ is gain, $l$ is loss, and the second term represents the counterfactual utility of a denied applicant.

C. Proposed Approaches

The authors compare several strategies to mitigate these uncertainties without imposing explicit fairness constraints (like demographic parity):

1. Naïve Policy: A standard logistic regression model that only learns from observed data (approved loans). It ignores counterfactuals and uncertainty.
2. Naïve Exploration: Randomly selects a subset of applicants in the "uncertain region" (near the decision threshold) to grant loans, regardless of group, to gather data.
3. Probability-Weighted Exploration: Samples applicants for exploration based on their predicted probability of repayment, prioritizing those the model is unsure about but might be good candidates.
4. Uncertainty-Aware Exploration: Specifically targets applicants where the confidence interval of the prediction overlaps with the decision threshold. This explicitly accounts for prediction uncertainty.
5. Counterfactual Utility Maximization: The core proposed method. It trains a model end-to-end by maximizing a differentiable objective that includes both observed rewards and expected counterfactual rewards.
   • It uses a soft decision function ($\sigma(\alpha(p_t - \tau_t))$) to approximate hard thresholds, allowing gradient-based optimization.
   • It explicitly estimates the utility of denied loans based on the model's predicted probability, effectively "filling in" the missing data.

3. Key Contributions

1. Taxonomy of Uncertainty: Provides a shared vocabulary for researchers and practitioners to identify how different types of uncertainty (epistemic vs. aleatoric, global vs. local) interact to produce unfairness in sequential systems.
2. Formalization of Fairness under Uncertainty: Conceptualizes how unequal epistemic uncertainty (due to lack of data) and feedback uncertainty (due to selective labels) compound to create structural disparities.
3. Demonstration of Uncertainty-Aware Exploration: Shows that simply accounting for uncertainty (specifically prediction and feedback uncertainty) can improve fairness metrics for disadvantaged groups without sacrificing the decision-maker's utility (profit) and without requiring explicit fairness constraints (which can be legally contentious).
4. Counterfactual Optimization: Introduces a method to optimize for counterfactual outcomes in RL settings, bridging the gap between observed data and potential outcomes.

4. Experimental Results

The authors used synthetic data (based on the framework by Baumann et al.) simulating historical bias, measurement bias, and representation bias. They evaluated performance over 10 quarters (time steps).

• Performance vs. Fairness:
  • In bias-free settings, all methods performed similarly, proving that uncertainty-aware methods do not inherently degrade utility.
  • In settings with historical and measurement bias, the Counterfactual Utility Maximization method significantly outperformed others.
• Fairness Metrics:
  • The Counterfactual method achieved better Selection Rate Differences, False Negative Rate (FNR) Differences, and Accuracy Differences between groups compared to Naïve and standard exploration methods.
  • Crucially, it achieved these fairness improvements while maintaining or slightly improving cumulative profit (utility).
• Sensitivity Analysis:
  • As the magnitude of measurement bias increased, the Counterfactual method maintained high utility while other methods saw their fairness metrics degrade.
  • The results demonstrate that ignoring unrealized gains (counterfactuals) penalizes both the lender (lost profit) and the customer (lost opportunity), especially for marginalized groups.

5. Significance and Implications

• Beyond Static Fairness: The paper argues that fairness cannot be treated as a static constraint in sequential systems. The process of learning (exploration) is where bias accumulates.
• Legal and Governance: By framing the problem through uncertainty, the authors provide a tool for auditors and regulators to identify where "known unknowns" (uncertainty) are driving disparate impacts. It suggests that uncertainty-aware exploration can be a legally defensible alternative to affirmative action, as it relies on statistical necessity (reducing uncertainty) rather than group preference.
• Practical Application: The approach offers a "diagnostic lens" for practitioners. Instead of just tuning hyperparameters for fairness, they can audit the decision pipeline to see where uncertainty is unevenly distributed and apply targeted exploration to correct it.
• Future Directions: The paper highlights the need for better legal frameworks regarding RL in high-stakes domains and calls for further research into continuous settings and causal integration.

In summary, the paper posits that uncertainty is a structural driver of unfairness in sequential decision-making. By explicitly modeling and exploring the "unobserved space" (counterfactuals) and accounting for prediction variance, decision-makers can simultaneously optimize for institutional utility and social equity.