Technical Summary: Evaluating Prediction-based Interventions with Human Decision Makers In Mind

Problem Statement

Automated Decision Systems (ADS) are increasingly deployed to support human decision-making in consequential settings such as criminal justice, healthcare, and education. While machine learning research often focuses on evaluating predictive model accuracy against benchmark datasets, these assessments fail to capture the complex interaction between humans and algorithms. Specifically, current experimental designs for evaluating prediction-based interventions rely on simplifying assumptions about human decision-making to derive causal estimates.

The core problem identified is that specific experimental design choices can induce cognitive biases in human decision-makers, significantly altering observed effect sizes. Existing paradigms often assume a "case-independent" model where the treatment status of one decision subject does not affect another (adhering to the Stable Unit Treatment Value Assumption, or SUTVA). However, this paper argues that human decision-makers (e.g., judges) possess internal states that evolve based on their exposure to algorithmic recommendations, creating dependencies across decision subjects that violate SUTVA and lead to biased treatment effect estimation.

Methodology

Theoretical Framework

The authors propose a novel causal model of algorithm-aided human decision-making that explicitly accounts for the decision-maker's internal state. Unlike the standard case-independent model, this framework indexes decisions by both the decision subject ($i$) and the decision-maker ($k$).

The model introduces two latent variables:

1. $J_{i,k}$: The responsiveness of judge $k$ for treatment unit $i$, representing the probability that the judge follows the algorithmic recommendation.
2. $\epsilon_{i,k}$: A random variable drawn from a Bernoulli distribution with parameter $J_{i,k}$, denoting whether the judge complies with the recommendation.

The decision function $D_{i,k}$ is defined as a function of the judge's default decision $\lambda_k(X_i)$ and the algorithmic recommendation $\bar{D}(\hat{Y}_i)$, modulated by the responsiveness $J_{i,k}$.

Cognitive Bias Models

The paper formalizes three specific models of cognitive bias that make judge responsiveness ($J_{i,k}$) dynamic and dependent on experimental design choices:

1. Treatment Exposure Model: Responsiveness increases as the average exposure to the predictive aid increases. Judges may rely on consistent information rather than adjusting protocols for each case, leading to higher compliance after frequent exposure.
2. Capacity Constraint Model: Responsiveness decreases as the rate of positive (high-risk) predictions increases. In high-stakes environments, decision-makers may reach a threshold where they become less responsive to further "high-risk" alerts.
3. Low Trust Model: Responsiveness decreases as the observed error rate of the algorithm increases. If judges observe realized outcomes and detect prediction errors (e.g., false positives), their trust and subsequent compliance diminish.

Experimental Analysis

The authors mathematically prove that under these models, the decision $D_{i,k}$ is not conditionally independent of prior treatment assignments $(Z_{1,k}, \dots, Z_{i-1,k})$ given the current treatment $Z_{i,k}$. This constitutes a violation of SUTVA.

To demonstrate the practical impact, the paper compares two randomization schemes:

• Uniform Randomization: Cases are randomized at the individual level (single-level), resulting in mixed treatment exposure for each judge.
• Two-Level Randomization: Judges are randomized to treatment clusters (e.g., one judge sees 100% treated cases, another 0%), creating distinct exposure profiles.

Semi-Synthetic Experiments

The authors validate their theoretical findings using data from a real-world Randomized Control Trial (RCT) by Imai et al. [2020] regarding Public Safety Assessment (PSA) risk scores in Dane County, Wisconsin. They simulate multiple judges and apply the three bias models to generate alternative decision sequences. They observe how modifying experimental design choices (treatment assignment, model thresholds, and accuracy) alters the estimated Average Treatment Effect (ATE).

Key Contributions

1. Formalization of Dynamic Responsiveness: The paper moves beyond the assumption that judge responsiveness is a static, inherent attribute. It formalizes how responsiveness is a dynamic function of experimental design choices, specifically treatment assignment ($Z$), model positive prediction rates ($P(\hat{Y}=1)$), and model correctness ($P(\hat{Y}=Y)$).
2. Proof of SUTVA Violation: The authors provide mathematical proofs showing that the proposed behavioral models induce interference between decision subjects, violating the SUTVA assumption commonly used in causal inference. This invalidates standard estimators that assume independence.
3. Demonstration of Estimation Bias: The paper demonstrates that standard randomization schemes (single-level) can lead to significant underestimation or overestimation of the Average Treatment Effect (ATE) depending on the specific bias model active. For instance, under the treatment exposure model, uniform randomization underestimates the ATE, whereas under the capacity constraint model, it may overestimate it.
4. Empirical Simulation: Using semi-synthetic experiments based on Imai et al. [2020], the authors show that modifying design choices (e.g., changing the proportion of treated cases per judge or the algorithm's threshold) can alter causal effect estimates by up to 10x for specific demographic subgroups.

Results

• SUTVA Violation: Theoretical analysis confirms that when judge responsiveness depends on historical exposure, the treatment status of one case influences the decision on subsequent cases handled by the same judge. Standard permutation tests used in previous studies (like Imai et al., 2020) fail to detect this interference because they preserve the average treatment proportion, rendering the order of cases irrelevant to the test statistic.
• Bias Directionality:
  • Treatment Exposure: Uniform randomization leads to an underestimation of the treatment effect compared to two-level randomization. The gap widens as judges become more sensitive to past exposure and the divergence between default and recommended decisions increases.
  • Capacity Constraint: Uniform randomization leads to an overestimation of the treatment effect. The gap increases with higher positive prediction rates.
  • Low Trust: Lower algorithmic accuracy leads to reduced responsiveness and an underestimation of the treatment effect, particularly impacting measures of decision correctness.
• Simulation Findings: In the semi-synthetic experiments, changing the treatment exposure regime (e.g., assigning 100% of cases to a judge vs. 50%) resulted in substantial deviations in the measured ATE. Specifically, for non-white female subjects, the estimated causal effect on decisions varied by a factor of 10 depending on the exposure regime.

Significance and Claims

The paper claims that the scientific validity of intervention-based evaluation schemes for ADS deployments is currently compromised by ignoring the dynamic nature of human decision-making. By treating prediction-based interventions as isolated prediction problems rather than policy interventions, researchers risk drawing incorrect conclusions about the efficacy of these systems.

The authors argue that future experimental designs must account for the "human-in-the-loop" as a dynamic agent whose behavior is shaped by the experiment itself. They suggest that:

• Standard single-level randomization is insufficient for estimating the causal impact of predictive systems when deployed as full-scale policies.
• Two-level randomization designs (randomizing at the decision-maker level) or multi-factor designs are necessary to accurately estimate treatment effects in the presence of judge bias.
• The findings extend beyond criminal justice to other domains like healthcare and education, where similar cognitive biases and interference effects are likely present.

The work serves as a call to action for the research community to refine experimental paradigms to better reflect the realities of human-algorithm interaction, ensuring that causal estimates reflect the true impact of predictive interventions on downstream outcomes.