Technical Summary: Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness

Problem Statement

Predictive machine learning (ML) models are increasingly deployed in high-risk domains (e.g., healthcare, criminal justice) to aid human decision-makers. While Randomized Control Trials (RCTs) provide high-quality evidence on the causal impact of specific ML interventions, a critical gap exists: it is often infeasible to conduct new RCTs for every model update or retraining iteration. Consequently, if a new model with higher predictive accuracy is developed after an initial RCT, there is no direct causal evidence regarding its downstream impact on outcomes (e.g., patient survival, recidivism).

Prior work has attempted to address this via partial identification frameworks. For instance, Nercessian et al. [2025] assume outcomes do not differ between models with agreeing predictions, while Chen and Oberst [2025] relax this by allowing individual-level outcome differences based on global performance metrics (e.g., precision), motivated by user trust. However, these frameworks fail to incorporate individual-level prediction correctness. In many scenarios, it is reasonable to expect that a correct prediction leads to non-inferior outcomes compared to an incorrect one, a notion the authors term "counterfactual correctness."

Methodology

The authors propose a partial-identification approach to construct bounds on the causal effect of a new, untrialed model ($\pi_e$) using data from a prior RCT involving a set of trialed models ($T$).

Causal Framework

The data-generating process is modeled as a Structural Causal Model (SCM) represented by a Directed Acyclic Graph (DAG). Key variables include:

• $D$: RCT arm (control vs. treatment).
• $\Pi$: The model used for prediction.
• $X$: Covariates/features.
• $Z$: The "true label" (ground truth), observed for all individuals in the RCT.
• $\hat{Z}$: The predicted label.
• $M$: A performance metric (e.g., precision) conditioned on $X$ and $\Pi$.
• $A$: Decision-maker action (unobserved/latent).
• $Y$: Downstream outcome.

The goal is to estimate $E[Y(\pi_e)]$, the expected outcome had the untrialed model $\pi_e$ been deployed.

Structural Assumptions

To derive bounds, the authors introduce four key assumptions:

1. Counterfactual Correctness (Assump. 3.1): Correct predictions cause no harm. Formally, for a fixed true label $z$, if a model predicts $z$ (correctly), the outcome is at least as good as if a model predicted $z' \neq z$ (incorrectly), holding other factors constant.

   • Falsification: Can be tested by comparing outcomes of two trialed models on the subset of data where one is correct and the other is wrong.

2. Conditional Performance Monotonicity (Assump. 3.2): Potential outcomes are monotonic with respect to model performance, conditional on correctness.

   • If the prediction is correct ($\hat{z}=z$), higher performance leads to better (or equal) outcomes.
   • If the prediction is incorrect ($\hat{z} \neq z$), higher performance leads to worse (or equal) outcomes (reflecting increased trust in a confidently wrong model).
   • Falsification: Tested by comparing models with different performance levels but identical predictions on the subset where the prediction is correct.

3. Neutral Prediction (Assump. 3.3): There exists a "neutral" prediction (e.g., deferring to a human) where model performance does not influence the outcome. This links the control arm to the evaluation of new models.

4. Bounded Outcomes (Assump. 3.4): The outcome variable $Y$ is bounded within known $[Y_{min}, Y_{max}]$.

Partial Identification Bounds

The authors define partitions of the model and covariate space based on whether trialed models agree with the untrialed model $\pi_e$ and their relative performance. They introduce a branch selector ($b_L, b_U$) that assigns each $(x, z)$ pair to one of four logic branches:

• Neutral: $\pi_e$ makes the neutral prediction.
• Monotonic: $\pi_e$ agrees with trialed models, and performance ordering allows bounding via monotonicity.
• Correct: $\pi_e$ makes the correct prediction, but no trialed model agrees with it (or they are all "better" in a way that doesn't help the lower bound). Here, the bound relies on the assumption that the correct prediction is no worse than any incorrect prediction.
• Default: Conservative fallback to $Y_{min}$ (lower bound) or $Y_{max}$ (upper bound).

The resulting bounds $L(\pi_e)$ and $U(\pi_e)$ are constructed by taking the expectation of the pointwise bounds over the distribution of $X$ and $Z$.

Estimation

The paper provides consistent and asymptotically normal estimators for these bounds. The estimation strategy addresses three challenges:

1. Estimating conditional expectations $E[Y|X, Z, \Pi]$ via nuisance models.
2. Handling non-smooth max/min operators (e.g., selecting the worst/best performing model in a set).
3. Estimating performance metrics ($M$) from finite data rather than assuming oracle knowledge.

The authors propose an Augmented Inverse Propensity Weighted (AIPW) estimator. Under specific margin conditions (ensuring sufficient separation between model performances and outcomes) and convergence rates for nuisance functions, the estimators are proven to be consistent and asymptotically normal, allowing for the construction of valid confidence intervals.

Key Contributions

1. Extension of Causal Frameworks: The work extends the framework of Chen and Oberst [2025] by incorporating individual-level counterfactual correctness. It assumes that if a new model predicts correctly, the outcome is no worse than if an incorrect prediction were made.
2. Subgroup Performance Integration: Unlike prior work relying on global performance metrics, this framework incorporates conditional subgroup predictive performance, reflecting the intuition that decision-makers develop heuristics about where models perform well or poorly.
3. Data-Driven Estimation: The authors provide consistent estimators for the bounds that do not require a priori knowledge of model performance measures; these are estimated from the data.
4. Tighter Bounds: Theoretical and empirical results demonstrate that incorporating counterfactual correctness leads to significantly tighter (more informative) bounds compared to previous partial identification methods.

Experimental Results

The authors evaluate their method using a semi-synthetic simulation based on the Imai et al. [2023] bail sentencing dataset.

• Setup: They simulate judge behavior and defendant outcomes (Failure to Appear) under various alerting models.
• Comparison: They compare their bounds against those derived from Chen and Oberst [2025].
• Findings:
  • In all experiments (changing alert thresholds, re-training models, and absence of RCTs), the proposed method produced consistently narrower bounds (e.g., width of 0.12 vs. 0.36 in one experiment).
  • The bounds consistently contained the true expected outcome.
  • Ablation studies suggest the improvement is primarily driven by the counterfactual correctness assumption rather than just stratifying performance by covariates.
  • The method remains robust (though with wider bounds) even when ground truth labels ($Z$) are partially missing, provided specific assumptions about the missingness mechanism hold.

Significance and Claims

The paper claims to offer a practical solution for evaluating the causal impact of updated ML models when re-running RCTs is infeasible. By leveraging the existence of ground truth labels in prior trials, the method utilizes the "directional" information provided by prediction correctness to tighten causal bounds.

The authors are modest in their claims, acknowledging limitations:

• Falsification Tests: These are in-expectation; individual-level violations of assumptions may go undetected.
• Missing Data: The method has limitations when true labels ($Z$) are missing, though they discuss mechanisms to handle this.
• Continuous Outputs: The current framework assumes discrete predictions; continuous outputs require discretization or simplified application.
• Observational Data: The current framework relies on RCT data; incorporating observational data is noted as a future direction.

Ultimately, the work provides a rigorous, assumption-based framework to move beyond predictive accuracy and reason about the downstream causal impact of AI systems in high-stakes decision-making environments.