2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support

Technical Summary: 2-Step Agent: How a Bayesian Decision Maker Learns from AI-Decision Support

Problem Statement
The paper addresses the lack of a clear understanding of how human decision-makers learn from Machine Learning Decision Support (ML-DS). While ML predictions are increasingly used to support high-stakes decisions in fields like healthcare and the judiciary, the mechanism by which agents incorporate these predictions into their mental models of the world remains underexplored. Specifically, the authors note that an ML prediction contains compressed information about the training data and the underlying causal mechanisms. Consequently, a rational agent can use a prediction not just as a forecast, but as evidence to update their beliefs about the population and the treatment effect. The core problem is modeling this learning process and determining under what conditions ML-DS improves or harms downstream outcomes.

Methodology: The 2-Step Agent Framework
The authors introduce a computational framework called the "2-Step Agent" to model this process. The framework consists of two distinct steps:

1. Bayesian Belief Update (Step 1): The agent possesses a prior belief about the data-generating process (the historical Structural Causal Model, or SCM) and the ML model’s training procedure. Upon observing a new patient’s covariates ($X_o$) and the ML model’s prediction ($Pred_o$), the agent performs a Bayesian update. This involves inferring the posterior distribution over population-level parameters (such as the distribution of covariates, the historical treatment policy, and the treatment effect) by treating the ML prediction as evidence derived from latent, interchangeable training variables. This step is computationally challenging because it requires inference through a "plate" of $n$ training instances.
2. Causal Inference and Decision (Step 2): Using the updated posterior beliefs from Step 1, the agent estimates the Conditional Average Treatment Effect (CATE) for the new patient. The agent then selects an action (e.g., treatment dosage) that maximizes expected utility based on this estimated CATE.

To make the Bayesian inference tractable, the authors derive a theoretical result for the linear Gaussian setting. They prove that the likelihood of the ML model’s parameters can be expressed as a function of population-level variables and a small set of sufficient statistics, rather than requiring sampling over the entire training plate. This reduction allows for efficient simulation of the agent’s learning dynamics.

Key Contributions

1. Framework Formulation: The paper presents the first explicit computational framework modeling how a Bayesian agent updates beliefs about a population and treatment mechanism after observing a prediction from an ML model trained on that population.
2. Tractability Result: The authors provide a proof (Theorem 2.8) showing that in a linear Gaussian setting, the complex Bayesian inference problem can be reduced to sampling a few sufficient statistics. This reduces computational cost and improves numerical stability, enabling end-to-end simulation.
3. Experimental Identification of Conditions: Through simulations, the authors identify specific conditions under which ML-DS is beneficial versus harmful.
4. Demonstration of Harm from Misaligned Priors: The paper shows that even with a well-specified ML model and a perfectly rational agent, a single misaligned prior belief can cause ML-DS to lead to worse outcomes than having no decision support at all.

Results
The authors conduct experiments using a univariate regression setup with no bias term. They simulate agents with varying prior beliefs regarding the treatment effect ($NE$), the historical treatment policy ($\alpha_{A\mu}$), and the covariate distribution ($\alpha_{X\mu}$).

• Corrected Beliefs: When the agent’s prior belief about the treatment effect ($NE$) is misaligned but other beliefs are correct, the agent learns from the ML-DS interaction. The posterior belief aligns with the true value, leading to better CATE estimates and superior downstream outcomes compared to no decision support.
• Harmful Learning: When the agent’s prior belief about the historical treatment policy ($\alpha_{A\mu}$) or the covariate distribution ($\alpha_{X\mu}$) is misaligned, the learning process can be detrimental. The agent attempts to reconcile the ML prediction with their incorrect priors, leading to erroneous updates in their belief about the treatment effect. This results in suboptimal treatment decisions and worse outcomes than if the agent had relied solely on their priors without ML-DS.
• Sensitivity: The results highlight that ML-DS is highly sensitive to the agent’s prior beliefs. Incorrect beliefs about the training data distribution can cause the agent to draw incorrect causal inferences from the prediction, thereby amplifying harm.

Significance and Claims
The paper claims to provide the first quantitative estimation of the harmful effects of ML-DS due to misaligned beliefs. It underscores the importance of the agent’s prior beliefs in determining the effectiveness of decision support systems. The authors argue that even under ideal conditions (rational agent, well-specified model), ML-DS can do more harm than good if the agent’s priors about the training data are misaligned. This finding challenges the assumption that ML-DS is inherently beneficial and highlights the need for proper documentation and training to ensure agents have aligned priors about the data-generating process and the model’s training context. The framework is presented as a general tool applicable to any field employing prediction models for decision support, though the current simulations are limited to non-causal, linear prediction models.