Choosing with unknown causal information: Action-outcome probabilities for decision making can be grounded in causal models

1. Problem Statement

The paper addresses a fundamental gap in decision theory regarding the origin of action-outcome probabilities.

• Context: Classical decision theory (von Neumann-Morgenstern and Savage) provides a formal criterion for rational choice: Maximization of Expected Utility (MEU). However, these frameworks rely on associative (probabilistic) information without specifying the source of the probabilities used.
• The Gap: While humans naturally engage in causal reasoning (distinguishing between observing an event and intervening to cause it), standard decision theory often treats probabilities as purely statistical associations.
• The Challenge:
  1. How to formalize decision-making when causal information is known (already addressed by Judea Pearl).
  2. How to formalize decision-making when the causal mechanism governing the environment is unknown to the rational agent.
  3. How to ground the probabilities required for MEU in causal models rather than just observational data, particularly in cases of incomplete causal knowledge.

2. Methodology and Theoretical Framework

The authors build a bridge between Classical Decision Theory and Causal Inference (specifically Pearl's Causal Graphical Models).

A. Fundamental Definitions

• Causal Environment: Defined as an uncertain environment $(\Omega, A, C, E)$, extended with a Causal Graphical Model (CGM) $G$. The CGM encodes causal relationships between actions ($A$) and consequences ($C$) as well as other environmental variables.
• Intervention ($do(\cdot)$): The core mechanism distinguishing causal from associative models. An action $a$ is modeled as an intervention $do(a)$, which modifies the joint distribution of variables by fixing the value of a variable and removing incoming edges to that node in the graph.
• Causal Decision Problem (CDP): A tuple $(\Omega, A, G, E, C, \succeq)$, where the decision-maker chooses an action to maximize utility based on the causal model.

B. Theoretical Approach

The paper proceeds in two main steps:

1. Overview of known causal information: It acknowledges Pearl's (2009) result that a rational agent with a known causal model $G$ chooses action $a^*$ to maximize $\sum P_G(c | do(a))u(c)$.
2. Extension to unknown causal information: The methodological core contribution is extending this to scenarios where the agent does not know the true causal model $G$.
   • Hypothesis: A rational agent with unknown causal structure holds subjective beliefs about a family of possible causal structures ($\mathcal{F}$).
   • Mechanism: The agent considers each possible causal structure $g \in \mathcal{F}$ as potential truth, calculates the expected utility within that structure using the $do$ operator, and then aggregates these utilities weighted by the probability the agent assigns to each structure.

3. Main Contributions

A. Theorem 4.2: Decision-making with unknown causal models

The primary theoretical contribution of the paper is Theorem 4.2, which provides a representation theorem for rational decision-making under unknown causal information.

• Statement: A preference relation $\succeq$ over actions in a finite, bounded causal decision problem is rational if and only if there exists:

  1. A utility function $u: C \to \mathbb{R}$.
  2. A probability distribution $P_C$ over a non-empty family of causal graphical models $\mathcal{F}$.

• Decision Rule: The agent chooses action $a$ over $b$ if and only if:
  $$\sum_{c \in C} u(c) \left( \sum_{g \in \mathcal{F}} P_g(c | do(a)) P_C(g) \right) \geq \sum_{c \in C} u(c) \left( \sum_{g \in \mathcal{F}} P_g(c | do(b)) P_C(g) \right)$$
  Where:

  • $P_C(g)$ is the agent's subjective belief that causal structure $g$ is the true model.
  • $P_g(c | do(a))$ is the probability of consequence $c$ given intervention $do(a)$ within the specific causal model $g$.

• Significance: This proves that action-outcome probabilities can be grounded in causal models even when the specific causal mechanism is unknown. The "probability" used in the expected utility calculation is a mixture distribution over causal structures.

B. Definition of Causal Nash Equilibrium

The authors apply Theorem 4.2 to game theory to define a Causal Nash Equilibrium.

• Context: In a Bayesian strategic game, players have incomplete information about the true causal model $\omega$ governing the environment.
• Definition: An action profile $a^*$ is a Causal Nash Equilibrium if no player $i$ can improve their Causal Expected Utility ($u^C_i$) by unilaterally changing their action, given their beliefs about the causal structure and the actions of others.
• Formula:
  $$u^C_i(a) = \sum_{c \in C} u_i(c) \sum_{\omega \in \Omega} P^\omega_i(c | do(a_i), a_{-i}) P_i(\omega)$$
  This formulation explicitly integrates the $do$ operator and thus differs from standard Nash equilibria, which are based on conditional probabilities $P(c | a_i, a_{-i})$.

4. Results

1. Formal Grounding: The paper successfully demonstrates that the probabilities required for the principle of expected utility can be derived from causal models, resolving the debate regarding the "origin" of these probabilities.
2. Generalization: The result generalizes Pearl's criterion for known causal models to the more realistic scenario of unknown causal mechanisms.
3. Computational Feasibility: The framework provides a computationally implementable criterion. An agent can learn a distribution over causal structures (e.g., via structure learning algorithms) and use the derived formula to make optimal decisions.
4. Game-Theoretic Extension: The paper establishes that game-theoretic concepts like the Nash equilibrium can be rigorously extended to environments where agents engage in causal rather than associative reasoning.

5. Significance and Implications

• Connecting AI and Philosophy: The work connects the philosophical debate between Causal Decision Theory (CDT) and Evidential Decision Theory (EDT) with practical machine learning and AI decision-making. It provides a mathematical justification for why agents should use causal reasoning.
• Robustness to Distributional Shifts: By grounding decisions in causal mechanisms (interventions) rather than correlations, the framework is theoretically more robust to changes in the environment (distributional shifts) where purely associative models would fail.
• Human-Centric AI: Since humans naturally use causal reasoning (and tend to ignore pure probabilistic information in favor of causal explanations), this framework offers a more "human-compatible" basis for designing rational AI agents.
• Strategic Interaction: The introduction of the Causal Nash Equilibrium opens new avenues for analyzing strategic interactions in economics and multi-agent systems where the underlying causal dynamics are uncertain.

Conclusion

The paper argues that rational decision-making under uncertainty is fundamentally a causal process. By extending the principle of expected utility to include a distribution over causal models, the authors provide a rigorous mathematical foundation for decision-making when causal mechanisms are unknown, ensuring that action-outcome probabilities are grounded in the structural reality of the environment rather than mere statistical association.