Technical Summary: Robust Counterfactual Policy Optimisation via Nondeterministic Causal Models

1. Problem Statement

Counterfactual inference in sequential decision-making seeks to determine what outcomes would have occurred had a different policy been executed, given an observed trajectory of states, actions, and outcomes. This is critical in safety-critical domains like healthcare, where rerunning experiments is often unethical or infeasible.

Existing approaches to counterfactual inference in Markov Decision Processes (MDPs) typically model the environment as a Deterministic Structural Causal Model (SCM). In this framework, all randomness is attributed to unobserved latent variables ($U$). However, real-world MDPs are inherently stochastic; randomness often arises from two distinct sources:

1. Latent Confounding: Unobserved factors (e.g., patient genetics or demographics) that influence transitions.
2. Irreducible Stochasticity: Fundamental randomness in the system (e.g., random number generators or aggregate noise) that cannot be reduced to latent variables.

The paper addresses the gap where both sources coexist. Standard SCMs fail to distinguish between these, potentially misattributing irreducible noise to latent confounding or vice versa. The authors propose formalizing counterfactual policy optimization under Probabilistic Nondeterministic Causal Models (PNSCMs), which allow for inherent stochasticity while explicitly bounding the influence of latent confounders via sensitivity analysis.

2. Methodology

2.1 Probabilistic Nondeterministic Causal Models (PNSCMs)

The authors model the environment as a PNSCM, replacing deterministic structural equations with conditional probability distributions. A key feature of PNSCMs is actualised refinement: given an observed setting $(u, v)$, the conditional distribution $P_X$ is refined to deterministically return the observed outcome $x$ when given the observed parents, while retaining its probabilistic nature for other inputs. This captures the semantics that we only learn the causal mechanism from the actual setting, not from counterfactual possibilities.

2.2 Sensitivity Analysis Framework

To handle unobserved global confounders ($U$) that remain fixed throughout a trajectory, the authors employ a sensitivity analysis approach similar to Kausik et al. [2024].

• Sensitivity Parameter ($\Delta$): The influence of the latent variable $U$ is bounded by a parameter $\Delta \ge 1$. This constrains the odds ratio between the conditional transition probability $P(s' | s, a, u)$ and the marginal transition probability $P(s' | s, a)$:
  $$\frac{1}{\Delta} \le \frac{\text{odds}(P(S' = s' | S = s, A = a, U = u))}{\text{odds}(P(S' = s' | S = s, A = a))} \le \Delta$$
  • $\Delta = 1$: No confounding (fully specified PNSCM).
  • $\Delta \to \infty$: Maximum confounding.

2.3 Calibrating Sensitivity ($\Delta$)

A major challenge is selecting $\Delta$ without domain knowledge. The paper proposes a principled method to derive an upper bound for $\Delta(s, a)$ from observed state variables:

1. Hiding Variables: For each observed state variable $i$, the authors compute the transition distribution $P(s' | s_{-i}, a)$ by marginalizing over $i$.
2. Odds Ratio Calculation: They calculate the maximum odds ratio between the full transition and the marginalized transition for each variable.
3. Proxy Selection: The sensitivity $\Delta(s, a)$ is set to the maximum of these ratios across all variables. A scaling factor $\Gamma$ can be applied to adjust the strictness of the bound.

2.4 Optimization Procedure

The goal is to find a policy $\pi$ that maximizes the worst-case counterfactual value $V^{CF}$ under the uncertainty of $U$.

• Actualised Refinement in Counterfactuals: For an observed transition $(s_t, a_t, s_{t+1})$, the counterfactual transition probability $P^{CF}_t$ is a mixture: with probability $P(U=u|\tau)$, the transition is deterministic (staying at $s_{t+1}$); otherwise, it follows the optimized transition probability $\tilde{P}$.
• Decoupled Optimization: To make the problem tractable, the authors decouple the optimization across time steps and state-action pairs, treating each $(t, s, a)$ triple independently. This yields a conservative lower bound on the true pessimistic value.
• Adversarial Formulation: The optimization involves an adversary that jointly selects the posterior $P(U|\tau)$ and the transition probabilities $\tilde{P}$ to minimize the value, subject to the sensitivity constraints. This non-convex problem is solved using Gurobi.
• Time-Homogeneity: The paper notes that incorporating time-homogeneity (requiring consistent transitions across time) creates a trilinear coupling solvable via gradient descent, though this only guarantees local minima.

3. Experimental Evaluation

The approach was validated on a Sepsis Treatment Simulator (Oberst and Sontag, 2019).

• Setup: The state space includes vital signs (heart rate, blood pressure, etc.). The action space involves toggling treatments.
• Confounder: Diabetes status ($U \in \{0, 1\}$) acts as a global latent confounder, affecting 20% of the population. The true transition matrix $P(S' | S, A, U)$ is known for evaluation but hidden during the policy derivation.
• Data: The method uses trajectories generated by a suboptimal policy (optimal with probability 0.5, random otherwise) on the marginalized transition matrix (where diabetes is hidden).

Key Results

1. Robust Improvement: The derived counterfactual policies ($\pi^*_\Delta$) consistently outperformed the observed suboptimal policy when evaluated on the fully observed MDP (where diabetes is known).
2. Proxy Accuracy:
   • For diabetic trajectories, the unscaled proxy ($\Gamma=1$) closely matched the "oracle" performance (the best possible result if the true diabetes sensitivity $\Delta^*$ were known).
   • For nondiabetic trajectories, the proxy was conservative (overestimating confounding), leading to value estimates that were lower than the oracle. However, the resulting policies remained close to the oracle's performance, demonstrating that the method recovers robust policies even when the value estimates are conservative.
3. Sensitivity to $\Gamma$: Adjusting the scaling factor $\Gamma$ allowed the authors to tune the trade-off between value estimation accuracy and policy robustness.

4. Key Contributions

• Formalization: The paper formalizes counterfactual policy optimization for MDPs under PNSCMs, explicitly separating latent confounding from irreducible stochasticity.
• Sensitivity Framework: It proposes a practical optimization problem for identifying robust counterfactual policies using a sensitivity analysis framework that bounds latent variable influence.
• Calibration Method: It introduces a data-driven method to calibrate the sensitivity parameter $\Delta$ using observed state variables, reducing reliance on expert domain knowledge.
• Empirical Validation: It demonstrates that robust counterfactual policies can be derived and validated on a realistic healthcare simulator with hidden global confounders.

5. Significance and Claims

The authors claim that this work provides the first practical optimization problem for identifying robust counterfactual policies under a sensitivity analysis framework within PNSCMs. The significance lies in addressing the limitation of existing deterministic SCM approaches, which cannot properly model environments with inherent stochasticity.

The paper concludes that the derived policies are robust when evaluated on the true, fully-specified environment, even when the sensitivity parameter is estimated from observed data. The authors modestly note that while the decoupled optimization provides a valid lower bound, future work is needed to explore time-homogeneous constraints and different types of latent variables (e.g., memoryless confounders) to obtain more accurate value estimates.