📊 statistics

Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction

This paper introduces a decision-theoretic framework for counterfactual decisions that defines "policy-coupled coverage" as the optimal link between uncertainty and action, leading to the development of the PC-RACP algorithm which achieves higher utility and rigorous finite-sample validity compared to existing methods.

Original authors: Yurui Zheng, Ying Jin

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yurui Zheng, Ying Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor, a marketing manager, or a policymaker. You have to make a big decision, like choosing a treatment, sending an email, or launching a policy. The problem is, you don't know the future. You have a prediction, but it's not a crystal ball; it's more like a weather forecast that might be wrong.

Usually, statisticians try to fix this by building a "safety net" called a prediction set. Instead of saying "It will rain," they say, "It will rain between 2 PM and 4 PM." This gives you a range of possibilities. If you catch the real outcome inside that range often enough (say, 95% of the time), the method is considered "valid."

The Paper's Big Problem:
This paper points out a flaw in how we usually build these safety nets when decisions change the outcome.

Think of it like this:

Standard Prediction: You predict the weather. Whether you predict rain or sun, the weather happens the same way. Your prediction set just needs to catch the actual weather.
Counterfactual Decision (This Paper): You are a doctor. If you prescribe Drug A, the patient gets Outcome A. If you prescribe Drug B, the patient gets Outcome B. The "outcome" isn't a fixed thing waiting to be caught; your choice of action creates the outcome.

The authors argue that if you use standard safety nets (which treat the outcome as fixed), you might end up with a safety net that is mathematically "valid" but practically useless. It might cover the wrong things because it didn't account for the fact that your decision changed the reality.

The Solution: "Policy-Coupled Coverage"
The authors introduce a new, smarter way to build these safety nets. They call it Policy-Coupled Coverage.

Here is the analogy:
Imagine you are playing a video game where you have to choose a path.

Old Way: You draw a map of all possible paths. You check if the map covers the terrain you might walk on, regardless of which path you pick.
New Way (Policy-Coupled): You first decide, "Okay, based on my map, I'm going to take Path A." Then, you draw a safety net specifically for Path A. You only care if the net catches the terrain you actually end up walking on.

The paper proves that this "self-referential" approach is the gold standard. It creates a perfect bridge between "I'm not sure what will happen" and "I need to make a safe decision."

How They Do It (The PC-RACP Machine)
The authors built a specific recipe (an algorithm called PC-RACP) to create these smart safety nets. Think of it as a three-step cooking process:

Learn the Map: They use past data to guess what happens if you take Action A, Action B, etc.
Pick the Best Path: They look at all those guesses and say, "If I want to be safe, which action looks the best?" They lock in that decision.
Calibrate the Net: Finally, they build the safety net only for that locked-in decision. They use a special math trick (conformal prediction) to make sure that, in the real world, this net catches the actual result 95% of the time.

What They Found (The Results)
They tested this on two things:

Fake Data (Simulations): They created a fake world where they knew the answers. They found that their new method was much better at helping the "decision-maker" get a good result while still staying safe. The old methods were either too risky or too cautious (wasting opportunities).
Real Data (Email Marketing): They used a real dataset from an email marketing campaign (sending emails to sell things).
- The Result: Their method helped the company make more money (higher utility) than the old methods.
- Why? The old methods were too scared to send emails because their safety nets were too wide and vague. The new method knew exactly when it was safe to send an email and when to hold back, leading to better decisions.

The Takeaway
The main message is simple: Don't treat your decision like a spectator.

If your decision changes the outcome (like a doctor's prescription or a marketing email), you cannot use standard "safety nets." You need a safety net that is "coupled" to your specific choice. The authors show that if you build your uncertainty around the decision you actually make, you get better results, higher rewards, and you stay just as safe as before.

They call this Policy-Coupled Coverage, and it turns uncertainty from a scary unknown into a reliable tool for making better choices.

Technical Summary: Prediction Sets for Counterfactual Decisions

1. Problem Statement

The paper addresses the challenge of making high-stakes decisions under predictive uncertainty in counterfactual settings. In standard prediction problems, the outcome $Y$ is fixed regardless of the action taken. However, in counterfactual decision-making (e.g., treatment selection, policy making, marketing), the realized outcome $Y(A)$ depends on the chosen action $A$ .

The core difficulty lies in the interdependence between uncertainty quantification and the decision rule:

Coupled Outcomes: The outcome to be "covered" by a prediction set is not fixed; it is determined by the action selected.
Validity vs. Optimality Gap: While conformal prediction provides statistical validity (coverage guarantees) for fixed outcomes, it is unclear how these guarantees translate to optimality when the decision rule itself alters the distribution of the realized outcome. Existing methods often treat potential outcomes independently or ignore the counterfactual structure, leading to suboptimal decisions or invalid coverage for the induced policy.

The central question is: What is the right interface between uncertainty quantification and counterfactual decision-making where actions determine which potential outcome is realized?

2. Methodology and Framework

The authors propose a decision-theoretic framework centered on Policy-Coupled Coverage and a two-stage construction procedure called Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP).

2.1 Decision-Theoretic Foundations

Risk-Averse Objective: The framework adopts a risk-averse objective where an agent seeks to maximize a high-probability lower bound (utility certificate) on realized utility.
Set-Induced Decision Rule: Given a collection of prediction sets $\{C(x, a)\}_{a \in \mathcal{A}}$ (one for each action), the agent employs a max-min rule:
$\pi_{RA}(x; C) = \arg\max_{a \in \mathcal{A}} \inf_{y \in C(x, a)} u(a, y)$
This rule selects the action that maximizes the worst-case utility within the predicted outcome sets.

2.2 Policy-Coupled Coverage (PCC)

The paper introduces Policy-Coupled Coverage as the necessary validity notion. Unlike standard per-action marginal coverage (which requires $P(Y(a) \in C(X, a)) \ge 1-\alpha$ for all $a$ ), PCC requires that the prediction sets cover the realized outcome under the policy induced by the sets themselves:
$P\left( Y(\pi_{RA}(X)) \in C(X, \pi_{RA}(X)) \right) \ge 1 - \alpha$
Theoretical Justification:

Minimax Optimality: Theorem 2.1 establishes that under distributional ambiguity (where the true distribution is unknown but satisfies PCC), the max-min rule $\pi_{RA}$ is the minimax-optimal decision rule.
Lossless Interface: Theorem 3.1 proves that optimizing prediction sets under PCC is equivalent to directly optimizing over policies and utility certificates (RA-DPO). Thus, prediction sets serve as a "lossless interface" for risk-averse counterfactual decisions.
Equivalence to Universal Coverage: Theorem 3.2 shows that optimizing under PCC is equivalent to optimizing under a stronger "universal coverage" condition (valid for any policy), though PCC is strictly weaker and more practical for the induced policy.

2.3 PC-RACP Algorithm

To construct these sets in finite samples, the authors propose a two-stage procedure:

Model Estimation & Policy Learning:
- Split data into training, learning, and calibration sets.
- Estimate conditional utility quantiles $\hat{\gamma}(x, t, a)$ and the optimal conditional coverage assignment rule $\hat{g}(x, \beta)$ .
- Use the learning set to find a preliminary parameter $\hat{\beta}$ that satisfies an approximate coverage constraint, inducing a fixed max-min policy $\hat{a}(\cdot)$ .
Weighted Conformal Calibration:
- Fix the policy $\hat{a}(\cdot)$ to disentangle the decision rule from the coverage guarantee.
- Use the calibration set (filtered to instances where the logged action matches $\hat{a}(X)$ ) to perform weighted conformal prediction.
- Adjust for covariate shift using importance weights based on the behavior policy.
- Construct the final prediction sets $\hat{C}(x, a)$ such that they satisfy the PCC guarantee for the induced policy $\hat{a}(\cdot)$ .

3. Key Contributions

Novel Validity Notion: Identification of Policy-Coupled Coverage as the decision-theoretically optimal interface between uncertainty and action in counterfactual settings.
Theoretical Equivalence: Proof that optimizing prediction sets under PCC is equivalent to both universal coverage optimization and direct risk-averse policy optimization, establishing prediction sets as a lossless interface.
Explicit Optimal Form: Derivation of the explicit form of population-optimal prediction sets (Theorem 3.4), which are threshold-based on conditional utility quantiles.
Finite-Sample Procedure: Development of PC-RACP, a practical algorithm that achieves rigorous finite-sample coverage guarantees for the realized outcome under the induced policy, even when the policy is learned from data.

4. Experimental Results

The authors validate the framework through simulations and a real-world email marketing experiment (Hillstrom dataset).

Synthetic Simulations:
- Coverage: PC-RACP achieves empirical coverage close to the target $1-\alpha$ . In contrast, a standard Risk-Averse Conformal (RAC) method (ignoring counterfactual structure) falls below the target, and a "Plug-in" baseline fails to provide exact coverage.
- Utility: PC-RACP consistently achieves the highest average utility certificate across various miscoverage levels $\alpha$ , outperforming both RAC and the Plug-in baseline.
Hillstrom Email Marketing Experiment:
- Coverage: PC-RACP maintains coverage close to the nominal level. RAC is overly conservative (coverage significantly above target).
- Utility: PC-RACP yields higher utility.
- Mechanism of Gain: Analysis reveals PC-RACP gains utility in two regimes:
  - Small $\alpha$ : By constructing sharper prediction sets (reducing the frequency of ambiguous sets like $\{0, 1\}$ ) for the dominant action.
  - Large $\alpha$ : By selecting the high-reward action (sending an email) more frequently than RAC, while maintaining valid coverage.

5. Significance and Claims

The paper claims to provide the first decision-theoretic framework that rigorously connects uncertainty quantification to counterfactual decision optimality.

Suboptimality of Ignoring Structure: The authors demonstrate that ignoring the counterfactual nature of the decision problem (i.e., treating outcomes as fixed) is suboptimal for both statistical validity and decision utility.
Decision-Aware Uncertainty: The work argues that uncertainty quantification in counterfactual problems must be designed around the decisions it supports. Policy-coupled coverage is presented as a foundational step toward a broader theory of decision-aware uncertainty quantification.
Practical Applicability: The proposed PC-RACP method offers a concrete, finite-sample valid procedure for high-stakes decision-making where actions influence outcomes, bridging the gap between conformal prediction theory and causal inference applications.