Technical Conclusion: Synthetic Counterfactual Labels for Efficient Conformal Counterfactual Inference (SP-CCI)

1. Problem Definition and Motivation

The work addresses the challenge of constructing reliable prediction intervals for individual counterfactual outcomes in scenarios with treatment imbalance. In causal inference, for a unit with covariates $X$ and observed treatment $T$, only one potential outcome $Y(T)$ is observed, while the counterfactual outcome $Y(1-T)$ remains unobserved.

Existing methods, particularly Conformal Counterfactual Inference (CCI) (Lei and Candès, 2021), provide marginal coverage guarantees for these counterfactual predictions. However, CCI relies on calibration data from the specific treatment arm of interest. In many real-world scenarios (e.g., medical studies where expensive treatments are rare, or online advertising with sparse exposure), the dataset for the target treatment arm ($D_1$) is significantly smaller than the control arm ($D_0$). This scarcity leads to overly conservative, wide prediction intervals, limiting the utility of conformal inference for decision-critical decisions.

Although synthetic data generated by pre-trained models could supplement the scarce calibration set, directly incorporating these synthetic labels into standard conformal inference procedures would break statistical validity, as synthetic outcomes represent biased approximations of the truth.

2. Methodology: SP-CCI Framework

The authors propose Synthetic data-powered Conformal Counterfactual Inference (SP-CCI), a framework that augments the calibration set with synthetic counterfactual labels while preserving marginal coverage guarantees through a debiased estimation procedure.

Core Components

1. Generation of Synthetic Data:

   • The method utilizes a pre-trained counterfactual generative model $\hat{P}_{Y(1)|X}$ (which can be a neural network or a Large Language Model) to generate synthetic counterfactual labels $\hat{Y}(1)$ for units in the larger control dataset $D_0$.
   • This creates an extended synthetic dataset $\tilde{D}_1 = \{(X_i, \hat{Y}_i(1))\}_{i=1}^{n_0}$.

2. Debiased Estimation of Non-Coverage (Integration of PPI):

   • To correct the bias introduced by synthetic labels, SP-CCI adapts Prediction-Powered Inference (PPI) (Angelopoulos et al., 2023a).
   • The calibration set $D_1$ (real treated data) is used to "correct" the synthetic estimates. Specifically, the $n_0$ synthetic samples are partitioned into $n_1$ groups, each assigned to a real treated sample $(X_i, Y_i) \in D_1$.
   • An unbiased estimator for the probability of non-coverage $\hat{L}_\eta$ is constructed using a weighted difference between the loss on synthetic data and the loss on real data:
     $$\hat{\ell}_{i,\eta} = \frac{1}{r} \sum_{j} \tilde{w}_j \ell_\eta(\tilde{X}_j, \tilde{Y}_j) - w_i [\ell_\eta(X_i, \hat{Y}_i) - \ell_\eta(X_i, Y_i)]$$
     where $w_i$ and $\tilde{w}_i$ are importance weights accounting for propensity scores and treatment probabilities.

3. Risk-Controlling Prediction Sets (RCPS):

   • The method uses Risk-Controlling Prediction Sets (RCPS) (Bates et al., 2021) to select the interval widening parameter $\eta$.
   • Instead of guaranteeing unconditional coverage, SP-CCI ensures that the risk of non-coverage is controlled with high probability ($1-\delta$) over the draw of the calibration dataset.
   • The widening parameter $\hat{\eta}$ is chosen such that the upper confidence bound of the estimated non-coverage probability does not exceed the target level $\alpha$.

4. Handling Weight Misspecification:

   • The framework includes theoretical guarantees for cases where importance weights are estimated rather than exactly known. It is shown that coverage guarantees can be maintained by adjusting the widening parameter to account for the error bounds of the estimated weights.

3. Main Contributions

• Methodology: Introduction of SP-CCI, which integrates real and synthetic calibration data via a debiased estimator for non-coverage, enabling the use of extensive control data to improve inference for scarce treatment arms.
• Theoretical Guarantees: Formal proof that SP-CCI achieves high-probability coverage (controlled by the parameter $\delta$) under both exact and approximate importance weighting. The work distinguishes this from the unconditional marginal coverage of standard CCI and offers an adjustable mechanism to control "failures."
• Efficiency: Theoretical and empirical demonstration that SP-CCI yields significantly narrower prediction intervals compared to standard CCI and the GESPI framework (Bashari et al., 2025), particularly when high-quality synthetic generators are available.

4. Experimental Results

The authors evaluated SP-CCI on synthetic data (following Lei and Candès, 2021) and the semi-synthetic IHDP dataset (Hill, 2011), comparing it with standard CCI and GESPI.

• Synthetic Data: SP-CCI consistently produced narrower prediction intervals than CCI and GESPI while maintaining the required coverage guarantees. Interval width decreased with the quality of the counterfactual generative model (Low/Medium/High quality).
• IHDP Dataset:
  • SP-CCI achieved an average prediction interval width (APIW) of 3.52 (high-quality synthetic labels) compared to 20.24 for standard CCI.
  • While GESPI could generate narrower intervals with a relaxed safety barrier, it suffered from a high coverage violation rate (CVR) of 42% when tightened, rendering it unsuitable for decision-critical scenarios. SP-CCI maintained a low CVR (2–6%) across all variants.
  • The method successfully utilized a pre-trained LLaMA-2-13B model as a generative source, achieving competitive performance despite the model being trained outside the specific dataset, demonstrating robustness to distributional shifts.
• Additional Experiments:
  • Policy Evaluation: In a wireless handover scenario, SP-CCI provided more precise performance quantification (narrower intervals) for fixed policies compared to CCI.
  • Extreme Imbalance: On the Twins dataset with extreme treatment imbalance (63 treated vs. 11,921 control units), CCI produced intervals nearly 20 times wider than those of SP-CCI to maintain coverage. SP-CCI maintained near-target coverage with significantly narrower intervals.

5. Significance and Claims

The work claims that SP-CCI represents a significant step forward in making conformal counterfactual inference practical for unbalanced datasets. By effectively leveraging synthetic data without compromising statistical validity, the method addresses the "curse of dimensionality" and data scarcity inherent in causal inference for rare treatments.

The authors emphasize that SP-CCI offers an adjustable trade-off between interval width and coverage reliability via the parameter $\delta$, a feature lacking in standard unconditional CCI. The work demonstrates that high-quality synthetic data, when properly debiased, can drastically improve the efficiency of causal inference, making it more applicable to real-world domains such as personalized medicine and policy evaluation, where treatment data is often scarce.

The work concludes modestly, noting that extensions to multi-arm treatments, continuous treatments, and scenarios where neither potential outcome is observed remain open problems for future work.