📈 economics

Conformal Inference for Counterfactuals and Individual Treatment Effects with Experiment Attrition

This paper introduces a novel method that combines conformal inference with established missing data techniques to generate robust and precise prediction intervals for individual treatment effects, effectively addressing the challenges of experiment attrition where traditional approaches often fail due to strong assumptions.

Original authors: Xiangyu Song

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangyu Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to figure out if a new study method actually helps students get better grades. You run an experiment: half the class uses the new method (Treatment), and the other half sticks to the old one (Control).

But here's the catch: Attrition.

Some students drop out of the study halfway through. Maybe they got sick, moved away, or just got bored. Now, you have a problem. You can see the grades of the students who stayed, but you have no idea how the students who left would have performed.

The Old Way: Guessing and Weighting

Traditionally, researchers have tried to fix this in a few ways:

The "Ignore Them" Approach: You only look at the students who stayed. But what if the students who dropped out were the ones struggling the most? Your results would be biased.
The "Fill in the Blanks" Approach (Imputation): You try to guess the missing grades based on the students who stayed. This works well if your guess is perfect, but if your assumptions about how students learn are wrong, your whole conclusion falls apart.
The "Weighting" Approach: You give more importance to the students who stayed who look like the ones who left. But this gets messy and can still be inaccurate if the "dropouts" were fundamentally different in ways you can't see.

The New Way: The "Safety Net" (Conformal Inference)

This paper introduces a new tool called Conformal Inference. Think of it not as trying to guess the exact missing grade, but as building a safety net around your predictions.

Instead of saying, "Student A would have gotten a B," the new method says, "We are 95% confident Student A would have gotten a grade between a B- and a B+."

Here is how the author, Xiangyu Song, improves this safety net for the specific problem of people dropping out of experiments:

1. The "Two-Step" Safety Net

The paper realizes there are two different kinds of missing information:

Step 1: For the students who stayed, we are missing one of their two potential grades (e.g., we know their grade with the new method, but not what they would have gotten with the old one).
Step 2: For the students who dropped out, we are missing both grades.

The author's method builds a safety net for Step 1 first, and then uses that to build a bigger, stronger safety net for Step 2.

2. The "Translator" (Handling the Shift)

Here is the tricky part: The students who dropped out might be different from the ones who stayed. Maybe the dropouts were more stressed or had less time. In statistics, this is called Covariate Shift. It's like trying to predict the weather in a desert using data from a rainforest. If you don't adjust for the difference, your prediction will be wrong.

The paper uses a clever mathematical "translator" (called a semiparametric efficient estimator) to adjust the safety net. It says, "Okay, the dropouts look different from the stayers, so let's widen or shift our safety net slightly to account for that difference." This ensures the net is still strong enough to catch the truth, even if the groups aren't identical.

3. Why It's Better

The author tested this method with computer simulations (like running the experiment a thousand times in a computer) and real-world data from political science studies.

The Old Methods: Either the safety net was too wide (useless because it covered everything from an F to an A+) or it was too narrow and missed the truth (the student actually got an F, but the net said they got a B).
The New Method: It found the "Goldilocks" zone. The safety nets were narrow enough to be useful (giving a precise range) but wide enough to be correct (catching the true result 95% of the time).

The Real-World Impact

The author re-analyzed two real studies:

A study on financial markets: They found that the students who dropped out actually reacted differently to the treatment than those who stayed. If you only looked at the students who stayed, you would have missed a huge part of the story.
A study on civic education in Tunisia: Similar story. The dropouts had different reactions. By using this new method, the researchers could see that the treatment effect was actually stronger (or weaker) when you included the people who left.

The Bottom Line

This paper gives researchers a new, robust way to handle the inevitable problem of people dropping out of studies. Instead of throwing away data or making wild guesses, they can now build a statistical safety net that is both precise and reliable.

It's like upgrading from a fishing rod that only catches the fish you can see, to a net that can accurately estimate the size and location of the fish that swam away, ensuring you don't miss the whole picture.

1. Problem Statement

The paper addresses the challenge of experimental attrition (missing outcome data due to non-response or dropout) in randomized controlled trials (RCTs), a common issue in social science research.

The Core Issue: When attrition occurs, researchers often lose the ability to observe potential outcomes for a subset of units. Standard approaches like Complete Case Analysis (CCA), Multiple Imputation (MI), or Inverse Propensity Weighting (IPW) rely on strong parametric assumptions (e.g., normality, correct model specification) or specific missing data mechanisms (e.g., Missing Completely at Random - MCAR).
The Gap: Existing methods often fail to account for covariate shift—the phenomenon where the distribution of covariates differs between the observed group ( $R=1$ ) and the attrition group ( $R=0$ ). Furthermore, many existing conformal inference methods for causal effects (e.g., Lei & Candès, 2021) produce overly conservative (wide) prediction intervals or rely on assumptions (like MCAR) that do not hold in practice (where Missing at Random - MAR is more realistic).
Goal: To construct valid, distribution-free prediction intervals for Individual Treatment Effects (ITE) and counterfactual outcomes specifically for the attrition group, while ensuring the intervals are narrow enough to be informative.

2. Methodology

The author proposes a novel framework combining Conformal Inference with Semiparametric Efficient Estimation to handle missing data under the Missing at Random (MAR) assumption.

A. Theoretical Framework

Setup: The paper uses the potential outcome framework. Let $Y(1), Y(0)$ be potential outcomes, $D$ the treatment, $X$ covariates, and $R$ the response indicator ( $R=1$ observed, $R=0$ attrition).
Assumptions:
1. Unconfoundedness: Treatment assignment is independent of potential outcomes given $X$ .
2. MAR (Unconfoundedness of Attrition): Attrition is independent of potential outcomes given $X$ and $D$ .
3. Overlap: Propensity scores for treatment and response are bounded away from 0 and 1.
The Two-Step Challenge:
1. Construct prediction intervals for counterfactuals and ITEs within the observed group (addressing covariate shift between treatment and control units).
2. Extrapolate these intervals to the attrition group (addressing covariate shift between observed and attrition groups).

B. The Proposed Algorithm: Semiparametric Efficient Conformal Inference

Instead of relying solely on empirical quantiles of nonconformity scores (which can be inefficient and yield wide intervals), the author derives the Efficient Influence Function (EIF) to identify the thresholds for prediction intervals.

Step 1: Observed Group (Counterfactuals & ITE):
- Uses Split Conformalized Quantile Regression (CQR).
- Derives the EIF ( $\psi_d$ ) for the quantile of the nonconformity score. This allows for "doubly robust" estimation, correcting for covariate shift between treatment and control groups within the observed sample.
- The threshold $\hat{\eta}_{\alpha, d}$ is identified by solving the sample moment condition of the EIF.
Step 2: Attrition Group (Extrapolation):
- Treats the ITE prediction intervals from Step 1 as "surrogate outcomes" for the attrition group.
- Applies a second layer of conformal inference to expand these intervals to cover the attrition group.
- Derives a second EIF ( $\psi_{\mathcal{C}}$ ) to handle the covariate shift between the observed ( $R=1$ ) and attrition ( $R=0$ ) groups.
- Uses an interval expansion function $\check{\mathcal{C}}_{ITE}(X)$ calibrated via the EIF to ensure the final interval covers the true ITE with probability $1 - (\alpha + \gamma)$ .
Implementation Details:
- Data Splitting: Uses a 3-fold split (Pre-training, Training, Calibration) to prevent overfitting of nuisance parameters (propensity scores and conditional distributions).
- Learners: Utilizes flexible machine learning models (Random Forest, BART, XGBoost, etc.) via SuperLearner to estimate nuisance parameters.
- Asymptotic Guarantees: The method ensures asymptotic coverage of $1 - (\alpha + \gamma)$ for the attrition group, provided nuisance parameters are estimated with sufficient accuracy (double robustness).

3. Key Contributions

Novel Framework: Introduces conformal inference as a robust tool for political science and causal inference, specifically tailored to handle attrition-induced missing data under MAR.
Methodological Innovation: Combines conformal inference with semiparametric efficiency theory. By using EIFs to determine nonconformity thresholds rather than raw empirical quantiles, the method achieves narrower prediction intervals while maintaining valid coverage, overcoming the "conservatism" of previous weighted conformal methods.
Comprehensive Evaluation: Provides a rigorous evaluation across different predictive models and data generating processes (DGPs), demonstrating that the method outperforms standard alternatives in both coverage and interval length.

4. Results

A. Simulation Studies

Setup: Monte Carlo simulations with sample sizes $N \in \{500, 1000, 5000\}$ and varying covariate correlations.
Comparisons: The proposed method (CISE) was compared against:
1. Multiple Imputation (Amelia II).
2. Weighted CQR with an unweighted nested framework (Lei & Candès, 2021).
Findings:
- Coverage: CISE achieves the nominal 95% coverage (or slightly above) across all sample sizes and models. In contrast, Multiple Imputation often failed to achieve nominal coverage (under-coverage) due to violated normality assumptions, while the unweighted nested CQR was overly conservative (coverage $\approx 1.0$ ).
- Interval Length: CISE produced significantly narrower intervals than the unweighted nested CQR (which were too wide to be useful) and maintained reasonable lengths compared to Multiple Imputation, without sacrificing coverage.
- Robustness: The method performed consistently well regardless of covariate correlation ( $\rho=0$ vs $\rho=0.9$ ).

B. Empirical Applications

The author reanalyzed two published studies:

Margalit and Shayo (2021): Field experiment on financial markets and political preferences (17.8% attrition).
- Result: CISE produced an ATE for the attrition group (0.190) that was nearly double the observed group's ATE (0.098). The overall ATE remained significant. CISE intervals were narrower than WCQR and comparable to MI, but with better theoretical guarantees.
Finkel et al. (2024): Civic education in Tunisia (36-40% attrition).
- Result: For specific outcomes (M2, M3), including the attrition group via CISE changed the substantive conclusions (e.g., shifting significance from 5% to 1% or revealing significance where none existed).
- Heterogeneity: Analysis revealed that the attrition group had heterogeneous treatment effects driven by unobserved confounders (e.g., political interest), which standard methods missed.

5. Significance and Implications

Robust Causal Inference: The paper provides a stronger foundation for causal inference in the presence of attrition, moving away from reliance on strong parametric assumptions (like normality) toward distribution-free guarantees.
Precision: By leveraging semiparametric efficiency, the method solves the trade-off between validity and precision, offering intervals that are both statistically guaranteed and practically useful (narrow).
Policy and Theory: The empirical results demonstrate that ignoring attrition can lead to biased estimates and incorrect substantive conclusions. The proposed method allows researchers to rigorously compare treatment effects across observed, attrited, and full samples.
Limitations & Future Work:
- The method still relies on the MAR assumption (cannot handle MNAR).
- High attrition rates can increase noise.
- Data splitting reduces the effective sample size for training.
- Future research directions include extending to MNAR settings, integrating with Double Machine Learning, and applying to panel data.

In summary, this paper presents a state-of-the-art solution for handling missing data in experiments, offering a method that is theoretically sound, computationally feasible, and empirically superior to existing alternatives in terms of both coverage reliability and interval precision.