AI-Assisted Variance Reduction in Randomized Experiments

Technical Summary: AI-Assisted Variance Reduction in Randomized Experiments

Problem Statement
The paper addresses the challenge of improving the precision of Average Treatment Effect (ATE) estimates in randomized experiments. While recent literature explores using Large Language Models (LLMs) and "digital twins" to simulate human responses, these synthetic predictions often suffer from bias and are poor substitutes for ground-truth human data when used in isolation. Existing hybrid estimation methods, such as Prediction-Powered Inference (PPI) and semiparametric efficient estimators, aim to combine AI predictions with human data but often lack robustness guarantees in randomized settings. Specifically, uncalibrated approaches can inflate variance if the AI predictions are not sufficiently accurate, and some methods introduce bias if the model is misspecified. The authors argue that researchers need a method that leverages AI predictions for variance reduction without compromising the unbiasedness of the estimator, even when prediction quality is modest or unknown.

Methodology
The authors propose a framework where AI-generated predictions are utilized strictly as auxiliary covariates within standard regression adjustment, rather than as direct substitutes for outcomes.

• Core Estimator: The paper advocates for including AI predictions ($\hat{Y}_i$) as covariates in a linear regression of the observed outcome ($Y^{obs}_i$) on the treatment indicator ($Z_i$). Specifically, they recommend the interacted regression specification:
  $$Y \sim 1 + Z + \hat{Y}(Z) + Z \times \hat{Y}(Z)$$
  where $\hat{Y}(Z)$ represents the AI prediction for the assigned treatment arm. This is analogous to adjusting for a prognostic score.
• Theoretical Justification: The authors demonstrate that this regression-adjusted estimator is a "calibrated" version of the model-assisted estimator. Unlike uncalibrated model-assisted estimators (which require the correlation between predictions and outcomes to exceed 0.5 to reduce variance), the regression-adjusted estimator possesses a "do no harm" property. It is asymptotically unbiased and guarantees that the variance is never greater than the unadjusted difference-in-means estimator ($Deff \leq 1$). If predictions are uninformative, the fitted coefficients shrink toward zero, and the estimator reverts to the standard difference-in-means.
• Handling Discrete Outcomes: For binary or count outcomes, the authors argue against using discretized AI predictions (e.g., classifying an LLM output as 0 or 1). Instead, they recommend adjusting for continuous predicted probabilities or scores. They provide implementation strategies for obtaining these continuous scores from LLMs, such as extracting log-probabilities of specific tokens or averaging over multiple stochastic draws.
• Data Sources: The framework accommodates predictions from various sources, including zero-shot LLMs, tabular foundation models (e.g., TabPFN), and standard supervised learning models (e.g., XGBoost, Ridge). The authors note that for models trained on the experimental data itself, cross-fitting is required to maintain validity, whereas zero-shot LLM predictions can be treated as fixed pre-treatment covariates.

Key Contributions

1. Simplicity and Robustness: The paper establishes that no new estimators are required; standard regression adjustment with AI covariates is sufficient. This approach inherits the robustness of the "do no harm" property, ensuring efficiency gains when predictions are informative and no penalty when they are not.
2. Comparison of Estimators: The authors systematically compare regression adjustment against uncalibrated model-assisted estimators and "digital twin" (imputation) estimators. They show that uncalibrated methods can inflate variance if prediction quality is low ($\rho < 0.5$), and digital twin estimators can introduce significant bias unless the AI model is perfectly specified.
3. Featurization of Unstructured Data: The paper highlights the unique capability of LLMs to featurize unstructured inputs (e.g., text, complex user profiles) into auxiliary covariates. This allows for variance reduction in settings where traditional tabular methods struggle to extract signal from raw text.
4. Implementation Guidance: The authors provide practical advice on extracting continuous scores from discrete LLM outputs and handling binary outcomes to maximize variance reduction.

Results
The paper validates its claims through synthetic experiments and three empirical applications: a survey mega-study, an email marketing A/B test, and a large-scale technology platform experiment.

• Synthetic Experiments: Simulations confirm that regression adjustment maintains a design effect $\leq 1$ across varying levels of prediction quality ($R^2_{\hat{Y}}$) and outcome predictability ($R^2_m$). In contrast, uncalibrated model-assisted estimators frequently inflate variance when prediction quality is low. The simulations also demonstrate that using continuous AI predictions for binary outcomes yields greater efficiency gains than using discretized predictions.
• Empirical Applications:
  • In the survey mega-study (involving digital twins), regression adjustment provided modest but real efficiency gains.
  • In the email marketing A/B test, adjusting for AI predictions alongside tabular covariates improved precision.
  • In the technology platform experiment, the approach was applied to sequential user data.
  • Across all settings, the gains were described as "real if modest," with the most significant benefits observed in studies containing substantial unstructured text data. The "do no harm" property was empirically confirmed; the adjusted estimators never performed worse than the unadjusted difference-in-means.

Significance and Claims
The paper claims that adjusting for AI-generated predictions should become a regular empirical practice in randomized experiments. The significance lies in the ability to leverage the predictive power of generative AI—particularly for unstructured data—without the risks associated with more complex hybrid estimators. The authors emphasize that while the efficiency gains may be modest in many structured settings, the approach is low-cost, robust, and particularly valuable for studies rich in text or complex inputs where traditional feature engineering is difficult. They conclude that the combination of guaranteed robustness ("do no harm") and potential for efficiency improvement makes this a superior strategy compared to uncalibrated alternatives or ignoring AI predictions entirely.