Integrating Heterogeneous Information in Randomized Experiments: A Unified Calibration Framework

This paper proposes a unified calibration framework that integrates heterogeneous internal and auxiliary information into randomized experiments under covariate-adaptive randomization via convex optimization, ensuring asymptotic validity and a no-harm efficiency guarantee while accommodating scenarios with growing numbers of strata and information sources.

The Gist

Imagine you are a doctor trying to figure out if a new medicine works. You run a clinical trial: you give the medicine to half your patients (the Treatment Group) and a sugar pill to the other half (the Control Group).

In a perfect world, the two groups would be identical in every way—same age, same diet, same genetics. But in reality, they aren't. Maybe the treatment group just happened to have more young people, or maybe the control group had more people who exercise. These differences are called covariates.

The Problem: The "Messy" Experiment

To fix this, scientists use a technique called Covariate-Adaptive Randomization (CAR). Think of this like sorting your patients into different "bins" (strata) based on a few key traits, like age and gender, before handing out the medicine. This ensures that within each bin, the groups are balanced.

However, there's a catch:

1. You can't sort by everything: You might sort by age and gender, but you can't sort by everything (like blood pressure, diet, or genetic markers) because you'd end up with too many tiny bins.
2. Data is everywhere: You have a mountain of extra data. You have historical data from past trials, real-world data from hospitals, and powerful AI models that can predict how patients might react.
3. The "Silo" Problem: Existing methods usually only look at the data inside the current experiment, within those specific bins. They ignore the rich history and external data, or they try to mix them in a way that breaks the math, potentially ruining the validity of your results.

The Goal: How do we use all this messy, different information (internal, external, AI-predicted) to get a more precise answer, without breaking the experiment?

The Solution: The "Unified Calibration Framework"

The authors of this paper propose a new method called a Unified Calibration Framework. Here is how it works, using a simple analogy:

1. The "Information Proxy" (The Clue Board)

Imagine you are a detective trying to solve a crime. You have a main suspect (the Treatment Effect), but you have a lot of clues.

• Internal Clues: What the suspect said in the room.
• External Clues: What the suspect's neighbors said, or what was found in their car.
• AI Clues: A computer program's prediction of where the suspect might go.

In this paper, the authors create a "Clue Board" (called the Information Proxy Vector). This board doesn't just hold one type of clue; it holds everything. It holds the predictions from your AI models, the data from your historical trials, and the data from your current experiment. It's a giant, flexible list of "best guesses" about how the patients would have reacted.

2. The "Calibration Weights" (The Balancing Act)

Now, you need to weigh these clues to get the final answer.

• Imagine you have a scale. On one side, you have the Treatment Group; on the other, the Control Group.
• The Calibration Weights are like little adjustable weights you put on the scale.
• The computer solves a puzzle: "How do I adjust these weights so that the 'Clue Board' looks exactly the same on both sides of the scale?"

If the "Clue Board" (all your extra data) looks balanced between the two groups after you adjust the weights, it means you've successfully corrected for the imbalances in your experiment.

3. The Magic Trick: "No-Harm" Efficiency

Here is the most important part of their discovery: You can never make things worse by adding more information.

Think of it like trying to hit a target with a bow and arrow.

• Old Method: You aim using only your eyes (just the current experiment).
• New Method: You aim using your eyes, plus a wind gauge, plus a laser sight, plus a weather report from last week.

The authors prove mathematically that even if your wind gauge is slightly broken, or your weather report is from a different city, using all of them together will never make your aim worse than just using your eyes alone. It will either make you hit the bullseye more often (more precise) or stay exactly the same. It is a "no-harm" guarantee.

Why This Matters in Real Life

The paper tested this on a real-world example: a study on whether giving people bank accounts in Uganda and Malawi helped them save money.

• They used data from Uganda to help analyze Malawi, and vice versa.
• They used AI models to predict savings behavior.
• Result: Their new method gave a much clearer, more precise answer than the old methods, with smaller margins of error.

The Takeaway

This paper is like giving scientists a universal adapter.

• Before, if you wanted to use historical data, you had to build a custom bridge. If you wanted to use AI, you had to build a different bridge.
• Now, you have one Unified Framework that plugs into any source of information. Whether it's a simple linear equation, a complex Deep Learning AI, or data from a trial 10 years ago, this framework can plug it in, balance the scales, and give you a better answer without breaking the rules of science.

In short: It's a smarter, safer, and more flexible way to combine all the data we have to find the truth in experiments.

Technical Summary

Here is a detailed technical summary of the paper "Integrating Heterogeneous Information in Randomized Experiments: A Unified Calibration Framework" by Wei Ma, Zeqi Wu, and Zheng Zhang.

1. Problem Statement

In modern randomized experiments, particularly those utilizing Covariate-Adaptive Randomization (CAR) (e.g., stratified block randomization, minimization), researchers often collect rich baseline covariates and auxiliary information from multiple sources. While CAR ensures balance on a small set of design covariates, it does not guarantee balance on all covariates, necessitating covariate adjustment during the analysis stage to improve estimation efficiency.

Current challenges include:

• Limited Information Integration: Existing adjustment methods (e.g., linear regression, Lasso, AIPW with machine learning) primarily rely on internal data within strata. They struggle to coherently integrate heterogeneous information sources, such as:
  • Cross-stratum information (borrowing strength across strata).
  • Aggregated predictions from multiple machine learning models.
  • External data from historical trials or real-world sources (RWD).
• Validity vs. Efficiency Trade-off: Incorporating external or complex internal information risks violating validity if the auxiliary data is biased or if the models are misspecified. Existing frameworks often lack a mechanism to guarantee that adding information will not increase asymptotic variance (the "no-harm" property).
• Theoretical Gaps: Standard asymptotic theory for covariate adjustment often assumes independent and identically distributed (i.i.d.) samples, which does not hold under CAR designs due to the induced dependence structure within strata.

2. Methodology: A Unified Calibration Framework

The authors propose a unified calibration framework that estimates the Average Treatment Effect (ATE) by integrating diverse information sources through calibration weights.

Core Components

1. Information Proxy Vector ($\xi_n$):
   The framework introduces a vector $\xi_n(X_i)$ that acts as a proxy for auxiliary information. This vector can be constructed flexibly to include:

   • Estimates of conditional mean functions ($h^*_{a[k]}(X)$) from various models (e.g., Random Forests, Neural Networks, Linear Regression).
   • Predictions from external datasets (historical trials, RWD).
   • Cross-stratum estimates.
   • The framework is model-agnostic: it does not require the information sources to be perfectly specified or unbiased.

2. Calibration Estimator ($\hat{\tau}_{cal}$):
   The estimator is defined as:
   $$\hat{\tau}_{cal} = \hat{\tau}_{sdim} + \frac{1}{n} \sum_{i=1}^n \hat{w}_i r_i$$
   Where:

   • $\hat{\tau}_{sdim}$ is the stratified difference-in-means estimator.
   • $r_i$ are residuals representing the part of the outcome unexplained by stratum means.
   • $\hat{w}_i$ are calibration weights obtained by solving a convex optimization problem:
     $$\min_{w} \sum_{i=1}^n D(w_i) \quad \text{subject to} \quad \frac{1}{n} \sum_{i=1}^n w_i \left( \frac{A_i - \pi_{n[k]}}{\pi_{n[k]}} \mathbb{I}(B_i=k) \right) (\xi_n(X_i) - \bar{\xi}_{n[k]}) = 0$$
   • $D(v)$ is a strictly convex discrepancy function (e.g., $D(v) = (v-1)^2/2$ for quadratic, or $v - \log v$ for empirical likelihood).

3. Handling CAR Dependence:
   Unlike standard i.i.d. settings, the authors condition on the realized stratum indicators and treatment assignments. They establish large-sample results using conditional laws of large numbers and conditional central limit theorems, addressing the complex dependence structures induced by CAR.

4. Cross-Fitting:
   To mitigate overfitting when $\xi_n$ is generated by machine learning, the authors incorporate cross-fitting (sample splitting), ensuring the validity of the asymptotic theory even with complex learners.

3. Key Contributions

• Unified Architecture: The framework recovers many existing covariate adjustment procedures (e.g., Bugni et al., Liu et al., Tu et al.) as special cases. It provides a single architecture for both internal (cross-stratum, multi-model) and external (historical, RWD) information borrowing.
• Robustness and "No-Harm" Guarantee:
  • Model-Agnostic Validity: The statistical inference remains valid even if the auxiliary information is biased or generated by inaccurate models.
  • Efficiency Guarantee: The authors prove a no-harm efficiency property: incorporating additional information sources into $\xi_n$ never increases the asymptotic variance. It either improves efficiency or maintains the baseline efficiency.
• General Inference Theory:
  • Establishes asymptotic normality and consistent variance estimation under CAR designs.
  • Extends theory to regimes where the number of strata ($K$) and the dimension of the information proxy ($d$) grow with the sample size ($n$).
  • Analyzes second-order bias, showing that specific choices of the discrepancy function $D(v)$ (e.g., empirical likelihood) can reduce bias compared to quadratic calibration.
• Semi-parametric Efficiency: The estimator achieves the semi-parametric efficiency bound if the linear combination of the true conditional mean functions lies within the span of the information proxy vector.

4. Results

Theoretical Results

• Theorem 1 & 3: Prove that $\sqrt{n}(\hat{\tau}_{cal} - \tau)$ converges to a normal distribution with a variance that is consistently estimable. The variance is strictly smaller than or equal to the variance of the stratified difference-in-means estimator.
• Theorem 2: Demonstrates that adding elements to $\xi_n$ (via a linear transformation) satisfies the no-harm property ($\text{Var}(\xi_n) \leq \text{Var}(\Lambda \xi_n)$).
• Theorem 4: Provides an expansion for the second-order bias, identifying conditions under which specific discrepancy measures (like $v - \log v$) yield estimators with zero second-order bias.

Empirical Results

• Simulation Studies:
  • Tested under linear, additive non-linear, and non-additive non-linear models with sample sizes $n \in \{500, 1000, 2000\}$.
  • Compared against standard estimators (sdim, AIPW with RF/NN/Linear).
  • Findings: Calibration estimators (especially those combining multiple ML models, e.g., cal_rfnn) consistently outperformed AIPW estimators in terms of Standard Deviation (SD) and Coverage Probability (CP), particularly in smaller samples. They were more robust to model misspecification and outliers.
• Empirical Application:
  • Applied to a field experiment on savings behavior in Uganda and Malawi (Dupas et al., 2018).
  • Strategy: Used data from one country to inform the estimation of the treatment effect in the other (external borrowing).
  • Outcome: The calibration estimator incorporating cross-country information (cal_info_X_Xβ) achieved the lowest standard errors (reductions of 6.3% to 11.2% compared to sdim) while maintaining valid inference. The results confirmed that subsidized bank accounts did not significantly increase total savings on average, consistent with prior literature but with higher precision.

5. Significance

This paper makes a significant contribution to causal inference in randomized experiments by:

1. Bridging the Gap: It offers a practical solution for leveraging the vast amounts of auxiliary data available in modern trials (historical controls, RWD, multiple ML models) without compromising statistical validity.
2. Theoretical Rigor: It provides the first rigorous asymptotic theory for calibration estimators under CAR designs with growing dimensions and strata, moving beyond the i.i.d. assumptions of previous literature.
3. Practical Utility: The "no-harm" guarantee encourages researchers to be more aggressive in incorporating diverse data sources, knowing that it will not degrade their estimates. The framework is flexible enough to accommodate various discrepancy measures and machine learning techniques, making it a versatile tool for modern clinical and social science trials.