The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

1. Problem Statement

The paper addresses a fundamental challenge in causal inference: how to effectively combine randomized experiments (the gold standard) with observational studies (abundant and cheaper but prone to bias).

• The Core Issue: Observational studies suffer from unmeasured confounding, leading to biased estimates of causal effects.
• The "Illusion of Learning": Citing Gerber, Green, and Kaplan (2004a), the authors highlight a phenomenon where, in the absence of prior information about the distribution of bias, observational data provides no meaningful improvement to the estimation of a causal parameter. Even if a researcher has many observational studies, if they are completely uncertain about the bias distribution (a flat prior), the posterior estimate of the causal effect remains identical to the experimental estimate alone. The observational data creates an "illusion" of increased precision without actually reducing bias or risk.

2. Methodology: An Empirical Bayes Framework

The authors propose a hierarchical Gaussian model to formalize the combination of data sources and apply Empirical Bayes (EB) methods to learn the bias distribution from the data itself.

The Hierarchical Model

• Experimental Study ($y_e$): $y_e \sim N(\theta^*, \sigma_e^2)$, where $\theta^*$ is the true causal effect.
• Observational Studies ($y_{o,j}$): $y_{o,j} \sim N(\theta^* + b_j, \sigma_{o,j}^2)$, where $b_j$ is the unknown bias for study $j$.
• Bias Distribution: The biases $b_j$ are assumed to be drawn from a population distribution $b_j \sim N(\mu, \gamma^2)$.

The goal is to estimate $\theta^*$ by learning the hyperparameters $(\mu, \gamma^2)$ of the bias distribution.

Three Scenarios Analyzed

The paper rigorously analyzes three distinct modeling approaches:

1. Flat Prior on Bias (The Illusion):

   • Assumes no knowledge of the bias distribution (effectively $\gamma^2 \to \infty$).
   • Result: The posterior mean for $\theta$ is exactly $y_e$. Observational data is ignored. The risk remains $\sigma_e^2$.

2. Empirical Bayes with Zero-Mean Assumption:

   • Assumes biases have a known mean of zero ($\mu=0$) but an unknown variance $\gamma^2$.
   • Method: Estimate $\hat{\gamma}^2$ via maximum marginal likelihood or moment matching.
   • Result: The illusion is "broken." Observational data shrinks the posterior mean toward the observational estimates, and the risk vanishes as $J \to \infty$.
   • Critique: The assumption that observational biases average to zero is often unrealistic in social sciences and other fields.

3. Empirical Bayes with Unknown Mean and Variance (The Return of the Illusion):

   • Assumes biases follow $N(\mu, \gamma^2)$ where both $\mu$ and $\gamma^2$ are unknown and estimated from the observational data.
   • Result: The illusion returns. The posterior mean collapses back to the experimental estimate $y_e$.
   • Paradox: While the point estimate does not change, the posterior variance decreases artificially. This leads to overconfidence (narrower credible intervals) without any actual reduction in bias or risk.

The Solution: Calibration Studies

To break the illusion without relying on the unrealistic zero-mean assumption, the authors introduce Calibration Studies.

• Definition: Observational studies targeting an intervention with a known true causal effect (typically zero).
• Mechanism:
  • Let $y_{c,k}$ be a calibration study where the true effect is 0. The estimate is $y_{c,k} \sim N(b_k, \sigma_{c,k}^2)$.
  • Because the true effect is known, any deviation in $y_{c,k}$ is purely due to bias $b_k$.
  • These studies allow the researcher to directly estimate the population parameters $(\mu, \gamma^2)$ of the bias distribution using only the calibration data.
• Calibrated Empirical Bayes (CEB):
  1. Fit $(\hat{\mu}, \hat{\gamma}^2)$ using the calibration studies ($y_c$).
  2. Plug these estimates into the full model combining the experiment ($y_e$) and observational studies ($y_o$).
  3. The resulting estimator $\hat{\theta}_{CEB}$ uses the observational data to shrink the estimate, but the shrinkage is informed by the learned bias distribution, not an unidentifiable one.

3. Key Contributions

1. Formalization of the Illusion: The paper provides a rigorous mathematical proof (Theorem 1 & 3) showing that without external information on bias, observational data cannot improve causal inference and may dangerously reduce uncertainty estimates (overconfidence).
2. Identification of the Zero-Mean Trap: It demonstrates that assuming a zero-mean bias distribution breaks the illusion but relies on a strong, often unjustifiable assumption.
3. Calibration Studies as a Remedy: It proposes and formalizes the use of calibration studies (negative controls) to identify the full bias distribution (mean and variance), thereby enabling valid Empirical Bayes shrinkage.
4. Theoretical Guarantees: The authors prove that under the Calibrated EB framework, the estimator achieves asymptotically vanishing risk as the number of observational and calibration studies increases (Theorem 4).

4. Results

• Simulations:
  • The Calibrated EB estimator ($\hat{\theta}_{CEB}$) achieves Mean Squared Error (MSE) scaling as $O(J^{-1})$, converging to the true parameter as sample size grows.
  • In contrast, the "Illusion" model (fitting mean and variance from observational data alone) fails to reduce MSE and produces misleadingly narrow confidence intervals.
• Semi-Synthetic Application (Water Usage):
  • Based on a large field experiment by Ferraro and Miranda (2013) regarding water conservation.
  • Setup: The authors created 100 subsets of data. One subset served as the "experiment" (randomized), while others were converted into "observational" (confounded) and "calibration" (pseudo-treatment with zero effect) datasets.
  • Findings:
    • The Illusion model produced a point estimate of -1.40 with a tight 95% credible interval [-1.84, -0.97], but this interval failed to cover the true effect (-0.48) (Coverage probability $\approx$ 0.006).
    • The Calibrated EB model produced a point estimate of -0.58 with a 95% interval [-0.90, -0.27], which successfully covered the true effect (Coverage probability $\approx$ 0.96).
    • The Calibrated EB model achieved a relative precision 109x higher than the reduced experiment alone, whereas the Illusion model only achieved 4.4x (at the cost of severe miscalibration).

5. Significance

• Paradigm Shift: The paper challenges the common practice of naively combining observational and experimental data. It warns that without understanding the bias distribution, adding observational data can be counterproductive by inducing false confidence.
• Practical Utility: It offers a concrete, actionable solution: Calibration Studies. By running "placebo" or "negative control" versions of observational designs, researchers can empirically learn the bias structure of their methods.
• Methodological Rigor: It bridges the gap between Bayesian and Frequentist perspectives, showing how Empirical Bayes can recover frequentist consistency (vanishing risk) in causal inference settings where traditional methods fail.
• Broad Applicability: The framework applies to any domain where observational data is abundant but biased (e.g., epidemiology, economics, political science), provided that calibration studies (interventions with known null effects) can be constructed or identified.

In summary, the paper argues that observational data is only useful if we can learn the distribution of its bias. Calibration studies provide the necessary "ground truth" to learn this distribution, shattering the illusion that observational data is inherently uninformative or dangerously misleading.