Technical Summary: Program Evaluation with Remotely Sensed Outcomes

1. Problem Statement

Traditional program evaluation often relies on survey-based outcomes, which are costly, difficult to scale, and sometimes infeasible to collect. Researchers increasingly utilize remotely sensed variables (e.g., satellite imagery, mobile phone activity, night lights) as low-cost, scalable proxies for economic outcomes. However, these variables are imperfect measurements.

The core challenge addressed is causal inference in settings where:

1. Experimental Sample: Contains randomized treatment assignment ($D$) and remotely sensed variables ($R$), but the true economic outcome ($Y$) is missing due to cost or logistical constraints.
2. Observational Sample: Links the outcome ($Y$) to the remotely sensed variable ($R$), but lacks randomized treatment assignment (treatment may be missing, deterministic, or confounded).

Existing methods often fail in this setting. Common two-step approaches (training a predictor on the observational sample and applying it to the experimental sample) suffer from attenuation bias when the remotely sensed variable is a post-outcome measure (i.e., $Y$ causes $R$, not vice versa). Similarly, Prediction-Powered Inference (PPI) methods often require joint observations of treatment, outcome, and proxy in both treatment arms, which are frequently unavailable, or they rely on stability assumptions regarding the outcome mechanism that may not hold across different populations.

2. Methodology and Identifying Assumptions

The authors propose a framework that nonparametrically identifies the Average Treatment Effect (ATE) by combining experimental and observational data under specific stability assumptions.

2.1 Causal Structure

The framework models the remotely sensed variable $R$ as post-outcome: variation in the economic outcome $Y$ causes variation in $R$.

• Experimental Sample ($S=e$): Observes $(X, D, R)$. $Y$ is missing.
• Observational Sample ($S=o$): Observes $(X, Y, R)$. $D$ may be missing or confounded.

2.2 Key Assumptions

1. Experimental Unconfoundedness: Standard randomization assumptions hold in the experimental sample ($D \perp \{Y(0), Y(1)\} | X, S=e$).
2. Stability of the Remotely Sensed Variable (The Core Assumption): The conditional distribution of the remotely sensed variable given the outcome and treatment is stable across samples:
   $$f(R | X, D, Y, S=e) = f(R | X, D, Y, S=o)$$
   This implies the sensing mechanism (e.g., the physics of how satellite images capture crop burning) is invariant, even if the outcome mechanism (how treatment affects $Y$) or the treatment mechanism differs between samples.
3. Completeness Conditions:
   • Complete Observational Cases: If $D$ is observed and varies in the observational sample, no further assumptions are needed.
   • Incomplete Observational Cases: If $D$ is missing or deterministic in the observational sample, an additional assumption of no direct effect is required: $D \perp R | X, Y$. This means the treatment affects the remotely sensed variable only through its effect on the outcome.

2.3 Identification Strategy

The treatment effect is identified via a conditional moment restriction. The authors derive a formula where the remotely sensed variable relates experimental treatment variation to observational outcome variation.

For a discrete outcome $Y \in \{y_1, \dots, y_K\}$, the conditional treatment effect vector $\theta(X)$ is identified by:
$$E[\Delta_e(X) - \Delta_o(X)^\top \theta(X) \mid X, R] = 0$$
Where:

• $\Delta_e(X)$ summarizes treatment variation in the experimental sample.
• $\Delta_o(X)$ summarizes outcome variation in the observational sample.

This allows for identification without specifying the functional form of the relationship between $R$ and $Y$. The framework supports arbitrary representations $H(X, R)$ of the remotely sensed variable, provided they satisfy a rank condition (relevance).

3. Estimation and Inference

3.1 Robustness to Misspecification

The authors develop an estimation algorithm (Algorithm 1) using sample splitting and cross-fitting. A key theoretical contribution is that valid $n^{-1/2}$ inference is achievable without rate conditions on the machine learning models used to predict outcomes, treatments, or sample indicators.

• The estimator remains valid even if the predictors (e.g., deep neural networks) are misspecified or lack convergence rates.
• This robustness stems from the Neyman orthogonality of the moment restriction.

3.2 Efficiency and Optimal Instruments

To improve efficiency, the authors connect the problem to optimal instrument theory (Chamberlain, 1987; Newey, 1993). The semiparametrically efficient representation $H^*(R)$ involves predicting:

1. The outcome $Y$ (from the observational sample).
2. The treatment $D$ (from the experimental sample).
3. The sample indicator $S$ (from the pooled data).
   Using these predictions allows for dimensionality reduction of high-dimensional $R$ while preserving causal information.

3.3 Diagnostic Tests

The framework provides two diagnostic tests:

1. Weak Instrument Test: Checks if the remotely sensed variable is sufficiently informative about outcome variation (analogous to weak instruments in IV).
2. Overidentifying Restrictions (J-Test): If different representations $H(R)$ and $H'(R)$ yield significantly different treatment effect estimates, it suggests a violation of the stability or no-direct-effect assumptions.

4. Empirical Results and Applications

The paper validates the method through calibrated simulations and real-world applications:

4.1 Simulations

• Bias: The standard two-step method exhibits large attenuation bias (underestimating effects), while the proposed method is approximately unbiased.
• Coverage: Confidence intervals for the proposed method achieve nominal coverage (approx. 95%), whereas the two-step method severely undercovers (as low as 0-25% in some scenarios).
• Comparison with PPI: When a validation sample is available, PPI methods can be unbiased but are less efficient than the proposed method. If the validation sample is non-random or baselines differ, PPI becomes biased.

4.2 Application 1: Smartcards and Village Poverty (India)

• Context: Evaluating the effect of biometric payment systems on village poverty.
• Setup: Experimental sample has treatment and satellite data; validation sample (non-random) has outcomes and satellite data; observational sample has outcomes and satellite data.
• Result: The proposed method yields estimates close to the "oracle" difference-in-means (using full outcome data), despite using outcome data for only a subset of villages. It demonstrates significant potential for survey cost reduction.

4.3 Application 2: Payments for Ecosystem Services and Crop Burning (India)

• Context: Evaluating incentives to reduce crop burning using satellite imagery.
• Setup: No experimental outcomes are available; only an observational sample links satellite data to ground truth.
• Result: The common two-step method underestimated the treatment effect by 47%. The proposed method identified a significantly larger effect (14.1% reduction in burning vs. 7.5% by the two-step method), correcting for the attenuation bias inherent in post-outcome surrogacy.

5. Key Contributions and Significance

1. New Identification Strategy: The paper provides the first nonparametric identification formula for treatment effects that combines experimental and observational data when the proxy is a post-outcome variable, relying on the stability of the sensing mechanism rather than the outcome mechanism.
2. Correction of Common Biases: It formally demonstrates that the widely used "two-step" prediction method suffers from attenuation bias in post-outcome settings, explaining why many existing studies may under-report intervention impacts.
3. Robust Inference: It establishes that valid inference is possible even with complex, potentially misspecified machine learning models, removing the need for strict rate conditions often required in Double Machine Learning.
4. Practical Diagnostics: It introduces testable overidentifying restrictions to validate the core assumptions, allowing researchers to check for stability violations using multiple representations of the remote sensing data.
5. Efficiency Gains: By leveraging optimal instruments that predict treatment and sample indicators in addition to outcomes, the method achieves higher precision than standard approaches.

The paper concludes that while remotely sensed variables are imperfect, they can be rigorously used for causal inference if the stability of the sensing technology is plausible. This framework enables researchers to reduce survey costs significantly while maintaining statistical validity, particularly in environmental and development economics.