Bounding Treatment Effects by Pooling Limited Information across Observations

Technical Summary: Bounding Treatment Effects by Pooling Limited Information across Observations

Problem Statement
The paper addresses the challenge of estimating Average Treatment Effects (ATE) and Average Treatment Effects on the Treated (ATT) under the assumption of unconfoundedness when the overlap condition is violated or when conditioning variables take on a large number of distinct values relative to the sample size. In such "limited overlap" or high-dimensional settings, standard inverse propensity score (IPS) weighting fails because the propensity score cannot be consistently estimated, leading to infinite variance or non-uniform asymptotic behavior. Conversely, relying solely on Manski bounds (which require no overlap) yields extremely wide intervals that often lack informativeness. The authors note that existing methods, such as trimming or overlap weighting, alter the parameter of interest, while extrapolation methods rely on potentially misspecified models.

Methodology
The authors propose a novel framework for constructing bounds that occupy an intermediate space between the "no pooling" of Manski bounds and the "unlimited pooling" required for point identification via IPS weighting.

1. Limited Pooling: The core innovation is constructing bounds as sample averages of functions of observed outcomes where the contribution of each outcome depends only on the treatment status of a limited number of observations sharing similar covariate values.
2. Order of Bounds:
   • First-Order Bounds ($Q=1$): These correspond to standard Manski bounds, utilizing only individual outcome and treatment status, requiring no overlap.
   • Second-Order and Higher-Order Bounds ($Q \ge 2$): These utilize information from pairs or small groups of observations with identical (or clustered) covariate values. By leveraging the unconfoundedness assumption within these small groups, the authors derive tighter bounds.
3. Reference Propensity Score: The bounds are parameterized by a "reference propensity score" $p^*(x)$. The bounds remain valid regardless of the value of $p^*(x)$. However, if $p^*(x)$ is close to the true propensity score $p(x)$, the bounds become sharp (optimal in terms of expected width). This allows researchers to incorporate prior knowledge or simple estimates (e.g., global treatment share) without risking invalidity if the guess is incorrect.
4. Generalization to Higher Orders: The paper generalizes these bounds to arbitrary order $q$. The bounds are constructed as polynomials in the unknown propensity score $p(x)$. To make them feasible, terms involving $[p(x)]^r$ are replaced by products of treatment indicators from $r$ distinct observations within the same covariate cluster.
5. Handling Continuous Covariates: Since the theoretical framework assumes discrete covariates to allow for repeated values, the authors propose a practical clustering procedure (hierarchical agglomerative clustering) to group continuous covariates into bins. This creates a finite set of "effective" covariate values, enabling the application of the bounds.
6. Inference: The authors develop asymptotically valid confidence intervals. They treat the distinct covariate clusters as the effective sample size ($m$). They address the issue of potentially empty confidence intervals (where the lower bound exceeds the upper bound in finite samples) by constructing a union of intervals based on Stoye (2020), ensuring the final interval is never empty while maintaining asymptotic coverage.

Key Contributions

• Novel Bounds: The paper provides the first systematic method to derive valid bounds on ATE and ATT that are robust to limited overlap and high-dimensional covariates without requiring the overlap condition or consistent estimation of the propensity score.
• Robustness to Misspecification: Unlike IPS estimators, these bounds remain valid even if the reference propensity score is misspecified. They offer a "safety net" that narrows as the reference score approaches the truth but does not collapse if it does not.
• Finite Sample Validity: The methodology is designed for scenarios where the number of distinct covariate values is large relative to the sample size, a regime where standard non-parametric estimation fails.
• Inference Procedures: The paper establishes the asymptotic normality of the sample bounds and provides a concrete procedure for constructing non-empty, valid confidence intervals.

Results

• Monte Carlo Experiments: Simulations demonstrate that when the overlap condition holds and the reference score is correct, the proposed bounds are tight and comparable to oracle estimators. When overlap is violated or the propensity score is misspecified, the proposed bounds (particularly with $Q=2, 3$) remain informative and correctly reflect partial identification, whereas standard IPS estimators and nearest-neighbor matching exhibit significant bias or dispersion. The bounds improve upon Manski bounds by narrowing the interval width while maintaining validity.
• Empirical Applications:
  • Right Heart Catheterization (RHC): Applied to Connors et al. (1996) data, the method yields a 95% confidence interval for the ATT of $[-0.117, -0.006]$, excluding zero and suggesting a negative effect (increased mortality). This conclusion holds without relying on the overlap condition or a correctly specified logit model.
  • Temporary Employment Program: Applied to the Dehejia-Wahba subsample and the PSID control group. For the Dehejia-Wahba sample (where randomization is preserved), the bounds are tight and consistent with simple mean differences. For the PSID control group (where overlap is limited), the bounds widen significantly to reflect the uncertainty, contrasting with the Manski bounds which are even wider.

Significance and Claims
The authors claim that their method offers a robust compromise between the extreme conservatism of Manski bounds and the fragility of point-identification methods under limited overlap. By allowing for "limited pooling" of information, the approach leverages the unconfoundedness assumption to tighten bounds without demanding the strong data requirements (large local sample sizes) necessary for propensity score estimation. The paper emphasizes that the method is particularly valuable in high-dimensional settings where reliable propensity score estimation is infeasible. The authors are modest regarding the discretization of continuous covariates, acknowledging it as a practical solution that introduces approximation error, the formal analysis of which is left for future work. They also note that while their bounds are optimal within the class of "limited pooling" bounds considered, they do not claim to be the globally optimal bounds possible under all conceivable assumptions.