Small Samples and Short Panels: Evaluating Policy Evaluation Methods with Realistic Data

Technical Summary: Small Samples and Short Panels: Evaluating Policy Evaluation Methods with Realistic Data

Problem Statement
Researchers in economics, public health, and political science frequently utilize longitudinal quasi-experimental designs, such as Difference-in-Differences (DiD) and Synthetic Control Methods (SCM), to evaluate policy interventions at the geographic level (e.g., states, counties, cities). However, these evaluations often face severe data constraints: limited numbers of geographic units and short panel lengths (few pre- and post-treatment periods). While DiD relies on the parallel trends assumption, which may be implausible without conditioning on extensive covariates (a difficult task in small samples), newer methods like Synthetic Difference-in-Differences (SDiD) and the Augmented Synthetic Control Method (ASCM) offer alternatives.

The core problem addressed is that the theoretical performance guarantees for SDiD and ASCM—specifically regarding bias and confidence interval coverage—are derived under asymptotic assumptions requiring large numbers of units and time periods. These assumptions often fail in realistic health policy evaluations where sample sizes are small (e.g., fewer than 10 control units) and panels are short (e.g., fewer than 20 pre-treatment periods). There is a lack of systematic evidence regarding whether these methods yield reliable estimates and valid inferences under such severe data limitations.

Methodology
To evaluate the performance of SDiD and ASCM in these realistic, small-sample settings, the authors employ a calibrated simulation strategy. Rather than relying on purely theoretical data generation, they inject known treatment effects (specifically setting the true effect to zero to test null properties) into simulated datasets derived from real-world empirical distributions.

• Data Calibration: The simulation parameters are estimated from two real-world datasets:
  1. Youth Risk Behavior Surveillance System (YRBSS): City-level youth soda consumption data ($N=7$ cities, $T=6$ periods), representing short panels with few units.
  2. Current Population Survey (CPS): State-level log wage data ($N=50$ states, $T=40$ years), allowing for the investigation of longer panels.
• Data Generating Process (DGP): Data is generated under a latent factor model ($Y = \Gamma\Psi' + W \cdot \tau + E$), a setting where both SDiD and ASCM possess favorable theoretical properties. The factor loadings ($\Gamma$) and time factors ($\Psi$) are estimated using interactive fixed effects from the calibration datasets.
• Simulation Design: The authors vary the number of treated units ($N_{tr}$), control units ($N_{co}$), pre-treatment periods ($T_{pre}$), and post-treatment periods ($T_{post}$). Scenarios include extreme limitations (e.g., $N_{co}=6$, $T_{pre}=5$).
• Methods Evaluated:
  • SDiD: Implemented via the synthdid R package, using placebo variance estimation.
  • ASCM: Implemented via the augsynth R package with ridge regression as the outcome model, utilizing conformal inference for hypothesis testing.
  • DiD: Included as a benchmark in settings where parallel trends holds exactly, to quantify efficiency losses.
• Metrics: Performance is assessed via bias, empirical standard error, and confidence interval coverage rates (calculated as the proportion of simulations where the true null $\tau=0$ is not rejected).

Key Results
The simulation results reveal distinct performance characteristics for SDiD and ASCM depending on sample size and panel length:

1. Bias and Precision:

   • Both methods exhibit moderate bias (typically $<0.1$ empirical standard deviations) across most scenarios.
   • ASCM generally produces smaller bias than SDiD, particularly when more units are available. However, in settings with more time periods and fewer treated units, ASCM can exhibit larger bias.
   • SDiD generally achieves lower empirical standard errors (higher precision) than ASCM, particularly in scenarios with long pre-treatment periods and few treated units (e.g., in the CPS data with $N_{tr}=1, N_{co}=10, T_{pre}=30$, ASCM variance was 2.26 times larger than SDiD).

2. Confidence Interval Coverage:

   • ASCM: The conformal inference procedure consistently overcovers (coverage $>95\%$) when the number of pre-treatment periods is small ($T < 20$), leading to a substantial loss in statistical power. However, overcoverage can persist even with longer panels (e.g., $T=30$) when the number of treated units is very small ($N_{tr}=1$). Nominal coverage is generally achieved when at least 20 pre-treatment periods are available, provided there are sufficient treated units.
   • SDiD: Coverage is highly dependent on the number of control units. With a single treated unit, nominal coverage is only attained with at least 25 control units. With multiple treated units, nominal coverage is achievable with as few as 10 control units.
   • DiD: Achieves nominal coverage regardless of sample size when the parallel trends assumption holds, though SDiD and ASCM sacrifice precision (relative efficiency ranges from 1.09 to 2.25) to gain robustness against parallel trends violations.

3. Small Sample Performance:

   • In the most constrained scenarios (e.g., $N_{co}=6, T_{pre}=5$), both methods show deviations from nominal coverage, though bias remains relatively low.
   • The authors note that while theoretical assumptions for asymptotic normality often fail in these small samples, the methods still perform "reasonably well" in terms of point estimation, though inference (intervals) requires caution.

Significance and Practical Guidance
The paper provides practical guidance for researchers evaluating policy interventions in settings with few units and short panels, particularly where the parallel trends assumption is suspect:

• Method Selection: SDiD and ASCM are recommended as primary analyses only when subject-matter knowledge suggests parallel trends is implausible (e.g., due to pre-intervention shocks). Otherwise, they should be used as sensitivity analyses to DiD.
• Inference Constraints:
  • ASCM should be used with caution in short panels ($T < 20$) due to conservative confidence intervals that reduce power; even with longer panels, coverage may be poor if the number of treated units is very small.
  • SDiD requires a sufficient number of control units (approx. 25 for single treated units, 10 for multiple) to achieve valid inference.
• Trade-offs: The study highlights that the robustness of SDiD and ASCM to parallel trends violations comes at the cost of precision compared to DiD when parallel trends actually holds.
• Limitations: The authors acknowledge that their simulations are calibrated to only two datasets and assume a latent factor model with simultaneous adoption. They do not claim prescriptive guidance for specific covariate adjustments or alternative variance estimators, noting that their results reflect default software implementations.

Conclusion
The study concludes that while SDiD and ASCM offer valuable tools for policy evaluation in data-scarce environments, their inferential properties are sensitive to sample size and panel length. Researchers must carefully consider the number of available control units and time periods before relying on these methods for hypothesis testing, as standard asymptotic guarantees may not hold in the small-sample settings common to health policy research.