Propensity score adjustment when errors in achievement measures inform treatment assignment

Technical Summary: Propensity Score Adjustment when Errors in Achievement Measures Inform Treatment Assignment

Problem Statement
The paper addresses a specific challenge in evaluating school-level interventions, such as the Texas Additional Days School Year (ADSY) initiative, where treatment assignment is driven by achievement gaps in demographic subgroups. In many schools, these subgroups (e.g., specific racial/ethnic groups) are small, leading to end-of-year test scores that are noisy measurements of the students' "true" average scores.

Under classical test theory, an obtained score ($W$) is the sum of a true score ($X$) and a random error ($\epsilon$). While the intervention assignment mechanism ($T$) is determined by the obtained scores ($W$), the authors argue that for valid causal inference, the propensity score (PS) should ideally balance schools on the unobserved true scores ($X$). Standard propensity score estimation using the noisy obtained scores ($W$) fails to balance the true scores ($X$) when the measurement error is "differential"—that is, when $W$ and $T$ are not conditionally independent given $X$. This occurs because the assignment mechanism explicitly responds to the noisy $W$, not just the underlying $X$. Furthermore, public reporting thresholds often withhold subgroup averages for very small groups, creating missing data issues that prevent standard PS estimation for those schools.

Methodology
The authors propose two methods to estimate propensity scores that balance on subgroup average true scores ($X$) rather than obtained scores ($W$), utilizing external estimates of measurement error variance (Conditional Standard Errors of Measurement, or CSEMs) provided by state education agencies.

1. Regression Calibration (RC) Propensity Score:

   • This approach replaces the error-prone obtained scores ($W$) in the PS model with Empirical Bayes (EB) predictions of the true scores ($\hat{X}$).
   • The EB predictions are derived from a two-level Hierarchical Linear Model (HLM) that accounts for variability between schools and between subgroups within schools.
   • The external estimate of the measurement error variance ($\Sigma_i$), derived from CSEMs, is crucial for identifying the random effects in the HLM.
   • The PS model is then a logistic regression of treatment assignment on the predicted true scores ($\hat{X}$) and other error-free covariates ($Z$).

2. Maximum Likelihood (ML) Propensity Score:

   • This approach jointly models the treatment assignment ($T$) and the obtained scores ($W$) conditional on the true scores ($X$) and error-free covariates ($Z$).
   • It assumes that $W$ captures the full effects of $X$ on $T$ (i.e., $T \perp W | X, Z$), acknowledging that $W$ acts as a "weak surrogate."
   • The target propensity score, $e(X, Z)$, is obtained by integrating $W$ out of the joint distribution of $T$ and $W$.
   • Since the resulting integral (a logistic-normal integral) is analytically intractable, the authors employ a fast approximation based on Monahan and Stefanski (1992), representing the integral as a location-scale mixture of normal distribution functions.
   • This method yields PS estimates that condition only on $X$ and $Z$, effectively marginalizing the noise.

Key Results
The authors evaluated these methods through simulation studies and an application to the Texas ADSY initiative.

• Simulation Study:

  • Overlap and Matching: The ML and RC methods significantly improved the overlap between intervention and control groups compared to a "naive" approach (standard logistic regression on $W$). In scenarios with small subgroups prone to high measurement error, the naive approach excluded up to 21% of intervention schools from analysis due to lack of matches (positivity violations), whereas the ML approach excluded only 2%.
  • Balance: Matching on ML and RC estimates resulted in better balance on the true subgroup scores ($X$) than matching on naive estimates.
  • Estimation Error: The resulting effect estimates (Average Treatment Effect on the Treated) had lower Root Mean Squared Error (RMSE) when using ML or RC PS estimates. The ML estimator, in particular, showed a favorable trade-off: while it exhibited slightly higher bias in specific heterogeneous error scenarios, it demonstrated substantially lower variance, leading to superior overall RMSE.

• Application to Texas ADSY:

  • The methods were applied to public school-level data where subgroup averages for small groups were withheld.
  • Unlike the naive approach, which requires imputation of missing data to function, the RC and ML methods could generate PS estimates for schools with withheld data using only the EB predictions and error variance estimates.
  • Matching on ML PS estimates achieved the best balance on the predicted true scores among the three methods, with the lowest average standardized differences.
  • A placebo test on 2019 scores for subgroups not used in the PS model (Asian and White students) revealed significant imbalances for Asian students across all methods, attributed to extremely small subgroup sizes (median of 0 students in some cases), highlighting the limits of balancing when data is sparse.

Significance and Claims
The paper claims that its proposed methods offer a practical solution for evaluating interventions where treatment assignment is driven by noisy, small-sample achievement measures. The primary contributions are:

1. Addressing Differential Measurement Error: The methods explicitly account for the fact that measurement error influences treatment assignment, moving beyond the assumption of nondifferential error often required by standard regression calibration.
2. Handling Missing Data: The approaches allow for PS estimation in schools where subgroup averages are withheld from public data, a common issue in education research, without requiring access to restricted student-level data for imputation.
3. Improved Positivity: By reducing the variability of the PS estimates, the methods improve the overlap between treatment and control groups, reducing the need to exclude intervention schools from analysis and thereby preserving sample size and precision.
4. Bias-Variance Trade-off: The ML estimator offers a compelling solution to the trade-off between confounder bias and bias from positivity violations, providing more precise effect estimates in the presence of significant measurement error.

The authors note that while their methods improve upon ignoring measurement error, they rely on the availability of external estimates of measurement error (CSEMs) and assume the error distribution is Gaussian (for the ML method). They acknowledge that future work is needed to understand the behavior of the ML estimator when different subgroups have differing measurement error distributions.