A Bayesian Approach to Estimating Effect Sizes in Educational Research

1. Problem Statement

Educational research frequently utilizes pretest-posttest designs with control and intervention groups. However, standard frequentist methods for calculating effect sizes (e.g., Cohen's d) face several limitations in this context:

• Ignoring Multilevel Structure: Students are nested within classes, creating correlated data. Standard methods often ignore this clustering, leading to biased estimates.
• Homogeneity Assumptions: Traditional t-tests often assume equal variances (homoscedasticity) across groups and time points, which is rarely true in real-world educational settings.
• Point Estimates vs. Distributions: Frequentist approaches provide single point estimates and bootstrapped confidence intervals, failing to capture the full uncertainty and distribution of the effect size.
• Ambiguity in Paired Designs: In within-group (paired) designs, researchers often conflate different effect size metrics (e.g., $d_s$ vs. $d_z$), leading to non-reproducible results. $d_s$ often overestimates effects by ignoring the correlation between pre- and post-tests.
• Differential Effectiveness: Standard methods report a single average effect size, obscuring how an intervention might vary in effectiveness across different classrooms (heterogeneity).

2. Methodology

The paper proposes a purely Bayesian framework implemented using the brms package in R (which interfaces with the probabilistic programming language Stan). The methodology involves the following steps:

A. Data Structure and Modeling

• Data: A real-world quasirandomized within-classroom dataset ($N=1377$ students, 62 classes) with pretest and posttest scores.
• Multilevel Modeling: The authors construct Bayesian linear mixed-effects models that explicitly account for:
  • Random Intercepts and Slopes: Nesting students within classes to capture between-class variance.
  • Heterogeneous Residual Variances: Allowing residual variances ($\sigma$) to differ across time points (pre/post) and conditions (control/intervention) rather than assuming a single global variance.
  • Robust Distributions: Using Student's $t$-distributions (and skew-normal where appropriate) to handle outliers and non-normality, rather than strict normality assumptions.

B. Effect Size Formulations

The paper defines and computes several effect sizes based on Hedges' total effect size concept ($\delta_t$), adapted for Bayesian multilevel models:

1. Between-Group Effect Size ($d_s$):

   • Analogous to Cohen's $d$ for pooled designs.
   • Formula: $\frac{\mu_2 - \mu_1}{\sqrt{\text{Total Variance}}}$.
   • Key Innovation: The denominator includes not just residual variance but also the variance components from random effects (class-level variance) and hyperparameter variances.
   • Usage: Comparing intervention vs. control at a specific time point.

2. Within-Group Effect Size ($d_s$ and $d_z$):

   • $d_s$: Calculated as the mean difference divided by the pooled standard deviation of the two time points. The authors note this often overestimates the effect in paired designs.
   • $d_z$: Calculated as the mean difference divided by the standard deviation of the difference scores (raw learning gain). This accounts for the correlation between pre- and post-tests.
   • Recommendation: Report both $d_s$ and $d_z$ for paired designs to provide a complete picture.

3. Normalized Learning Gain ($d_n$):

   • Used when ceiling effects are present (e.g., scores cannot exceed 100%).
   • Formula: $\frac{Post - Pre}{1 - Pre}$.

C. Implementation Details

• Priors: The authors utilize informative normal priors for fixed effects and residual variances to ensure model stability, though they demonstrate that results are robust to prior choice via Leave-One-Out (LOO) cross-validation.
• Posterior Inference: Instead of p-values, the study relies on 95% Credible Intervals (CrI) and the full posterior distribution of the effect sizes.
• Differential Effectiveness: By extracting the posterior samples for each class, the authors compute the standard deviation of the effect size across classes, quantifying how much the intervention's effectiveness varies by classroom.

3. Key Contributions

1. Total Variance Effect Sizes: The paper provides a rigorous mathematical derivation for calculating effect sizes that incorporate all sources of variance in a multilevel model (residual, random intercepts, random slopes, and hyperparameter variances).
2. Heteroscedasticity Handling: It demonstrates how to naturally model heterogeneous residual variances across conditions and time points within the Bayesian framework, avoiding the bias of assuming homoscedasticity.
3. Differential Effectiveness Metric: A novel contribution is the ability to estimate the standard deviation of the effect size across clusters (classes). This allows researchers to see not just if an intervention works on average, but how consistently it works across different contexts.
4. Practical Tutorial: The paper serves as a comprehensive guide for using brms and Stan to implement these complex models, providing code listings for ANOVA, Welch's t-test equivalents, and paired designs.
5. Guidelines for Reporting: The authors propose a standardized reporting framework (Table 7) recommending specific effect sizes ($d_s, d_z, d_n$) based on the study design (Within-Class vs. Between-Class, Paired vs. Pooled).

4. Results

Using the provided dataset, the study found:

• Intervention Efficacy: The intervention condition outperformed the control condition at both pretest and posttest.
• Comparison of Methods:
  • Pretest: The Bayesian multilevel $d_s$ was 0.22 (95% CrI [0.07, 0.37]), compared to a classical non-multilevel $d_s$ of 0.28.
  • Posttest: The Bayesian multilevel $d_s$ was 0.21 (95% CrI [0.05, 0.37]), nearly identical to the classical 0.20.
  • Within-Group: For the control group, the Bayesian $d_s$ was 1.18 (vs. classical 1.40), and $d_z$ was 1.11 (vs. classical 1.21). The Bayesian estimates were generally lower, suggesting classical methods may overestimate gains by ignoring multilevel structure and correlations.
• Differential Effectiveness:
  • The standard deviation of $d_s$ across classes was 0.09 at pretest (homogeneous effect).
  • At posttest, the standard deviation increased to 0.18, indicating the intervention's effectiveness became more heterogeneous across classes.
  • Crucially, the Bayesian analysis revealed that in some specific classes, the control condition actually outperformed the intervention (negative class-wise $d_s$), a nuance lost in average-only reporting.

5. Significance and Implications

• Refined Interpretation: The study argues that relying solely on average effect sizes is insufficient. The "differential effectiveness" (variance of the effect size across classes) is a critical metric for educational policy, indicating that an intervention may work well in some contexts but fail in others.
• Robustness: By using Bayesian methods with Student's $t$-distributions and heterogeneous variances, the estimates are more robust to outliers and violations of normality common in educational data.
• Transparency: The approach forces researchers to explicitly model the data structure (clustering) and variance heterogeneity, reducing the risk of Type I/II errors associated with model misspecification.
• Future Research: The authors call for simulation studies to test these methods under various conditions (e.g., floor/ceiling effects, missing data) and suggest that while variable-centered (multilevel) methods are powerful, they should eventually be complemented by person-centered methods to capture individual heterogeneity.

In conclusion, this paper provides a methodological blueprint for moving beyond simple point estimates in educational research, advocating for a Bayesian multilevel approach that captures the full complexity, uncertainty, and context-dependency of educational interventions.