A median-based estimate of effect size - toward more intuitive comparisons in mathematics education research

Technical Summary: A Median-Based Estimate of Effect Size

Problem Statement
Mathematics education research frequently relies on Null-Hypothesis Significance Testing (NHST), which yields binary "significant/not significant" conclusions heavily dependent on sample size. While effect size estimates (such as Cohen's d) address the magnitude of differences, the authors argue that standard measures often fail to provide intuitive or meaningful interpretations for the specific nature of mathematics education data. Common issues include:

1. Distributional Assumptions: Cohen's d and Hedges' g assume normally distributed data and rely on means and standard deviations, making them vulnerable to skewness, heavy tails, and outliers—features common in test scores, Likert scales, and learning gains.
2. Interpretability: Standardized mean differences are unitless and often difficult for practitioners to interpret contextually.
3. Data Types: Much mathematics education data is "naturalistic" (e.g., classroom assessments) rather than strictly experimental or random samples, rendering traditional sampling assumptions and significance testing potentially disingenuous or misapplied.
4. Over-reliance on Point Estimates: Reporting only point estimates without confidence intervals can lead to over-interpretation of effects, particularly in small datasets or those with low variance.

Methodology
The authors propose a median-based indicator of effect size, denoted as M(A, B), designed to compare two numeric data sets, A and B.

• Definition: The estimator is defined as $M(A, B) = 2|\gamma(A)|/|A| - 1$, where $\gamma(A) = \{a \in A : a \geq \text{median}(B)\}$. Conceptually, this measures the probability that a random data point from A is greater than or equal to the median of B, adjusted to a scale of [-1, 1].
• Theoretical Properties:
  • It functions as a "median-based probability of dominance," contrasting with classical stochastic dominance (Prob(A > B)) used in Mann-Whitney U tests.
  • It is resistant to skewness, heavy tails, and outliers because it relies on the median rather than the mean.
  • For normally distributed data, it has a functional, non-linear relationship with Cohen's d ($M = 2\Phi(d) - 1$), but the authors argue this relationship should not be used to derive M from d for non-normal data.
  • It approximates Cliff's delta when distributions are symmetric but behaves more robustly than Cliff's delta when data is skewed.
• Inference Procedure: To address the uncertainty of point estimates, the authors employ a frequentist bootstrap procedure. They generate probability distributions for both the median-based estimator and Cohen's d by resampling with replacement (e.g., $10^4$ to $10^5$ times). This allows for the calculation of 95% confidence intervals (CIs) for the effect size means, providing a transparent view of the reliability of the estimate.

Key Contributions and Results
The paper introduces the median-based estimator as a complementary tool to existing measures, demonstrating its utility through a series of examples drawn from mathematics education literature and datasets.

1. Robustness to Non-Normality: In examples involving skewed distributions (e.g., pre-test/post-test gains, early algebra intervention data), the median-based estimator provided stable results where Cohen's d was potentially misleading due to non-normality or extreme outliers.
2. Intuitive Interpretation: The authors demonstrate that $M(A, B)$ can be interpreted visually (the proportion of one distribution exceeding the median of another) and probabilistically (e.g., "equivalent to drawing a black ball from an urn").
3. Empirical Examples:
   • Large Effects: In cases such as the impact of an early algebra intervention (Blanton et al., 2015) or grade-level pre-test differences, the median-based estimator confirmed large effects (e.g., $M \approx 0.93$) with tight confidence intervals, aligning with visual histograms.
   • Medium Effects: Comparisons of problem types (interleaved vs. blocked practice) and curriculum models showed moderate effects ($M \approx 0.38 - 0.47$) with confidence intervals bounded away from zero.
   • Small/Null Effects: In comparisons of gender differences in TIMSS scores or textbook efficacy, the median-based estimator yielded small point estimates ($M \approx 0.16$). Crucially, the bootstrap confidence intervals for these cases often included zero (e.g., $[-0.19, 0.51]$), signaling that the observed differences might be due to chance, a nuance sometimes obscured by point estimates alone.
4. Comparison with Cohen's d: The study highlights that while Cohen's d and the median-based estimator often correlate, they can diverge significantly in skewed data or when variances are small. The authors note that small variances can artificially inflate Cohen's d, whereas the median-based approach remains stable.

Significance and Claims
The authors do not claim that the median-based estimator should replace Cohen's d, Hedges' g, or other established measures. Instead, they position it as a productive addition to the researcher's repertoire for quantifying and interpreting data differences.

The paper's primary significance lies in its advocacy for:

• Contextual Grounding: Providing an effect size metric that aligns more closely with how researchers and practitioners reason about student performance (e.g., "how many students improved beyond the median of the control group?").
• Transparency and Honesty: Emphasizing that effect size is a probability distribution, not just a number. The authors argue that reporting 95% confidence intervals via bootstrap methods is essential for honest reporting, particularly for small datasets or those with low variance, to prevent over-interpretation of point estimates.
• Robustness: Offering a method that does not require the assumption of normality, making it suitable for the "naturalistic," skewed, and heterogeneous data frequently encountered in mathematics education research.

Ultimately, the paper suggests that combining the median-based estimator with confidence intervals and visual data representations (smooth histograms) better addresses the fundamental questions of whether an effect is real, how large it is, and whether it is practically useful.