Hypothesizing an effect size by considering individual variation

1. Problem Statement

The paper addresses a critical gap in experimental design and statistical inference: the difficulty researchers face in formulating realistic hypotheses regarding average treatment effects (ATE) prior to conducting a study.

• Current Limitations: Existing approaches to hypothesizing effect sizes are flawed:
  • Meta-analysis/Literature Review: Suffers from "garbage-in-garbage-out" due to publication bias and selection on statistical significance.
  • Expert Elicitation: Experts often rely on biased literature or personal hopes, leading to overconfidence.
  • Cohen's Conventions: Standard labels (small, medium, large) are often wildly optimistic in modern contexts of incremental improvement.
  • Pilot Studies: Estimates from small pilot studies are too noisy to be useful for power calculations, often leading to severe underestimation of required sample sizes and overestimation of power.
  • Bayesian "Cringe": A cultural reluctance to make restrictive assumptions leads to overly diffuse priors and unrealistically large effect size expectations.
• Consequences: These flawed hypotheses result in studies with low statistical power, inflated Type M (magnitude) and Type S (sign) errors, and contribute to the replication crisis. Researchers often fixate on a single average effect number, ignoring the underlying heterogeneity of individual responses.

2. Methodology

The authors propose a paradigm shift: instead of hypothesizing a single scalar value for the average effect, researchers should hypothesize a distribution of individual treatment effects. This approach embeds the specific problem of estimating an average into a larger, more tractable problem of modeling variation.

The Proposed Protocol (Anchoring and Adjustment)

The method utilizes a heuristic of "anchoring and adjustment" to construct a distribution of effects:

1. Anchor (Plausible Effect Size): Start with a realistic effect size ($X$) assuming the treatment works as intended for a responsive individual (based on literature or subject-matter reasoning).
2. Define Range: Consider the range of effects under real-world conditions (e.g., from 0 to $X$, or potentially negative values).
3. Construct Distribution: Convert this range into a probability distribution (e.g., a normal distribution centered at $0.5X$).
4. Estimate "Pure Nulls": Hypothesize the proportion of the population ($p_{null}$) for whom the effect is exactly zero (e.g., non-compliers, those metabolically unable to respond, or those unaffected by the mechanism).
5. Calculate ATE: The expected average treatment effect is derived as:
   $$ATE \approx (1 - p_{null}) \cdot 0.5X$$
   • Rule of Thumb: For direct interventions, divide the plausible effect by 4. For indirect interventions (e.g., marketing), divide by 20 or more.

Formative Interviews

The authors tested this protocol with three researchers (medical/engineering). They found that while researchers could easily identify individuals with the largest and smallest effects, they struggled to estimate the proportion of the population falling in the middle. This validated the approach of defining the range by extremes and focusing the researcher's cognitive load on estimating the proportion of "pure nulls."

3. Key Contributions and Results

The paper demonstrates the application of this framework across three domains, showing how it corrects overly optimistic expectations:

• Medicine (Binary Outcomes):

  • Scenario: A drug intended to prevent death.
  • Insight: In an efficacy trial (tightly controlled), the sample may be enriched with "save-able" patients, yielding a high ATE (e.g., 0.65). However, in an effectiveness trial (broad population), the sample includes many "unsave-able" (Type 3) and "already-safe" (Type 1) individuals.
  • Result: The same drug might show an ATE of 0.20 in a real-world setting. The framework forces researchers to explicitly model the mix of patient types, preventing the uncritical transfer of efficacy trial results to broader populations.

• Political Science (Numerical Data):

  • Scenario: The effect of knowing a gay person on attitudes toward same-sex marriage.
  • Insight: Many respondents are at the extremes of the scale (1 or 5) and cannot move. Only a fraction of the population is "malleable."
  • Result: By modeling the population as a mix of fixed views and malleable views, the authors derived a realistic ATE of 0.1 on a 5-point scale, explaining why previous studies found small, barely significant effects.

• Critique of Published Literature:

  • Scenario: A study claiming a 42% increase in adult earnings from a childhood intervention.
  • Insight: An average effect of 42% implies that for the effect to vary realistically (some near zero, some massive), the "success" cases would need to show >100% gains, which is implausible.
  • Result: The framework suggests the reported 42% is a massive overestimate driven by low power and selection bias, reinforcing skepticism toward such claims.

4. Significance and Implications

The paper offers a fundamental reorientation of how statistical power and priors are conceptualized:

1. Inversion of Workflow: Instead of starting with an average effect and asking if it varies, researchers should start with the distribution of individual effects and the composition of the sample. The average effect becomes a derived consequence of these assumptions.
2. Contextualizing Study Purpose: The paper highlights that the "correct" effect size depends on the study's goal:
   • Proof of Concept: Focus on the sub-population with the largest effect (maximizing power).
   • Population Average: Requires representative sampling or post-stratification to account for heterogeneity.
   • Mechanism Understanding: Requires sampling to estimate the variance of the effect itself.
3. Addressing the Replication Crisis:
   • The framework explains "failed replications" not as evidence of non-existence, but as evidence of heterogeneous treatment effects across different populations or contexts.
   • It moves the field away from the binary "is there an effect?" (Null Hypothesis Significance Testing) toward modeling response surfaces and dynamic relations.
4. Practical Design: By forcing researchers to explicitly consider "pure nulls" and the range of individual responses, the method naturally leads to more conservative (and realistic) effect size estimates, preventing the "winner's curse" where significant findings are grossly overestimated due to low power.

Conclusion

Gelman et al. argue that treating treatment effects as a single underlying quantity is a conceptual error. By hypothesizing a distribution of effects rather than a single number, researchers can construct more robust priors, design studies with appropriate power, and interpret results with a necessary nuance regarding population heterogeneity. This approach transforms effect size estimation from a guess into a structured exercise in modeling real-world variation.