📊 statistics

Critical Values Robust to P-hacking

This paper proposes a model of hypothesis testing that incorporates p-hacking to derive larger, "robust" critical values—calibrated using medical science data to be equivalent to classical values at one-fifth the significance level—thereby ensuring that significant results occur with the desired frequency even when researchers adjust their behavior to new standards.

Original authors: Adam McCloskey, Pascal Michaillat

Published 2026-07-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam McCloskey, Pascal Michaillat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. In the world of science, this "mystery" is often a question like, "Does this new medicine actually work?" or "Is this coin fair?" To solve it, scientists use a special tool called a hypothesis test. Think of this test as a very strict judge. The judge starts by assuming the suspect is innocent (this is called the "null hypothesis"). The judge only declares the suspect guilty (rejects the null) if the evidence is so overwhelming that it would happen by pure chance less than 5% of the time. This 5% limit is called the significance level, and the specific number the evidence must beat to be considered "guilty" is the critical value.

But here is the problem: real life isn't always as neat as a courtroom. Scientists, just like anyone else, want to win. They want their findings published, their careers to soar, and their theories to be famous. This creates a huge temptation to "cheat" a little bit, not by lying, but by trying again and again until they get the result they want. This is called p-hacking. It's like rolling a die over and over again until you finally get a six, and then only telling your friends about that one six while hiding all the ones, twos, and threes. If everyone does this, the "innocent" suspects start getting convicted way too often, and the whole justice system of science starts to crumble. This is why the "replication crisis" is happening: many famous scientific discoveries turn out to be false because the original scientists kept rolling the dice until they got lucky.

This paper by Adam McCloskey and Pascal Michaillat tackles this cheating problem head-on. They don't try to stop scientists from rolling the dice; they know that as long as the rewards are high, scientists will keep trying. Instead, they built a mathematical model to figure out how to change the rules of the game so that even if scientists keep rolling, the "innocent" suspects are still protected. They found that the current rules are too easy to beat. To fix this, they propose a new, tougher "critical value" that acts as a much higher bar for the evidence to clear.

Here is how their solution works, using a playful analogy. Imagine a scientist is a gamer trying to beat a level in a video game. The "level" is finding a significant result. In the old system, the game had a low difficulty setting. If the scientist failed the first time, they could just hit "retry" as many times as they wanted until they won. Because they could keep trying, they eventually won almost every time, even if the game was supposed to be impossible to beat by chance. The paper shows that this makes the "win" meaningless.

The authors suggest a new rule: Make the game harder. But not just a little harder. They realized that if you make the game harder, the scientist will just play more times to try to win. So, you have to make the game so hard that even with all the extra tries, the scientist only wins the right amount of times (5% of the time, if that's the goal).

To figure out exactly how hard to make the game, the authors looked at real-world data from medical studies. They found that about 20% of studies are stopped before they are finished because the scientists run out of money, time, or energy. This means there is an 80% chance a scientist can finish a single experiment. Using this number, they calculated that to stop the cheating, the "difficulty" of the game needs to be increased drastically.

Their main finding is a simple, powerful rule of thumb: To fix p-hacking, you need to act as if your significance level is five times smaller than it actually is.

If a scientist wants to claim a result is significant at the standard 5% level, they shouldn't use the usual critical value (which is 1.96 for a two-sided test). Instead, they should use the critical value that corresponds to a 1% level (which is 2.58).

Why does this work? The authors show that when you raise the bar to 2.58, scientists will indeed try more times. They will run more experiments, drop more data points, and tweak more models in their desperate attempt to cross that higher line. But because the line is so high, even with all that extra effort, they will only succeed in crossing it by pure luck about 5% of the time. The "cheating" is still happening, but it no longer breaks the system.

The paper explicitly rules out the idea that we can just tell scientists to "stop cheating" or that we can fix this by simply counting how many times they tried. The authors argue that scientists will always adjust their behavior to whatever rules we set. If we set a rule that says "you can only try three times," scientists will just find a way to try four times. The only way to guarantee the system works is to set a critical value that accounts for the fact that scientists will keep trying until they run out of resources.

The authors are quite sure about their math. They didn't just guess; they built a rigorous model using probability theory and "optimal stopping" (a fancy way of saying "when to quit the game"). They tested their idea with different scenarios, like adding costs to experiments or making later experiments harder to finish, and their solution held up. They also compared their findings to a popular suggestion by other researchers to simply lower the significance level to 0.5%. The authors suggest that while lowering the level is a good idea, their model shows exactly how much to lower it based on how much resources scientists have to keep trying.

In the end, the paper offers a practical shield for science. It suggests that if we want to trust our scientific results again, we need to stop using the old, easy-to-beat numbers. We need to adopt these "robust critical values." For a standard two-sided test, that means moving the goalpost from 1.96 to 2.58. It's a higher bar, yes, and it means fewer "breakthroughs" will be announced every day. But the trade-off is that the breakthroughs we do announce will actually be real, and the "innocent" null hypotheses will finally be safe from being convicted by a lucky roll of the dice.

Technical Summary: Critical Values Robust to P-hacking

Problem Statement
P-hacking—the practice of engaging in behaviors to increase the likelihood of reporting statistically significant results—is prevalent in scientific research but absent from classical hypothesis testing theory. Consequently, under classical critical values, true null hypotheses are rejected more frequently than the nominal significance level ( $\alpha$ ) suggests. This occurs because scientists face strong incentives (publication, career advancement) to hunt for significance and possess sufficient flexibility in data collection and analysis to do so without violating norms. Classical corrections for multiple testing (e.g., Bonferroni) fail to resolve this issue because they treat the number of experiments as fixed. In reality, scientists adjust their behavior in response to new standards; if a correction makes significance harder to achieve, scientists will simply conduct more experiments until they find a significant result, thereby overwhelming the correction.

Methodology
The authors develop a structural model of hypothesis testing that explicitly incorporates p-hacking behavior using optimal stopping theory.

The Scientist's Environment: A scientist tests a null hypothesis ( $H_0$ ) against an alternative ( $H_1$ ). The scientist receives a high payoff ( $v_s$ ) for a significant result and a lower payoff ( $v_i$ ) for an insignificant one.
Resource Constraints: Conducting experiments consumes finite resources (time, money, stamina). The total resource limit $L$ follows an exponential distribution. The cost of each experiment is random. A scientist stops when resources are exhausted (index $K$ ) or when a significant result is found.
P-hacking Strategy: The scientist conducts experiments sequentially ( $n=1, 2, \dots$ ). If a result is insignificant and resources remain, the scientist continues to the next experiment. If a result is significant, the scientist stops and reports it. If resources are exhausted before significance is reached, the scientist reports the best (maximum) statistic obtained so far, which is necessarily insignificant.
Completion Probability ( $\gamma$ ): The model defines $\gamma$ as the probability that the first experiment is completed before resources are exhausted. This parameter captures the ease of conducting research and the prevalence of resource constraints.
Derivation of Robust Critical Values:
- The authors derive the expected number of experiments $E(N(z))$ run by a scientist under a critical value $z$ when $H_0$ is true.
- They calculate the actual probability of a Type I error ( $S^*(z)$ ), which is the probability that the reported statistic exceeds $z$ . This is shown to be $S^*(z) = S(z) \times E(N(z))$ , where $S(z)$ is the classical survival function.
- The robust critical value ( $z^*$ ) is defined as the value that satisfies $S^*(z^*) = \alpha$ . This ensures that even when scientists optimally adjust their p-hacking behavior to the new standard, the frequency of false rejections remains at the desired level $\alpha$ .

Key Contributions and Results

Optimal Stopping Behavior: The model demonstrates that the optimal strategy for a scientist is to continue experimenting until a significant result is found or resources are depleted. This leads to a geometric distribution of the stopping time.
The Robust Critical Value Formula: The paper derives a nonstandard Bonferroni correction. The robust critical value $z^*$ for a significance level $\alpha$ is equivalent to the classical critical value for a reduced significance level $\alpha^*$ , where:
$\alpha^* = \frac{\alpha}{E(N(z^*))}$
Crucially, $E(N(z^*))$ is the expected number of experiments under the robust critical value, not the classical one. The authors show that $z^*$ is implicitly defined by:
$z^* = Z\left( \alpha \cdot \frac{1 - \gamma}{1 - \alpha\gamma} \right)$
where $Z$ is the inverse survival function of the test statistic.
Behavioral Response: Unlike previous corrections, this approach accounts for the behavioral response: as the critical value becomes more stringent, scientists p-hack more (increase $E(N)$ ). The robust critical value is always larger than the classical critical value to offset this increased experimentation.
Generalizability: The robust critical value derived under the assumption of independent experiments remains valid for a wide range of other p-hacking strategies (e.g., pooling data, removing outliers, searching specifications) as long as these strategies induce positive dependence across test statistics.

Numerical Illustration and Calibration
The authors calibrate the model using data from the medical sciences (Dwan et al., 2008) regarding the lifecycle of studies.

Calibration: They estimate the completion probability $\gamma$ by analyzing the fraction of approved studies that were never started or stopped early. The estimated completion probability is $\gamma = 80\%$ .
Rule of Thumb: With $\gamma = 0.8$ $γ = 0.8$ , the effective significance level required to maintain a 5% Type I error rate is approximately $\alpha/5$ $α /5$ .
- For a one-sided z-test with $\alpha=5\%$ , the robust critical value is 2.33 (vs. classical 1.64).
- For a two-sided z-test with $\alpha=5\%$ , the robust critical value is 2.58 (vs. classical 1.96).
Sensitivity: The robust critical values are relatively insensitive to variations in $\gamma$ . Even if $\gamma$ varies between 70% and 90%, the critical values for a two-sided test remain between 2.42 and 2.79.

Significance and Claims
The paper claims to provide a theoretical foundation for adjusting significance standards to account for the reality of p-hacking.

Restoring Credibility: By using robust critical values, the scientific community can ensure that the rate of false positives (Type I errors) does not exceed the advertised significance level, even in the presence of optimal p-hacking.
Comparison to Alternatives: The authors contrast their approach with pre-analysis plan registration. While registration prevents certain forms of p-hacking, it may stifle exploratory data analysis. Robust critical values allow for exploration while maintaining statistical validity. Furthermore, registration does not prevent scientists from repeating experiments until a significant result is found; robust critical values address this specific limitation.
Theoretical Link to Proposals: The analysis provides a theoretical underpinning for proposals to lower significance levels (e.g., from 5% to 0.5% as suggested by Benjamin et al., 2018). The paper shows that the magnitude of the required reduction depends on the completion probability (resource availability) of the field. A 10-fold reduction corresponds to a completion probability of 90%, while a 5-fold reduction corresponds to 80%.

The paper concludes that robust critical values are a necessary tool to maintain the informativeness of hypothesis tests in an environment where scientists have incentives and opportunities to hunt for significance.

Technical Summary: Critical Values Robust to P-hacking

More like this