📊 statistics

Statistical Significance Revisited

This paper examines recent calls for reforming statistical significance practices—such as abandoning the standard 0.05 threshold and moving beyond binary hypothesis testing—by analyzing the strengths and shortcomings of these proposed changes.

Original authors: Reason Machete

Published 2026-05-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Reason Machete

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Is there a real pattern in the data, or is it just a coincidence?

For nearly a century, scientists have used a specific tool called a Significance Test (and its main output, the p-value) to help answer this. Think of the p-value as a "suspicion meter." If the meter reads very low (usually below 0.05, or 5%), the detective says, "This is too unlikely to be a fluke; there must be a real pattern here!"

However, recently, many people have argued that this tool is broken. They say it leads to too many "false alarms" (finding patterns that aren't really there) and that the 5% threshold is arbitrary. This paper, written by R. L. Machete, takes a deep dive into these complaints and asks: Should we throw the tool away, tighten the rules, or just learn to use it better?

Here is a breakdown of the paper's arguments using simple analogies.

1. The Two Schools of Thought: The Gatekeeper vs. The Gambler

The paper starts by explaining the history of this tool.

Fisher (The Gatekeeper): He invented the test to see if a result was "surprising" enough to reject the idea that nothing is happening (the Null Hypothesis). He used a 5% threshold but warned against treating it like a rigid law.
Neyman & Pearson (The Gamblers): They added a second side to the bet. They asked: "If there is a real effect, how likely are we to miss it?" They introduced the idea of Type I errors (false alarms) and Type II errors (missing a real signal).

The Analogy: Imagine a metal detector at an airport.

Fisher says: "If it beeps, it's suspicious."
Neyman & Pearson say: "We need to balance the beeping. If we make it too sensitive, we catch every belt buckle (Type I error). If we make it too dull, we miss real weapons (Type II error)."

2. The Big Controversy: "Let's Make the Rules Stricter!"

Recently, some scientists (like Benjamin et al.) argued that the 5% threshold is too loose. They want to change it to 0.5% (or 0.005).

Their Logic: If we make the metal detector much harder to trigger, we will catch fewer fake alarms. This should double the number of times we successfully repeat an experiment (replication).
The Paper's Counter-Argument: The author says this is a trade-off. If you make the detector harder to trigger, you will definitely miss more real weapons (Type II errors).
- The Cost: You might stop finding real discoveries because the bar is set so high that only the "super obvious" ones pass.
- The Math: The paper shows that while lowering the threshold does reduce false alarms, it doesn't automatically fix the "replication crisis" unless you already know how many true discoveries exist in the first place—which is usually impossible to know.

3. The "Bayesian" Detour: Guessing the Odds

Some critics argue that p-values are misleading because they don't account for how likely a theory was to begin with. They suggest using Bayesian methods, which require you to guess the "prior probability" (how likely you think the hypothesis is before you even start).

The Paper's View: The author argues that in the real world, we often cannot know these prior probabilities. You can't easily guess what percentage of all scientific theories are actually true.
The Metaphor: Trying to use Bayesian methods for every study is like trying to drive a car by guessing the traffic density before you even leave the driveway. It's a nice idea, but in practice, we often don't have that data. The paper suggests that sticking to error probabilities (how often the test makes a mistake) is more practical.

4. The "Throw It Away" Movement

Some reformers say: "Stop using p-values and thresholds entirely! Just report the numbers and let people decide."

The Paper's View: The author thinks this is dangerous.
- The Analogy: Imagine a doctor saying, "Don't tell me if the patient has a fever or not. Just give me the temperature and let me guess."
- The Problem: Humans are bad at making binary decisions (Yes/No) without a clear line. If you remove the threshold, you haven't removed the decision; you've just hidden the line. Eventually, someone still has to decide if the result is "good enough."
- Confidence Intervals: The paper agrees that we should report Confidence Intervals (a range of likely values), but it argues these shouldn't replace p-values. They should work together, like a map and a compass.

5. The "Severity" Test: A New Way to Look at the Data

The paper introduces a concept from a philosopher named Mayo called Severity.

The Idea: It's not enough to just pass a test. You have to ask: "If this hypothesis were false, how likely would it have been to pass this test anyway?"
The Metaphor: Imagine a suspect passes a lie detector.
- Standard Test: "They passed, so they are innocent."
- Severity Test: "If they were guilty, would this specific lie detector have caught them? If the machine is broken and lets everyone pass, then 'passing' doesn't prove anything. But if the machine is very strict and they still passed, that is severe evidence of innocence."
The author suggests using this "severity" thinking to dig deeper into the data rather than just looking at a single number.

6. Real-World Example: African Savanna Fires

To prove these tools work, the author looks at data about fires in Africa.

The Question: Do fires happen in 5-year cycles?
The Result: The math showed a strong pattern (a very low p-value).
The Application: The author didn't just say "It's significant." They used the Severity concept to say, "We are 95% sure that the cycle is real and not a fluke." They also used Confidence Intervals to show exactly how strong that cycle is.
The Lesson: The tools worked together to give a clear, confident answer without needing to change the rules of the game.

The Final Verdict

The paper concludes that we are at a crossroads.

Don't throw the tools away: P-values and significance tests are still useful for distinguishing real science from noise.
Don't just tighten the screws: Making the threshold 0.005 might reduce false alarms, but it will also hide real discoveries.
Use the tools wisely: Scientists should be the masters of the tools, not the slaves. They should use p-values, confidence intervals, and severity checks together to make informed decisions.
The Real Enemy: The problem isn't the math; it's the behavior. Things like "data dredging" (searching for patterns until you find one) and "optional stopping" (stopping an experiment as soon as you get a good result) are the real causes of bad science. Changing the p-value threshold won't fix bad behavior; only better ethics and incentives will.

In short: The paper argues that we shouldn't panic and rewrite the rules of statistics. Instead, we should trust the existing tools, use them with more nuance (like checking for "severity"), and focus on fixing the human behaviors that lead to bad science.

Technical Summary: Statistical Significance Revisited

Problem Statement
The paper addresses the ongoing controversy surrounding Null Hypothesis Significance Testing (NHST), specifically the reliance on the $p$ -value and the standard significance threshold of $\alpha = 0.05$ . Recent calls for reform, including proposals to lower the threshold to 0.005, abandon significance testing entirely in favor of confidence intervals, or report $p$ -values continuously without thresholds, have sparked debate regarding the reproducibility crisis in science. The author identifies a lack of consensus on how to balance Type I and Type II errors, the validity of using Bayesian diagnostic screening formulas to justify threshold changes, and the feasibility of replacing error-statistical methods with decision theory or cost-benefit analysis.

Methodology
The paper employs a theoretical and analytical approach to evaluate various reform proposals. The methodology includes:

Theoretical Derivation: The author derives the relationship between the significance threshold ( $\alpha$ ), the Type II error rate ( $\beta$ ), and the False Positive Rate (FPR) using the diagnostic screening formula (Bayesian posterior probability of the null). This involves analyzing the partial derivatives of the FPR with respect to $\alpha$ and $\beta$ .
Trade-off Analysis: The paper mathematically demonstrates the inherent trade-off between Type I and Type II errors in frequentist frameworks, contrasting this with the assumptions made in diagnostic screening models where reducing Type I error is claimed to simultaneously reduce FPR without increasing Type II error.
Reproducibility Probability Modeling: The distribution of $p$ -values under the alternative hypothesis is analyzed using the log-normal approximation (based on Hung et al., 1997) to derive formulas for reproducibility probability.
Cost Function Analysis: A frequentist cost minimization framework is constructed to determine optimal critical values based on Type I and Type II error costs, comparing this to Bayesian expected utility theory.
Severity Assessment: The paper applies Mayo's (2018) severity function to evaluate claims, demonstrating how severity can supplement confidence intervals without abandoning hypothesis testing.
Illustrative Example: The theoretical concepts are applied to a real-world dataset regarding the autocorrelation of burnt area proportions in sub-Saharan African savannas (2001–2020) to demonstrate the calculation of $p$ -values, confidence limits, and severity.

Key Contributions

Critique of Threshold Reduction: The paper challenges the proposal to lower the significance threshold from 0.05 to 0.005. It demonstrates that while lowering $\alpha$ reduces the False Positive Rate, it inevitably increases the Type II error rate ( $\beta$ ). The author argues that the net benefit depends heavily on the "true positive rate" (or positive predictive value) at the current threshold, a quantity that is practically impossible to estimate accurately in most fields.
Bayesian vs. Frequentist Distinction: The author clarifies that the diagnostic screening formula used by proponents of threshold reduction (e.g., Benjamin et al., 2017) is fundamentally Bayesian, relying on prior probabilities ( $\phi$ ) that are often unknown or uncomputable. The paper argues that mixing this Bayesian posterior probability with frequentist error probabilities creates confusion.
Defense of Thresholding: The paper argues that abandoning thresholds entirely is impractical because decision-making often requires dichotomous choices. It posits that confidence intervals and $p$ -values are dual to one another and both rely on underlying thresholds; reporting $p$ -values continuously without a threshold merely creates an "undisclosed threshold."
Cost-Benefit Limitations: The analysis of cost functions shows that the optimal critical value for minimizing expected cost often does not align with standard significance levels (e.g., 0.05) and that in many real-world scenarios (e.g., global warming, drug development), quantifying the specific costs of errors is impractical.
Severity as a Supplement: The paper advocates for the use of severity functions to probe the parameter space further after a significance test, offering a way to assess the strength of evidence for specific claims without discarding the error-statistical framework.

Results

Threshold Reduction Analysis: Mathematical analysis confirms that reducing $\alpha$ reduces the False Positive Rate even when $\beta$ increases, but the magnitude of this reduction is highly sensitive to the prior odds ratio. The paper derives that doubling the replication rate via a 10-fold reduction in the threshold is only possible if the prevailing true positive rate is exactly 4/9.
Reproducibility Probability: The paper finds that reproducibility probabilities derived from $p$ -value distributions are theoretical constructs that may not reflect empirical reality, as the prevailing scenario is not necessarily the alternative hypothesis.
Cost Analysis: In the illustrative cost function graphs, the minimizing critical value was found to be less than 1.5 (corresponding to $\alpha > 0.07$ ), suggesting that standard thresholds are not always the cost-minimizing choice, yet the application of cost analysis is often limited by the inability to quantify error costs.
Savanna Fire Example: In the application to savanna fires, a $p$ -value of $2.37 \times 10^{-4}$ led to the rejection of the null hypothesis. The severity function confirmed that the claim $\beta_1 > 0.30529$ passed with a severity of 0.95, and the calculated reproducibility probability was 93.8%.

Significance and Claims
The paper claims that the current calls for reform often fail to account for the fundamental trade-offs between Type I and Type II errors and the practical impossibility of estimating prior probabilities required for Bayesian diagnostic screening. The author argues that:

Significance tests should not be abandoned but rather understood as tools for controlling error probabilities, not as definitive proofs of truth.
The "replication crisis" is not solely a statistical threshold issue but involves broader scientific practices (e.g., data dredging, optional stopping).
While confidence intervals and severity assessments are valuable supplements, they do not eliminate the need for thresholds in decision-making.
The scientific community should not be "slaves" to rigid thresholds but should use them as guides while embracing uncertainty. The author concludes that innovation thrives under freedom and that the 0.05 threshold, despite its flaws, has served as a faithful tool for identifying false discoveries, and that science is ultimately self-correcting. The paper aligns with the American Statistical Association's (ASA) stance that $p$ -values should not be the sole basis for decisions but should not be retired.