📊 statistics

The Rise of Null Hypothesis Significance Testing (NHST): Institutional Massification and the Emergence of a Procedural Epistemology

This paper argues that Null Hypothesis Significance Testing (NHST) became the dominant statistical framework in postwar science not because it solved technical inference problems, but because its mechanical, context-stripping procedures functioned as a vital social technology that enabled the mass expansion and coordination of diverse scientific networks.

Original authors: Carol Ting

Published 2026-03-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Carol Ting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Puzzle: Why Do We Keep Using a "Broken" Tool?

Imagine you are a mechanic. You have a wrench that is slightly bent, doesn't fit every bolt perfectly, and sometimes gives you the wrong reading. A group of expert engineers has been screaming for 60 years, "Stop using this wrench! It's dangerous! We have better tools!"

Yet, every single mechanic in the world still uses that bent wrench. They use it to fix cars, build bridges, and even perform surgery.

This is the puzzle the paper solves. The "bent wrench" is NHST (Null Hypothesis Significance Testing), a statistical method used to decide if a scientific study is "real" or just a fluke. The "experts" are statisticians who have been trying to fix it for decades.

The author, Carol Ting, argues that we aren't keeping this tool because we are stupid or lazy. We keep it because it is the perfect tool for a specific, massive problem: the explosion of science after World War II.

The Story: How We Got Here

1. The Two Inventors (The "Forced Marriage")

In the early 1900s, two brilliant statisticians invented two different ways to do math:

Fisher was like a detective. He wanted to look at the evidence and say, "Hmm, this result is so weird that the null hypothesis (the idea that nothing is happening) is probably wrong." He believed you needed to use your brain and judgment to decide what the numbers meant.
Neyman and Pearson were like traffic cops. They didn't care about the specific case; they cared about the long run. They wanted a rule: "If the light is red, stop. If it's green, go." They wanted a system that, over 1,000 years, would make the fewest mistakes.

The Problem: These two ideas were actually incompatible. You can't be a detective and a traffic cop at the same time.

2. The Great Expansion (The "Baby Boom" of Science)

After World War II, the world changed. The US government poured billions of dollars into science. The GI Bill sent millions of veterans to college. Suddenly, everyone wanted to be a scientist.

The Demand: Universities needed to teach statistics to everyone, not just a few elite geniuses.
The Shortage: There weren't enough expert statisticians to teach everyone.

3. The "Cookbook" Solution

Because there were too many students and too few teachers, universities needed a way to teach statistics quickly. They couldn't teach the deep, complex "detective work" of Fisher or the rigorous "traffic rules" of Neyman.

So, they created a hybrid monster: NHST.

They took Fisher's "p-value" (the number that tells you if something is weird).
They took Neyman's "pass/fail" rule (if the number is below 0.05, you pass).
They threw away all the hard parts: the context, the judgment, the assumptions, and the nuance.

The Result: A "Cookbook" method. You don't need to understand why the cake rises; you just follow the recipe: "Mix ingredients, bake at 350, check if the toothpick comes out clean." If it comes out clean, you declare, "Success!"

The Core Argument: Why the "Broken" Tool is Actually a Feature

The paper argues that NHST isn't a failure of science; it's a social technology designed for mass production.

Analogy: The Airport Security Scanner

Imagine you are an airport security chief. You have 10,000 passengers to screen in one hour. You have two options:

The Expert Approach: A highly trained detective inspects every bag, asks the passenger questions, and uses their intuition to decide if they are dangerous.
- Result: It takes 20 minutes per person. You can't screen anyone else. The airport stops.
The Machine Approach: A metal detector beeps if it finds metal. If it beeps, you pat them down. If it doesn't, they walk through.
- Result: It's fast. It's automated. It doesn't care who the person is or why they have a belt buckle. It just follows a rule.

NHST is the metal detector.

It strips away the "context" (the person's story, the specific situation).
It replaces "expert judgment" with a "mechanical rule" (Is the p-value < 0.05?).
It allows universities to hire non-expert teachers to teach statistics.
It allows grant committees to quickly sort through thousands of applications without needing to be experts in every field.

Why Can't We Just Fix It?

The paper suggests that fixing NHST is hard because the "flaws" are actually features that make the system work.

The "Black Box": NHST is a "black box." You put data in, and a "Significant!" or "Not Significant!" label pops out. You don't need to know how it works inside. This is great for mass production because it makes science "portable." Anyone can use it, anywhere, without needing a PhD in math.
The "Currency": In the world of science, the p-value is like money. If you have a "Significant" p-value, you get a job, a grant, or a promotion. If you don't, you get nothing. The system is built around this currency.

The Conclusion: A Warning for the Future

The author warns us that we are in a trap.

We know NHST is technically flawed (it creates false positives, it ignores real-world importance).
But we can't stop using it because our entire system of hiring, funding, and publishing relies on it.

The paper ends with a sobering thought: We might be doing the same thing with AI and Big Data. We are creating new "black boxes" (like the Bayes Factor or machine learning algorithms) that promise to automate decisions. Just like NHST, these new tools might become popular not because they are scientifically perfect, but because they are fast, easy to use, and fit our massive, crowded system.

In short: NHST dominates not because it is the best way to find the truth, but because it is the best way to manage a world where everyone is trying to do science, and we don't have enough experts to go around. It traded accuracy for scalability.

Based on the paper "The Rise of Null Hypothesis Significance Testing (NHST): Institutional Massification and the Emergence of a Procedural Epistemology" by Carol Ting, here is a detailed technical summary:

1. Problem Statement

The paper addresses a persistent paradox in the scientific community: despite over sixty years of sustained criticism, statistical reform efforts, and documented misconceptions regarding Null Hypothesis Significance Testing (NHST), the framework remains the dominant method for adjudicating scientific claims in applied fields (social and biomedical sciences).

The Core Puzzle: Why does a method widely acknowledged by statisticians to be technically flawed, logically inconsistent, and prone to misinterpretation (e.g., conflating statistical significance with practical importance) continue to function as the "currency" of scientific validation?
The Gap: Existing explanations often focus on individual cognitive errors or lack of statistical education. The author argues these are insufficient to explain the persistence and ubiquity of NHST, suggesting a need to look beyond individual cognition to institutional and historical structures.

2. Methodology

The study employs a historical and sociological analysis grounded in Actor-Network Theory (ANT).

Theoretical Framework: The paper utilizes ANT concepts, specifically "black-boxing" (where complex processes are sealed off to function as simple inputs/outputs) and the "translation model" (how ideas are modified as they move between different networks).
Historical Reconstruction: The author traces the genealogy of NHST from its theoretical origins in the early 20th century through the post-World War II era. This involves analyzing the distinct, often incompatible, frameworks of R.A. Fisher (inductive inference/significance testing) and Neyman & Pearson (decision-theoretic hypothesis testing).
Institutional Analysis: The paper situates the rise of NHST within the specific historical context of postwar institutional massification in the United States, characterized by the GI Bill, the Cold War, massive government funding (NSF, NDEA), and the rapid expansion of higher education and scientific research.

3. Key Contributions

The paper makes several theoretical and historical contributions:

NHST as a "Forced Marriage": It clarifies that NHST is not a coherent theoretical framework but a hybrid "offspring" of two incompatible schools of thought. Fisher's framework (focused on evidential weight of a single hypothesis) was mechanically fused with Neyman-Pearson's framework (focused on long-run error control and decision rules between two hypotheses).
The Concept of "Procedural Self-Sufficiency": The author introduces this concept to explain NHST's endurance. NHST functions as a social technology that allows scientific networks to operate without deep, context-specific epistemic judgment. It purges the need for expert deliberation regarding modeling assumptions and data generation contexts.
Institutional Massification as the Driver: The paper argues that NHST did not rise because it solved the technical problem of inference, but because it solved the social problem of scaling. During the postwar boom, the demand for scientific manpower and research output vastly outstripped the supply of trained statisticians.
Reframing "Flaws" as "Features": The technical slippages of NHST (e.g., ignoring modeling assumptions, dichotomizing results, ignoring effect sizes) are reinterpreted not as errors, but as functional features that enabled scalability. By turning inference into a mechanical "cookbook" routine, it allowed non-experts to produce "defensible" scientific claims.

4. Results and Analysis

The analysis traces the stabilization of NHST through three main phases:

Pre-War (Elite Circles): Statistics was a niche, elite discipline. Fisher and Neyman-Pearson debated fiercely, but their complex theories remained within small networks of specialists. Training was apprenticeship-based and slow.
Post-War Expansion (The Supply Crisis):
- Demand Shock: The GI Bill and Cold War funding (e.g., NDEA) caused an explosion in student enrollment and research output across psychology, sociology, education, and medicine.
- Supply Lag: There was a severe shortage of qualified statisticians. Universities relied on "self-taught" instructors from non-statistical departments to teach statistics.
- The "Cookbook" Solution: To manage this crisis, pedagogy shifted toward standardized, routinized procedures. Textbooks and instructors stripped away the complex theoretical nuances of Fisher and Neyman-Pearson, creating a simplified, hybrid "NHST" that required minimal expertise to execute.
The Emergence of the Black Box:
- NHST became an Obligatory Passage Point (OPP). To get published, funded, or tenured, researchers had to use the $p \leq 0.05$ threshold.
- The procedure became procedurally self-sufficient: It allowed researchers to bypass the need for deep contextual understanding of data generation. The $p$ -value acted as a "coin flip" that simultaneously decided the outcome and served as the justification for the decision, effectively black-boxing the inferential process.
- This allowed for mass production of science: Researchers could be interchangeable, and results could be audited quickly without re-evaluating the underlying epistemic assumptions.

5. Significance and Implications

Resolution of the Reform Paradox: The paper explains why statistical reform has failed for decades. Reformers treat NHST as a technical problem to be fixed by better education or new metrics. However, the paper argues NHST is a social/institutional solution to massification. As long as the institutional pressure to scale scientific production exists, the "procedural self-sufficiency" of NHST will remain attractive.
Warning on Technical Substitution: The author warns that simply replacing NHST with other metrics (e.g., Bayes Factors) without addressing the institutional incentives for massification may lead to the same outcome: the new metric will simply be black-boxed and used as a mechanical decision rule, replicating the flaws of NHST.
Path Forward: Effective reform requires a social strategy alongside technical changes. This involves making the "institutional work" of NHST visible to practitioners—teaching them that NHST is a tool for bureaucratic coordination and credentialing, not a direct measure of truth. It calls for a shift in how scientific institutions value and evaluate research, moving away from reliance on standardized, procedurally self-sufficient metrics.

In conclusion, the paper posits that NHST is the "technology of trust" for an era of institutional massification, where the primary goal is the efficient, scalable production of scientific knowledge rather than the rigorous, context-dependent pursuit of truth.