📊 statistics

The Difference Between "Replicable" and "Not replicable" is not Itself Scientifically Replicable

This paper argues that aggregating binary replication verdicts across non-exact experiments fails to reliably distinguish between "replicable" and "not replicable" results because inherent heterogeneity creates irreducible uncertainty and unidentifiable variance, thereby undermining the statistical foundations used to declare a replication crisis.

Original authors: Berna Devezer, Erkan O. Buzbas

Published 2026-04-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Berna Devezer, Erkan O. Buzbas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why We Can't Easily Tell "True" Science from "False" Science

Imagine you are a judge trying to decide if a new recipe is "good" or "bad." You ask 50 different chefs to try making it.

If 45 chefs say, "It tastes great!" and 5 say, "It's terrible," you might conclude the recipe is replicable (good).
If 25 say "Great" and 25 say "Terrible," you might conclude it is not replicable (bad).

This paper argues that this way of judging is broken.

The authors, Berna Devezer and Erkan O. Buzbas, claim that in modern science, we cannot reliably tell the difference between a "good" result and a "bad" one just by counting how many times it worked in different labs. Even if you run thousands of experiments, the math says the line between "replicable" and "not replicable" is invisible.

The Core Problem: The "Perfect Copy" Myth

To understand why, we need to look at what a "replication" actually is.

An Exact Replication: This is like a photocopier. You take a document, and the machine spits out an identical copy. In science, this would mean every single detail (the same people, the same equipment, the same time of day, the same instructions) is identical, and only the random luck of the draw changes.
The Reality: Exact copies are almost impossible. In real life, replications are like hand-copying a document. One person uses a blue pen, another uses a pencil. One writes on a desk, another on a train. One is tired, one is energetic.

The paper calls this "non-exactness." Because every lab is slightly different, every experiment is slightly different.

The Two Models: The "Shared Secret" vs. The "Independent Gamble"

The authors use two statistical models to explain what happens when we mix these imperfect copies.

1. The Benchmark Model (The "Shared Secret")
Imagine a group of chefs who are all secretly following the same master recipe, but they are all slightly bad at reading it.

If the recipe is "hard to read" (high non-exactness), even if you get 1,000 chefs to try it, their results will still be messy.
The Lesson: There is a "floor" to how precise you can be. No matter how many more chefs you add, the uncertainty never goes away because the "messiness" of the recipe itself is the problem, not the number of chefs.

2. The Operational Model (The "Independent Gamble")
This is how science actually works. Each lab runs its own unique experiment and reports a simple "Yes" (it worked) or "No" (it failed).

The authors show that when you only get a "Yes/No" answer from each lab, you lose all the information about how different the labs were.
The Analogy: Imagine you are trying to guess the average temperature of a city.
- Scenario A: You ask 100 people to look at the same thermometer. They all see the same number, but maybe they are all looking at it wrong.
- Scenario B: You ask 100 people to look at 100 different thermometers. Some are broken, some are in the sun, some are in the shade.
- If you only ask them "Is it hot? (Yes/No)," you cannot tell the difference between Scenario A and Scenario B. You just get a list of "Yes" and "No." You have no idea if the thermometers are broken or if the weather is just chaotic.

The "Replication Crisis" is a Statistical Illusion

The paper argues that the famous "Replication Crisis" (the idea that half of science is fake) might be a misunderstanding caused by bad math.

The Trap: Scientists often say, "We tried to replicate 100 studies, and only 36% worked. Therefore, science is in crisis."
The Reality: The authors say that 36% is just a number that mixes apples and oranges. Because every lab is different (non-exact), the "36%" doesn't tell us if the original science was bad, or if the labs just couldn't agree on how to measure it.
The Verdict: You cannot look at a list of "Yes/No" results and say, "This result is definitely true" or "This result is definitely false." The data structure simply doesn't contain enough information to make that call.

Why "More Replications" Doesn't Fix It

Usually, we think: "If we aren't sure, let's do more experiments!"

The Paper's Twist: If the experiments are "non-exact" (different labs, different people, different tools), adding more experiments is like adding more people to a room where everyone is shouting in different languages. You just get more noise.
The "Effective Sample Size": The authors show that a study with 100 different labs might only have the statistical power of 5 or 6 perfect, identical experiments. The "noise" of the differences between labs cancels out the benefit of having more labs.

The Many Labs 4 Example

The authors tested their theory on a real project called "Many Labs 4," where 17 labs tried to replicate a famous psychology experiment about thinking about death.

The Result: The labs were all different. Some did it online, some in person. Some used different questions.
The Math: When they calculated the "messiness" (heterogeneity) of these 17 labs, they found it was so high that the data could not possibly tell them if the original result was true or false. The "Yes/No" answers were too blurry.
The Conclusion: Even with a massive, well-funded project, the data was too "fuzzy" to declare a winner.

Summary: What Should We Do?

The authors are not saying we should stop doing science or stop replicating.

Replication is still good for learning how things work, finding the right measurements, and understanding the details.
Replication is bad when we try to use it as a simple "Pass/Fail" test to declare a whole field of science a "crisis" or a "success."

The Final Metaphor:
Trying to decide if a scientific result is "real" by counting "Yes/No" votes from different labs is like trying to judge the quality of a song by asking 100 people to hum it back to you. Some will hum it in a different key, some will forget the lyrics, and some will hum it faster. If you just count how many people "got it right," you aren't judging the song; you are just judging how well the crowd can mimic the song.

The paper concludes that the tools we are currently using to declare a "crisis" in science are mathematically incapable of doing the job they are assigned. We cannot reliably draw a line between "replicable" and "not replicable" with the current methods.

1. Problem Statement

The paper addresses a fundamental flaw in current metascientific practices regarding replicability rates. The scientific community routinely aggregates binary verdicts (replicated vs. not replicated) from multiple studies to estimate a mean replicability rate ( $\mu$ ) for a field or a specific result. This rate is then used to demarcate "trustworthy" results from "suspect" ones or to declare a "replication crisis."

The authors argue that this practice is statistically unsound because:

Non-exactness: Real-world replications are rarely "exact" (statistically equivalent to the reference). They differ in subject pools, procedures, and contexts, introducing heterogeneity.
Data Structure Limitation: Standard replication studies report only one binary verdict per experiment. This data structure lacks the information necessary to distinguish between sampling noise and between-experiment heterogeneity.
The Core Claim: The difference between a "replicable" result (high $\mu$ ) and a "not replicable" result (low $\mu$ ) cannot be reliably distinguished using current methods, even with large numbers of replications. The statistical machinery used to produce these verdicts loses its discriminatory power under realistic conditions of non-exactness.

2. Methodology

The authors formalize the problem using two distinct statistical models to capture the ontological sources of non-exactness:

A. The Benchmark Model (Shared Latent Rate)

Assumption: All $m$ replications are exchangeable and depend on a single latent replicability rate $\phi$ , which is drawn from a population distribution with mean $\mu$ and intraclass correlation $\rho$ .
Mechanism: The total number of successes follows a Beta-Binomial distribution.
Key Insight: Even if $\rho$ (the degree of heterogeneity) is known, the variance of the estimated mean $\hat{\mu}$ has an irreducible variance floor as $m \to \infty$ :
$\lim_{m \to \infty} V(\hat{\mu}) = \mu(1-\mu)\rho$
This means adding more replications cannot eliminate uncertainty if $\rho > 0$ . High and low replicability regimes will have overlapping confidence intervals, making them indistinguishable.

B. The Operational Model (Conditionally Independent Rates)

Assumption: This model reflects actual practice where each of the $m$ experiments is unique (non-exact) and has its own replicability rate $\phi_i$ , drawn independently from a population distribution.
Data Structure: Each experiment $i$ contributes exactly one binary verdict ( $k_i = 1$ ).
Key Insight: When $k_i = 1$ , the parameter $\rho$ (heterogeneity) becomes non-identifiable from the data. The joint distribution of the binary outcomes collapses to a simple Binomial distribution $Bin(m, \mu)$ , regardless of the true value of $\rho$ .
Consequence: Researchers cannot tell if they are operating in a low-heterogeneity regime (where more replications help) or a high-heterogeneity regime (where they do not). The data structure systematically understates uncertainty.

C. Numerical Examples and Reanalysis

The authors validate their theory through:

Generative Models: Simulating heterogeneity arising from (a) sample heterogeneity (different subject populations) and (b) stimulus/dosage heterogeneity (delivery errors and bias). They show how these physical sources map to high $\rho$ values.
Case Study (Many Labs 4): A reanalysis of the "Terror Management Theory" replication project.
- They used an auxiliary Normal random-effects model on continuous effect sizes (Hedges' $g$ ) to estimate the implied $\rho$ .
- They mapped these estimates to the Benchmark Model to demonstrate that the resulting variance intervals are so wide that binary classification is impossible.

3. Key Contributions

Formalization of Non-Identifiability: The paper proves that under the standard "one binary verdict per experiment" design, the degree of non-exactness ( $\rho$ ) is mathematically unidentifiable. Consequently, the precision of the mean replicability rate cannot be assessed.
The Irreducible Variance Floor: It establishes that heterogeneity creates a hard limit on precision. Unlike sampling error, which decreases with $N$ , heterogeneity-induced variance remains constant regardless of the number of replications.
The "Replication Paradox": The authors demonstrate that increasing the number of replications in a heterogeneous setting can lead to a false sense of precision (if $\rho$ is ignored) or simply fail to distinguish high/low replicability (if $\rho$ is acknowledged).
Critique of Aggregation: They argue that aggregating replicability rates across diverse literatures (e.g., "Psychology as a whole") produces a meaningless average that conflates incommensurable experimental regimes.

4. Key Results

Loss of Discriminability: Under realistic heterogeneity (e.g., $\rho = 0.15$ ), the 95% Highest Density Intervals (HDI) for a "high" replicability rate (e.g., $\mu=0.80$ ) and a "low" rate (e.g., $\mu=0.20$ ) overlap significantly, even with $m=50$ replications.
Effective Sample Size: The effective number of independent replications ( $m_e$ ) drops drastically as $\rho$ increases. For example, in the Reproducibility Project: Psychology ( $m=100$ ), if $\rho=0.20$ , the inferential weight is equivalent to only 5 exact replications.
Many Labs 4 Analysis:
- The estimated $\rho$ for the ML4 sequence was high (posterior mean $\approx 0.37$ to $0.59$).
- At this level of heterogeneity, the 95% HDI for the estimated replicability rate spans nearly the entire [0, 1] interval.
- The data cannot distinguish whether the underlying effect is truly replicable or not; the sequence is "non-diagnostic."
Bias and Noise: Systematic bias (e.g., under-delivery of stimuli) shifts the mean $\mu$ , while noise inflates $\rho$ . Both distort the apparent replicability rate and undermine the ability to make binary verdicts.

5. Significance and Implications

Refutation of the "Crisis" Narrative: The authors argue that the "replication crisis" cannot be established by current methods. A low replicability rate (e.g., 36%) is statistically indistinguishable from a healthy literature full of small, noisy studies because the base rate of true effects and the degree of heterogeneity are unknown.
Demarcation Failure: The binary distinction between "replicable" and "not replicable" is not a scientifically replicable property of a result under current methodologies.
Recommendations for Practice:
- Stop Binary Verdicts: Researchers should avoid reducing replication outcomes to simple "success/fail" counts for demarcation purposes.
- Richer Data Structures: To recover information about heterogeneity, studies need continuous outcomes, multiple measurements per experiment ( $k > 1$ ), or hierarchical designs.
- Shift in Focus: Replication should be used for estimating effect magnitudes, characterizing variability, and calibrating procedures, rather than acting as a "tribunal" to judge the truth of individual results.
Theoretical Advancement: The paper suggests that resolving replicability issues requires either stronger theoretical predictions (to define the anchor) or the identification of "minimum viable experiments" to eliminate auxiliary noise, rather than just standardizing protocols around arbitrary reference studies.

In conclusion, the paper provides a rigorous statistical proof that the current paradigm of aggregating binary replication verdicts is fundamentally incapable of distinguishing between high and low replicability in the presence of the inevitable heterogeneity found in scientific practice.