💰 quantitative finance

The disruption index is biased by citation inflation

This paper argues that the reported decline in scientific disruptiveness is an artifact of "citation inflation"—driven by increasing reference list lengths and self-citations—rather than a genuine decrease in innovation, rendering the disruption index temporally biased and unsuitable for cross-temporal analysis.

Original authors: Alexander M. Petersen, Felber Arroyave, Fabio Pammolli

Published 2026-07-20

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander M. Petersen, Felber Arroyave, Fabio Pammolli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to measure how "cool" or "revolutionary" a new invention is by looking at how many people talk about it. In the world of science, researchers do something similar using a tool called the Disruption Index. Think of this index as a scorecard for scientific papers and patents. It tries to answer a simple question: Did this new work break the old rules and start a fresh path, or did it just build a little addition onto an existing house? If a paper is truly disruptive, future scientists will cite the new paper but ignore the old ones it stood on. If it's just "consolidating," they'll cite both the new paper and the old ones.

For a long time, scientists have been using this scorecard to track the history of innovation. They noticed something worrying: the scores seemed to be dropping. It looked like modern science was becoming less revolutionary and more like everyone was just treading water, copying each other, and making small tweaks. This sparked a big debate: Are we losing our ability to invent big things? Or is something else going on? To understand this, you need to know that scientific papers have two main parts: the new ideas they present and the "reference list" at the back, which is a list of all the older papers they are standing on.

This paper, written by a team of researchers, argues that the "decline in disruption" isn't because scientists are losing their spark. Instead, they suggest the scorecard itself is broken because of a phenomenon they call "citation inflation." Just like money can lose value when too much of it is printed, the value of a single citation changes when the number of citations explodes. The authors claim that because scientists are writing longer and longer reference lists and citing each other in more complex, self-referencing ways, the math behind the Disruption Index is getting skewed. They aren't saying science is bad; they are saying the ruler we are using to measure it has stretched.

The Great Inflation of References

Imagine you are a student writing a history essay. In the 1960s, you might have had to find 9 books to support your argument. You'd read them, pick the best ones, and list them at the bottom. Now, fast forward to today. You have the internet, and your essay might need 23, or even 50, sources. This isn't just because there are more books; it's because the "supply" of references has inflated.

The authors of this paper point out that the number of references per article has grown dramatically. In the 1960s, the average paper had about 9 references. By the 2000s, that number had jumped to 23. That's a 2.6-fold increase over 50 years. But here is the kicker: the Disruption Index is a math formula that looks at the ratio of "new" citations to "old" citations. When the reference list (the "old" stuff) gets huge, the math gets messy.

Think of the Disruption Index like a game of "Hot Potato." If you are the new paper, and you want to win the "Disruption" prize, you need future players to throw the potato to you and not throw it to the old papers you cited. But if your reference list is a massive bag of 50 old papers, the odds go up that at least one of those old papers is a "superstar" that everyone cites anyway. Even if your new paper is brilliant, the sheer number of old papers you cited means that future scientists are almost guaranteed to cite at least one of them. This makes your new paper look less "disruptive" simply because the denominator of the math equation got too big.

The "Self-Hug" Problem

There is a second part to this inflation story, and it involves how scientists cite each other. The authors call this triadic closure, which is a fancy way of saying "closing the triangle." Imagine three friends: Alice, Bob, and Charlie. If Alice cites Bob, and Bob cites Charlie, there is a high chance Alice will also cite Charlie. This creates a tight-knit triangle of citations.

In the past, these triangles happened naturally. But today, the authors argue, these triangles are forming more often because of self-citations and strategic citing. Scientists might cite their own previous work, or the work of their close colleagues, to boost their own numbers or help their journal. This creates a web of citations that are tightly connected. The Disruption Index counts these tight triangles as "consolidation" (not disruption). So, as scientists get better at building these citation webs, the index thinks they are being less disruptive, even if they are actually doing great work. It's like a judge scoring a gymnast lower just because they are wearing a very shiny, complicated costume that makes them look less agile, even if their flips are perfect.

The Simulation: Turning Off the Inflation

To prove their point, the researchers didn't just look at real data; they built a computer simulation. They created a fake world of scientific papers that grew over time, just like the real world. They programmed this fake world to have two different settings:

The "No Inflation" World: Here, the number of references per paper stayed the same, and scientists didn't change how they cited each other.
The "Inflation" World: Here, they programmed the papers to have longer reference lists and more "triadic closure" (more self-citing and group-citing), just like in the real world.

When they ran the simulation with the "No Inflation" settings, the Disruption Index stayed steady. It didn't drop. But when they turned on the "Inflation" settings—making reference lists longer and citation patterns more clustered—the Disruption Index started to plummet, exactly like it does in real life.

This suggests that the drop in disruption scores isn't because scientists are becoming less creative. It's because the metric is sensitive to the length of the reference list. The authors found that as the number of references ( $r$ ) goes up, the "extraneous citation" rate ( $R_k$ ) goes up, which forces the Disruption Index ($CD$) to get closer to zero. In their simulations, when they capped the reference list length at a fixed number (like 25 references) after a certain point, the Disruption Index stopped dropping and actually started to rise again. This confirms that the "decline" was an artifact of the growing reference lists, not a decline in innovation.

What This Means for the Future

The authors are careful to say they haven't "solved" the problem of measuring innovation, but they have identified a major flaw in the current ruler. They argue that the Disruption Index, as it is currently used, is biased and unsuitable for comparing different time periods. You can't compare a paper from 1960 with a paper from 2020 using this index because the "currency" of citations has changed.

They also point out that this inflation affects other measurements. For example, some studies claimed that larger research teams produce less disruptive work. But the authors found that when they accounted for the fact that larger teams also tend to have longer reference lists, the relationship changed. It turns out that larger teams might actually be more disruptive, but the citation inflation was hiding that fact.

So, what's the takeaway? The scientific world isn't necessarily losing its edge. Instead, the way we measure that edge needs an upgrade. The authors suggest that journals might need to put a "cap" on how many references a paper can have, or develop new ways to measure disruption that don't get confused by the sheer volume of citations. Until then, we should be very careful about using the Disruption Index to say that science is becoming less revolutionary. It might just be that the scoreboard is broken, not the players.

Technical Summary: The Disruption Index is Biased by Citation Inflation

Problem Statement
Recent analyses of scientific publication and patent citation networks (e.g., Park et al., Nature, 2023) have suggested that intellectual contributions are becoming less "disruptive" over time. The disruption index ($CD$) is a bibliometric measure designed to quantify the degree to which a publication supersedes the sources it cites. However, this paper argues that the reported temporal decline in disruptiveness is not a reflection of a changing innovation system, but rather an artifact of systematic shifts in citation network structure unrelated to scientific progress. Specifically, the authors identify "citation inflation" (CI)—a systematic, time-dependent increase in the number of links within citation networks—as the primary driver rendering the $CD$ index unsuitable for cross-temporal analysis.

Methodology
To demonstrate that the decline in $CD$ is a measurement artifact rather than a real phenomenon, the authors employ three complementary lines of critique:

Deductive Analysis: The authors re-examine the mathematical definition of the disruption index. $CD_p$ is defined as the ratio $(N_i - N_j) / (N_i + N_j + N_k)$ , where:
- $N_i$ : Citations to the focal paper $p$ that do not cite its references.
- $N_j$ : Citations to both $p$ and its references (triadic closure/consolidation).
- $N_k$ : Citations to the references of $p$ that do not cite $p$ (extraneous citations).
  The authors argue that as the length of reference lists ( $r_p$ ) increases over time, the likelihood of citing highly-cited papers within those lists increases. This causes $N_k$ (the denominator term) to grow unbounded, while the numerator ( $N_i - N_j$ ) remains bounded. Consequently, $CD_p$ is mathematically predisposed to converge to zero as reference lists expand.
Empirical Analysis: Using the Microsoft Academic Graph (MAG) dataset (approx. 29.5 million articles from 1945–2012), the authors analyze the correlation between the disruption index and citation inflation drivers. They employ linear regression models with yearly fixed effects to control for secular growth, analyzing the relationship between $CD$, the number of references ( $r_p$ ), and team size ( $k_p$ ).
Computational Modeling: The authors utilize a generative network model (growth and redirection model) to construct synthetic citation networks. This approach allows for the isolation of specific variables. They simulate networks under six scenarios varying:
- The presence or absence of citation inflation (growth in reference list length, $g_r$ ).
- The rate of triadic closure (redirection mechanism, $\beta$ ).
- Citation window sizes ($CW$).
  By controlling these parameters, they test whether the decline in $CD$ persists in the absence of citation inflation.

Key Results

Mathematical Bias: The analysis confirms that $CD$ is biased because the term $R_k = N_k / (N_i + N_j)$ is an extensive quantity that grows with the size of reference lists. As reference lists lengthen, $N_k$ increases disproportionately, driving $CD$ toward zero regardless of the actual disruptive nature of the publication.
Empirical Correlation: In the MAG dataset, the average number of references per article ( $r(t)$ ) increased from ~9 in the 1960s to ~23 in the 2000s (a 2.6-fold increase). This growth correlates strongly with the decline in $CD$. Regression analysis shows a significant negative relationship between $CD$ and the logarithm of the number of references, while the relationship with team size ( $k_p$ ) is positive when controlling for reference length, contradicting previous findings that suggested a negative correlation.
Simulation Evidence:
- In synthetic networks where reference list lengths are constant ( $g_r = 0$ ), $CD$ does not exhibit a systematic downward trend, even when publication volume grows.
- When reference list lengths grow exponentially ( $g_r > 0$ ), $CD$ declines rapidly, mirroring empirical observations.
- Increasing the rate of triadic closure ( $\beta$ ) also affects $CD$, but citation inflation ( $g_r$ ) is the dominant factor driving the convergence to zero.
- Extending the citation window ($CW$) does not resolve the bias; $CD$ still converges to zero as $r(t)$ increases.
Intervention Simulation: A hypothetical policy intervention capping reference list lengths at a fixed value (25) after a specific time point ( $T^*$ ) results in an immediate reversal of the $CD$ trend from decreasing to increasing. This confirms that the decline is driven by the expansion of reference lists rather than intrinsic changes in scientific behavior.

Significance and Claims
The paper claims that the observed decrease in scientific disruptiveness is a "fundamental measurement problem" caused by citation inflation, not a genuine shift in the nature of innovation. The authors assert that:

Temporal Bias: The disruption index is temporally biased and unsuitable for cross-temporal comparisons. Comparing $CD$ values across different eras is akin to comparing prices without adjusting for monetary inflation.
Confounding Variables: The index is confounded by shifts in scholarly citation practices, including the increasing length of reference lists and the rising rate of self-citations and triadic closure.
Policy Implications: The authors suggest that to mitigate this bias, journals could consider capping reference list lengths (e.g., based on article type or page count). Such a policy could stabilize the $CD$ metric and reduce the systematic bias introduced by citation inflation.
Resource Availability: To facilitate further research into bias-free metrics, the authors have made an ensemble of synthetic citation networks available, which can be used to test alternative citation-based indices for systematic bias.

In conclusion, the authors argue that researchers and policymakers must distinguish between "nominal" counts and "real" values in scientific evaluation. Without accounting for the time-dependent growth of citation networks, metrics like $CD$ will continue to produce misleading interpretations of scientific progress.

The Great Inflation of References

The "Self-Hug" Problem

The Simulation: Turning Off the Inflation

What This Means for the Future

Technical Summary: The Disruption Index is Biased by Citation Inflation

More like this