Inflationary bibliometric values revisited: Indicator validity under falling production costs

In the world of academic research, there is a long-standing habit of counting things to measure quality. For decades, scientists and university administrators have looked at how many papers a researcher writes, how many times those papers are cited by others, and how many authors worked on a single project. These numbers act as a scorecard, helping to decide who gets a job, a grant, or a promotion. However, this system has faced a major challenge before. Twenty years ago, researchers noticed that the sheer volume of papers and citations was growing much faster than the number of scientists actually doing the work. This "inflation" meant that simply counting papers was no longer a fair way to judge quality, because the numbers were being boosted by more people working together rather than by better science. The solution at the time was to stop counting raw numbers and start comparing researchers to their peers, asking how a scientist performed relative to the average in their specific field.

Today, a new force is changing the landscape of scientific writing. The rise of powerful artificial intelligence tools has made it significantly cheaper and faster to draft and revise scientific text. This shift raises a critical question: if it becomes easier to produce a paper, does that change the fundamental nature of the papers themselves, or does it just mean there are more of them? If the cost of producing a paper drops, does the distribution of quality change, or does it simply expand? This is the core puzzle addressed by Ahmed Hamid Mahmoud in a recent study. He investigates whether the old methods of comparing researchers are still valid when the cost of creating a scientific paper is no longer constant.

Mahmoud's work begins by separating the effects of this new, cheaper production environment into two distinct possibilities. The first possibility is what he calls a "scale" change. Imagine a factory that suddenly gets a new machine allowing it to produce twice as many widgets. If the quality of every widget remains exactly the same, but there are just more of them, the overall shape of the quality distribution stays the same. In the world of science, this would mean that artificial intelligence allows researchers to publish more papers, but the ratio of excellent papers to average papers remains unchanged. The second possibility is a "shape" change. This would occur if the new tools changed the nature of the work itself, perhaps making it easier to produce low-quality papers or altering the mix of high and low-quality work in a way that distorts the distribution. If the shape changes, the old rules for judging quality might no longer apply.

The study uses a mathematical model of how scientific citations are distributed to test these scenarios. The researchers found that if the change is purely a "scale" effect—meaning more papers are produced but the underlying pattern of quality remains the same—then the relative indicators used today are still safe. Comparing a researcher to their peers, or looking at the ratio of their citations to the field average, continues to provide a valid measure of performance. These methods are robust against a simple increase in volume. However, the situation changes drastically if different researchers have unequal access to these cheap production tools. If some scientists can use artificial intelligence to publish many more papers while others cannot, simply counting the total number of papers or total citations becomes misleading. In this scenario, the counts no longer reflect the quality of the work; they merely reflect who had access to the cheaper tools. The ranking would show who had the advantage, not who did the better science.

The paper also examines what happens if the "shape" of the quality distribution actually changes. If the mix of high and low-quality work shifts, the standard benchmarks used to judge researchers become broken. For example, a threshold that once separated "good" papers from "average" ones might suddenly classify a large number of papers incorrectly if the underlying distribution has warped. The study shows that if this shape deformation occurs, the old, fixed standards will fail to identify the true quality of researchers. However, the paper offers a solution. It suggests that the field can detect these changes by looking at the specific patterns of how papers are distributed within a group. By estimating the current "shape" of the distribution alongside the standard rankings, evaluators can adjust their judgments. The recommendation is to report a researcher's position within their current group, while also explicitly stating the state of the environment they are being judged in. This dual reporting ensures that a researcher is not penalized or unfairly boosted by a shift in the system itself.

The author is careful to note that while the evidence for a massive increase in the volume of papers and the use of artificial intelligence is strong, it has not yet been proven that the fundamental shape of the quality distribution has changed. The study does not claim that the old methods are useless, nor does it say that artificial intelligence has ruined science. Instead, it provides a clear framework for understanding what is happening. It confirms that the relative comparisons developed to fix the inflation of the past are still valid as long as the production environment changes uniformly. It warns that if access to these new tools is uneven, raw counts will lie. And it provides a method to detect if the very nature of the work is changing, allowing the scientific community to adapt its evaluation rules before they become obsolete. The ultimate conclusion is that the tools for measuring science are still sound, but they must be used with a clear understanding of the environment in which they are applied. The question for the future is not whether to abandon these tools, but whether to monitor the system closely enough to know if the ground beneath them is shifting.