The Invisible Infrastructure of Biomedical Research: An Exploratory Analysis of Software Mentions and Sentence-Level Citation Marking
This exploratory analysis of nearly 800,000 software mentions in biomedical papers reveals that sentence-level citation marking is generally low (10.7%) and varies significantly by tool type, with statistical packages being cited least frequently, highlighting the need for targeted policy interventions and improved validation methods.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Modern science relies heavily on invisible tools. When a biologist sequences a strand of DNA or a statistician analyzes a massive dataset, they are not just using a pen and paper; they are running complex computer programs. These programs are the engines of discovery, shaping what researchers can conclude and how fast they can work. Yet, the system scientists use to give credit for their work—the academic citation—was built for books and articles, not for software. When a researcher mentions a program in their paper, it often appears as a simple name in a sentence, like a tool sitting on a workbench. Because these names are rarely linked to a formal reference, the people who built the software often receive no credit for their labor. This lack of recognition makes it hard to justify the time and money needed to keep these tools running, and it leaves a gap in the record that makes it difficult for others to repeat the experiments.
A new study by Md Motasim Billa at the University of Arkansas at Little Rock investigates exactly how often this happens and whether the problem is the same for every type of software. The researcher looked at nearly 800,000 mentions of software names inside nearly 140,000 biomedical research papers published between 1980 and 2022. They did not just count how many times software was mentioned; they looked closely at the sentences to see if the mention was accompanied by a formal citation marker, such as a number in brackets or a superscript that points to a reference list. This distinction is crucial because a name floating in a sentence is visible to a human reader but invisible to the automated systems that track scientific impact.
The researcher found that formal citation is surprisingly rare. Across the entire collection of papers, only about 11 percent of software mentions carried a citation marker in the same sentence. The vast majority, nearly 70 percent, were just bare names written into the text. However, the study revealed that this low rate is not uniform across all types of tools. The likelihood of a software tool being cited depends heavily on what kind of tool it is. Machine-learning frameworks, which are used to train artificial intelligence models, were cited in about 18 percent of their mentions. Programming languages were cited in roughly 16 percent of cases. In stark contrast, general-purpose statistical packages, the workhorses of data analysis, were cited in only 2 percent of mentions. Reference managers, tools used to organize bibliographies, were cited in just 1 percent of cases.
This pattern suggests that the way scientists talk about software is not a single, broken habit, but a set of different conventions depending on the tool. Statistical software is often treated as standard equipment, like a microscope or a test tube, rather than as a unique work of authorship that deserves a specific credit. When a scientist says they used a statistical package, they are often following a methodological convention that does not require a citation. Conversely, machine-learning frameworks often have associated research papers that are considered the primary way to cite the tool, leading to higher citation rates. The study also found that the way a tool is named in the text changes the odds of it being cited. If a sentence includes a web link to the software, it is more likely to have a formal citation. If the sentence names the company or developer who made the software, it is much less likely to have a citation, suggesting that naming the creator often replaces the need for a formal reference in the eyes of the author.
The researcher also tested how well current methods can tell the difference between a software name and a regular word that happens to look like one. Many software tools have names that are common English words, such as "star," "cluster," or "string." The study found that standard lists of known software tools did not contain any of these ambiguous names. When the researcher tried to use simple rules based on the surrounding words to figure out if a word like "cluster" referred to software or just a group of things, the rules worked moderately well but failed often when the context was technical. This highlights a significant challenge: the tools used to automatically track software mentions are not yet good enough to handle the most common, ambiguous names without human help or more advanced technology.
The study concludes that while the problem of under-citing software is real, it is not a simple case of scientists forgetting to give credit. Instead, it is a complex issue where different types of tools are treated with different rules. The gap is widest for the statistical packages that drive much of modern research. Because the data used in this study was not a perfect random sample and the methods for identifying software were not fully validated against human experts, the findings are best seen as a map of where the problem is most severe, rather than a final census. The results suggest that a single rule telling all scientists to "cite your software" may not work. Instead, policies might need to be tailored to specific types of tools, particularly those that are currently treated as invisible equipment, to ensure the builders of the digital infrastructure of science finally get the credit they deserve.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.