A Multi-Surface Consistency Audit of Software Citation Metadata

In the world of modern science, software has become as essential as a microscope or a telescope. It is the tool researchers use to crunch numbers, simulate climate change, and model the behavior of atoms. Because this software is a product of human labor and intellectual effort, the scientific community has agreed that it deserves to be cited, just like a research paper. When a scientist uses a specific program to make a discovery, they should give credit to the people who wrote it, the version they used, and the place where they can find it. To make this possible, a system has been built where software projects can leave behind digital "name tags." These tags are machine-readable files that contain the project's title, the names of its authors, the version number, and a unique identifier that points to where the software lives online. The idea is that if a researcher, a librarian, or a computer program looks at any of these tags, they will all tell the same story about the same piece of software.

However, a recent study asked a simple but unsettling question: do these name tags actually agree with each other? The researchers, led by Pengyin Shan at the University of Illinois, decided to audit the digital self-descriptions of 117 open-source research software projects. They looked at seven different places where a project might describe itself: a specific file meant for citations, a data file for archives, a record in a public registry where software is downloaded, and the text on the project's main web page. They treated the software as the single object and these various files as different "surfaces" or faces of that object. If the system works perfectly, every surface should say the same thing. If they disagree, it creates confusion about who did the work, which version was used, and where to find the code. The study found that the system is far from perfect. In fact, the majority of the projects examined were telling different stories to different parts of the internet.

The researchers built a careful process to check these projects. They gathered a group of 117 software tools, including high-performance computing tools used for supercomputing and quantum computing, as well as a smaller group of tools that had been accepted by community journals. For each project, they downloaded snapshots of the seven different surfaces on a single day. They then used a computer program to normalize the information, translating the different file formats into a common list of six key facts: the title, the author, the version, the year, the license, and the unique identifier. They compared every pair of surfaces that existed for a project to see if the facts matched. To ensure their method was accurate, they manually checked a large sample of the computer's judgments, confirming that their tool was correct nearly 99 percent of the time.

The results revealed a significant lack of consistency. Out of the 117 projects, only 62 had at least two surfaces that could be compared. Among those 62 projects, 52 of them, or roughly 84 percent, contained at least one major conflict between their surfaces. This means that for most of these projects, a person reading the citation file would get different information than someone reading the archive record or the download page. The disagreements were most common in the lists of authors and the titles of the software. While the year and the license were usually consistent, the core details that identify the work often did not match. The study found that the problem was not limited to one type of software; it appeared in high-performance computing projects, in tools reviewed by the Journal of Open Source Software, and in packages accepted by the pyOpenSci community.

One of the most common reasons for these conflicts was a specific mix-up between the software and the research paper that described it. In many cases, the files meant to describe the software were actually describing the academic paper about the software instead. For example, a file might list the title of a conference presentation or the author of a journal article, while another file listed the name of the code itself. This happened in half of the verified conflicts. The researchers noted that this is not necessarily a mistake in how the files were written, but rather a reflection of a deeper confusion about what the "citable object" actually is. Is it the code, or is it the paper about the code? When the files try to point to the paper, they often stop pointing to the software, causing the metadata to drift apart.

Another source of disagreement came from the public registries where software is downloaded, such as PyPI for Python or npm for JavaScript. These registries often use short, technical names for packages, like "mpi4py," while the software's own files use a descriptive title like "MPI for Python." When a computer tries to match these two surfaces, it sees a title mismatch. The study found that a large portion of the title and author disagreements were driven by these registry records. Furthermore, the researchers found that the most up-to-date information was often missing from the places where automated tools look for it. The files that packaging tools read, such as the registry records, were the least likely to contain persistent identifiers like ORCIDs for authors, which are crucial for tracking who did the work.

The study also looked at a specific feature called a "preferred citation," which allows a software project to tell users to cite a different object, usually a paper, instead of the software itself. The researchers found that in 28 out of 32 projects that used this feature, following the instruction led to a record that disagreed with the software's own metadata. This means that even when a project tries to guide users to a specific citation, the guidance often points to a record that conflicts with the other information the project has published about itself. This creates a situation where a user following the rules might end up citing a paper that does not match the version of the software they actually used.

The researchers concluded that the current infrastructure for software citation is incomplete. While the tools to declare metadata exist and are widely used, there is no system in place to check if the different declarations agree with each other. The study showed that for the majority of projects that can be checked, the software disagrees with itself on at least one critical piece of information. This fragmentation means that credit is often split, provenance is unclear, and automated systems cannot reliably track the use of research software. The researchers released their data, their code, and their list of projects so that others can repeat the audit, check if things have improved over time, or use the data to build tools that can automatically fix these inconsistencies. The work highlights that before we can fully trust the digital credit system for software, we must first ensure that the software tells a single, consistent story across all the places it speaks.