What Citations Get Wrong: A Full-Corpus Audit of Reference Existence and Claim Support in a Major NLP Conference
This paper audits the ACL 2026 proceedings to find that while nearly all cited references exist, a single-run automated audit of claim support failed to replicate due to an unlogged model configuration, demonstrating that rigorous citation verification requires strict operational logging to ensure reproducibility.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast library of modern science, a citation is more than just a name in a list; it is a promise. When an author writes a sentence and points to another paper, they are making two distinct commitments. First, they promise that the paper they are pointing to actually exists and can be found. Second, and far more difficult to verify, they promise that the paper they are pointing to actually says what the author claims it says. For decades, the scientific community has worried primarily about the first promise, especially with the rise of artificial intelligence. There is a growing fear that computer programs are inventing fake papers, filling reference lists with titles and authors that never existed, effectively polluting the scholarly record with fiction. This anxiety has been loud, but the evidence to support it has been thin. Meanwhile, the second promise—the accuracy of the claim itself—has received far less attention, even though it is the older, more fundamental test of whether a scientist is telling the truth about their sources.
A researcher named Tao An decided to test both promises at once, not on a small sample, but on an entire year's worth of papers from a major computer science conference. The team audited every single reference in the 2026 proceedings of the Association for Computational Linguistics, a collection containing nearly 4,500 papers and over 200,000 references, to check if they pointed to real documents. For a smaller, sampled subset of these citations, they also checked whether the document actually supported the sentence that cited it. The goal was to see if the fear of AI-generated fake references was grounded in reality, and to measure how often an author misrepresents what their sources actually say.
The investigation began with the first promise: existence. The researchers built a system to check if every single reference pointed to a real, retrievable work. They found that the vast majority of references, about 91 percent, successfully resolved to real literature. The small number of references that did not match were mostly not fake papers at all; they were things like blog posts, software documentation, or parsing errors where the computer had misread the text. When the team manually reviewed the tiny fraction of references that looked suspicious, they found only two instances out of nearly 210,000 where a paper was confirmed to be completely nonexistent. This result suggests that on this major platform, the fear of AI filling the literature with made-up papers is largely unfounded. The references are almost always real.
However, the story changed completely when the researchers moved to the second promise: whether the cited paper actually supported the claim. This is a much harder task, requiring the system to read the cited paper and compare it to the sentence that referenced it. The team used a two-stage process involving computer models to flag citations that seemed to misrepresent their sources. In their first attempt, the system flagged a very small number of errors, suggesting that authors were mostly accurate. But when the researchers ran the exact same process again, using the same computer code and the same papers, the results were wildly different. The second and third runs found a number of errors roughly six times higher than the first run.
This inconsistency became the most important finding of the study. The researchers traced the cause to a hidden variable: the specific computer model used to make the initial judgment had not been recorded. In the first run, the system had silently defaulted to a weaker model that missed many errors. In the subsequent runs, a different, stronger model was used, which caught many more problems. Because the first run used a model that was never logged, the researchers could not say for certain what that first result actually measured. The lesson was stark: a single automated audit, no matter how carefully it is checked by hand, cannot be trusted if the process cannot be repeated with the exact same tools. The measurement itself was unstable.
When the researchers looked at the data that did replicate, they found that the errors were real, but they were not the kind of machine-generated nonsense people feared. The confirmed mistakes were subtle human errors. An author had taken a real paper and claimed it said the opposite of what it actually argued, or they had cited a broad finding when the paper only supported a narrow one. These were not hallucinations created by an artificial intelligence; they were the kinds of mistakes a tired researcher might make when rushing to finish a manuscript. The errors were invisible to any tool that only checks if a reference exists, because the papers were real, but the connection between the claim and the source was broken.
The study also revealed that the peer review process, where other scientists check papers before they are published, did not seem to catch these support errors. When the team compared the published conference papers to a set of unreviewed drafts, the rate of citation errors was statistically the same. This suggests that the current system of peer review is not designed to check every citation against its source, a task that would take far too much time for volunteer reviewers. Instead, the responsibility for ensuring that a citation actually supports the claim likely falls on the author themselves before they submit their work.
Ultimately, the audit provided a clear picture of where the problems lie. The fear that the scientific record is being flooded with fake, non-existent papers is not supported by the data; the references are almost always real. The real problem is the accuracy of the claims attached to those references. These errors are subtle, human, and difficult to detect automatically because the tools used to find them are not yet stable enough to give a single, reliable number. The study concludes that before we can trust machines to police the integrity of science, we must first ensure that the machines themselves are consistent and that their own errors are fully understood. The defects that remain are not a sign of a system collapsing under the weight of artificial intelligence, but rather a reminder of the enduring complexity of human scholarship.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.