💬 NLP

The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers

This study reveals that AI-assisted survey papers exhibit a persistent 17% rate of unresolvable "phantom" citations, indicating that large language models systematically degrade the integrity of scientific citation chains by hallucinating metadata despite often retrieving correct titles.

Original authors: H. Kemal İlter

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: H. Kemal İlter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific research as a massive, interconnected library. For centuries, if a scholar wanted to prove a point, they had to walk to the shelves, find the exact book, and point to the specific page. This "chain of custody" ensured that anyone else could go find that same book and verify the truth.

Now, imagine a fleet of incredibly fast, super-smart robots (AI) has been hired to write these research papers. They are amazing at summarizing ideas and finding the names of famous books. But, it turns out, they are terrible at finding the exact location of those books on the shelves.

This paper, titled "The 17% Gap," is a forensic audit of what happens when these robots write survey papers in Artificial Intelligence. Here is the breakdown in simple terms:

The Problem: The "Lazy Research Assistant"

The authors argue that AI tools act like lazy research assistants.

What they do right: They know the title of a famous paper. They know the author's name. They keep the story of the research coherent.
What they do wrong: When asked for the specific "address" (like a DOI number, volume, or page count) to find that paper, they start guessing. They make up numbers that look real but lead to nowhere.

It's like a tour guide who knows the name of a famous museum perfectly but gives you the wrong street address. You drive there, but you just end up in an empty field.

The Investigation: A Digital Detective Story

The researchers picked 50 recent AI survey papers (published between late 2024 and early 2026) and checked every single citation inside them. They looked at 5,514 references in total.

They used a "hybrid verification pipeline," which is just a fancy way of saying they used a multi-step checklist:

Direct Check: Did the link work immediately?
Recovery Attempt: If the link was broken, could they fix the typo and find the real paper?
The "Phantom" Check: If they couldn't find the paper at all, even after trying to fix the text, they labeled it a "Phantom."

The Results: The 17% Gap

The audit revealed a startling number: 17% of all citations were "Phantoms."

This means that in these papers, nearly 1 out of every 6 references leads to a dead end. You can't find the source material.

The researchers broke these "Phantoms" down into three types of failures:

The "Ghost" (5.1%): Pure hallucinations. The AI made up a paper that doesn't exist at all. It's a complete lie.
The "Broken Link" (16.4%): The paper exists, but the AI got the address wrong (e.g., a fake DOI number). It's like having the right book title but the wrong ISBN.
The "Syntax Error" (78.5%): This is the biggest group. The paper is real, and the AI knew the title, but the text got garbled during the copying process (like a PDF turning into a jumbled mess of letters). The AI couldn't read the address correctly, so it failed to find the paper.

Crucially, the study found that the vast majority of "Phantoms" aren't because the AI is lying; it's because the AI is sloppy or the text extraction is messy.

The Trend: A Stable Rot

The researchers checked if this problem was getting better or worse over time. They found it wasn't changing. The error rate has stabilized at this 17% level.

They call this "Epistemic Decay." It's like link rot on a massive scale. The scientific graph looks strong on the outside, but underneath, the connections are snapping.

The Warning: Muller's Ratchet

The paper uses a biological concept called Muller's Ratchet to explain why this is dangerous.

The Analogy: Imagine a ratchet wrench that only turns one way. Once a mistake is made, it can't be undone.
In Science: If an AI writes a paper with a fake citation, and a human reads that paper and cites the fake one in their new paper, the error spreads. It becomes part of the "official" record.
The Math: If 17% of citations are broken, the "integrity" of the literature drops by half in just 3.7 generations of papers. Without a way to fix it, the library of knowledge slowly turns into a collection of dead ends.

The Conclusion

The paper concludes that we have a structural problem. We have automated the writing of science but haven't automated the verification of it.

The authors suggest three fixes:

Fix the scanners: Improve the tools that read PDFs so they don't garble the text (fixing the 78.5% "Syntax Errors").
Check the addresses: Make it mandatory for journals to automatically check if every link works before a paper is published.
Train the AI: Teach the AI models to only pull citations from verified databases, not to guess.

In short: AI is great at summarizing the story of science, but it's currently terrible at providing the footnotes. If we don't fix this, we risk building a future of science on a foundation of broken links.

Technical Summary: The 17% Gap: Quantifying Epistemic Decay in AI-Assisted Survey Papers

Problem Statement
The paper addresses a critical, unquantified risk in the scientific literature of Artificial Intelligence: the systematic degradation of citation chains due to the integration of Large Language Models (LLMs). While "hallucinated papers" (non-existent references) are a known artifact, the authors argue that a more insidious structural error has emerged. They hypothesize that AI tools function as "lazy research assistants" that successfully retrieve semantically coherent titles but hallucinate the bureaucratic metadata (DOIs, volume numbers, page ranges) required to locate them. This creates a "link rot" at scale, where the citation graph appears robust but is functionally severed, threatening the chain of custody required for reproducible science. The authors posit that this is not a transient noise issue but a structural equilibrium of decay, potentially leading to an irreversible accumulation of errors akin to Muller's Ratchet in evolutionary biology.

Methodology
The study employs a forensic audit framework to distinguish between benign formatting errors ("Sloppiness") and verifiable non-existence ("Phantoms").

Corpus: The authors analyzed 50 recent survey papers in Artificial Intelligence (published between September 2024 and January 2026) containing a total of 5,514 unique citations.
Verification Pipeline: A hybrid pipeline was deployed involving:
1. Identifier Extraction: Regex patterns to extract DOIs and arXiv IDs.
2. Direct Resolution: HTTP status code verification for DOIs and API checks for arXiv IDs.
3. Entropy Filtering: A "Space Ratio Entropy Filter" ( $\rho \ge 0.10$ ) was applied to exclude citations corrupted by PDF extraction artifacts (e.g., missing whitespace) from being classified as hallucinations.
4. Fuzzy Matching: For citations without valid identifiers, title-based searches were conducted via Crossref and Semantic Scholar APIs using Levenshtein similarity ratios.
Classification Logic: Citations were categorized into four states:
- Valid: Exact match or successful resolution ( $s^* \ge 85\%$ ).
- Sloppy: Recoverable via fuzzy matching ( $50\% \le s^* < 85\%$ ).
- Phantom: Irrecoverable ( $s^* < 50\%$ ).
- Unknown: Failed entropy filter (parsing artifacts).
Phantom Taxonomy: Phantoms were further subdivided into three diagnostic modes:
- Ghost: Pure hallucinations ( $s^* < 25\%$ ).
- Broken Link: Valid titles with hallucinated identifiers leading to HTTP 404s ( $25\% \le s^* < 50\%$ ).
- Syntax Error: Real papers where PDF extraction corrupted the text, preventing a match ( $25\% \le s^* < 50\%$ ).

Key Results

Phantom Rate: The study detected a persistent 17.0% Phantom Rate (939 out of 5,514 citations). The 95% confidence interval is [16.0%, 18.0%].
Integrity Breakdown: Only 41.0% of citations were immediately verifiable. An additional 9.7% were recovered as "Sloppy" (valid papers with broken metadata). The "Unknown" category accounted for 32.3%.
Diagnostic Distribution: Contrary to the assumption that most errors are pure fabrications, the breakdown reveals:
- 78.5% are "Syntax Errors" (parsing artifacts corrupting real papers).
- 16.4% are "Broken Links" (hallucinated identifiers attached to valid titles).
- 5.1% are "Ghosts" (pure hallucinations with no discernible match).
Temporal Stability: Longitudinal analysis shows a flat trend ( $\hat{\beta}_1 = +0.07$ pp/month, $p=0.72$ ), suggesting the 17% rate is an equilibrium state rather than a transient phase.
Decay Modeling: Using a discrete-time Markov model, the authors calculate that with a 17% phantom rate, the half-life of citation integrity is approximately 3.7 generations, leading to exponential decay of verifiable knowledge in the absence of repair mechanisms.

Significance and Claims
The paper claims to provide the first quantification of "epistemic decay" in AI-assisted literature, moving beyond anecdotal evidence of hallucinations to a structural analysis of the citation graph.

Structural Fragility: The findings suggest that the scientific record is becoming decoupled from evidentiary truth. The AI tools prioritize semantic plausibility over metadata accuracy, creating a "bullshit" dynamic where citations look correct but lead nowhere.
Lower Bound Reality: The authors note that their 17% figure is likely a lower bound. If "Unknown" citations (32.3%) are treated with a conservative validity prior, the adjusted phantom rate could reach 26.7%.
Endemic Feature: The stability of the error rate over 16 months indicates that high-entropy citation practices have stabilized as an endemic feature of the field, driven by the speed of AI-assisted writing outpacing human verification.
Call for Intervention: The paper argues for a shift in verification standards, proposing "Algorithmic Proof of Existence" (automated DOI checks) at the manuscript submission stage to prevent the propagation of broken links. It concludes that until the cost of verifying citations is addressed, the scientific record remains vulnerable to a "slow, silent, and plausible decay."