arXiv💬 cs.CL

How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations

This study presents a large-scale audit of 10 commercial LLMs revealing significant variation in citation hallucination rates across models and domains, while demonstrating that prompt-induced fabrication can be effectively mitigated through multi-model consensus, within-prompt repetition, and a lightweight bibliographic classifier that detects phantom citations without external database queries.

MZ Naser

Published 2026-03-05
📖 5 min read🧠 Deep dive
View on arXiv ↗PDF ↗

Imagine you ask a very smart, well-read robot to write a research paper for you. You tell it, "Please list the most important books and articles that prove your point." The robot, eager to please, types out a long list of citations with authors, titles, and years. It looks perfect. It looks professional.

But here's the catch: Many of those books and articles don't actually exist.

This paper is a massive investigation into exactly how often this happens, why it happens, and how we can catch the robot lying before it ruins your homework (or your PhD).

Here is the breakdown of the study, translated into everyday language with some helpful analogies.

1. The Great "Fake Book" Audit

The researchers didn't just guess; they put 10 different popular AI models (like the brains behind ChatGPT, Claude, and others) through a grueling test.

  • The Test: They asked these AIs to write about four different topics (like building bridges, climate change, medicine, and computer science) and requested references.
  • The Scale: They generated nearly 70,000 citations.
  • The Check: They ran every single one of those citations through a "truth detector" (checking against three massive databases of real academic papers).

The Result: The AIs were lying a lot.

  • Some models were "honest" about 88% of the time.
  • Others were lying more than 50% of the time.
  • The Big Discovery: When the researchers asked the AIs questions without asking for references, zero fake citations appeared. This proves the AIs aren't naturally prone to lying; they only start fabricating facts when you specifically ask them to "cite your sources." It's like a student who knows the material but starts making up sources only when the teacher says, "Show me your work."

2. The "Time Travel" Trap

The researchers noticed something weird about when the AIs lied.

  • The "Old Classics" Test: When asked for "seminal" (famous, old) papers, the AIs did okay.
  • The "New News" Test: When asked for "recent" papers, the lying skyrocketed.

The Analogy: Imagine a librarian who has read every book in the library up to 2023. If you ask for a book from 1990, they can find it easily. But if you ask for a book published last week, they haven't read it yet. To avoid admitting they don't know, they invent a title that sounds like a real book from last week. The AIs are doing the same thing; they are guessing at recent events because their "memory" (training data) is outdated.

3. The "Crowd Wisdom" Solution

So, how do we stop the lying? The researchers found two clever, low-tech tricks that work like a "voting system."

  • Trick #1: The "Three-Headed Monster" Rule.
    If you ask three different AIs the same question and they all give you the exact same citation, it is almost certainly real (95% chance).
    • Why? It's hard for three different liars to accidentally invent the exact same fake book title. But it's easy for three honest librarians to find the same real book.
  • Trick #2: The "Ask Again" Rule.
    If you ask the same AI the same question three times, and it gives you the same citation twice, it's probably real.
    • Why? Fake citations are random guesses. Real citations are memories. If the AI remembers the same thing twice, it's likely a real memory.

4. The "Fake ID" Detector (The AI Classifier)

The researchers also built a special tool—a "lie detector" that doesn't need to check the internet. It just looks at the shape of the citation string.

The Analogy: Think of a fake ID. Even if the photo looks good, the font might be slightly wrong, or the address might be too short.

  • Real Citations: Usually have longer author names, more authors listed, and older publication years.
  • Fake Citations: Often have very short author names, fewer authors, and strangely recent years (because the AI is trying to sound "up to date").

They trained a computer program to spot these "suspicious shapes." It can scan a list of citations and flag the likely fakes in a split second, saving you from having to check every single one manually.

5. Bigger Isn't Always Better

You might think, "If I use the newest, most expensive AI model, it will lie less."

  • The Reality: Not necessarily.
  • One company (OpenAI) made a new model that lied much less than its older version.
  • Another company (Anthropic) made a new model that lied more than its older version.

It turns out that just making a model "smarter" or "bigger" doesn't automatically fix its ability to tell the truth about references. It depends on how the company trained it and what data they fed it.

The Bottom Line

AI is a powerful tool for writing, but it is a terrible librarian when it comes to making up references.

  1. Don't trust it blindly: If an AI gives you a citation, assume it might be fake until proven otherwise.
  2. Use the "Voting" method: If multiple AIs agree on a source, it's likely real.
  3. Check the "ID": If a citation looks too simple or too recent, be suspicious.

The study ends with a clear message: AI can help you write, but you must be the one to verify the facts. The robot is the writer; you are the editor.

More like this

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

The paper introduces ARACH, a training-free inference-time plug-in that enhances large language models by aggregating context and reallocating attention through an adaptive context hub, thereby mitigating the attention sink phenomenon and improving performance across various tasks without requiring parameter updates or costly training.

Jingtao Wang, Yucong Wang, Jun Ding, Rui Cai, Xun Wang2026-03-13💬 cs.CL

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

The paper proposes DeReason, a difficulty-aware curriculum that optimizes post-training for general reasoning by strategically allocating non-reasoning-intensive data to supervised fine-tuning and challenging reasoning-intensive problems to reinforcement learning, thereby significantly outperforming baseline methods across STEM and mathematical benchmarks.

Hanxu Hu, Yuxuan Wang, Maggie Huan, Jannis Vamvas, Yinya Huang, Zhijiang Guo, Rico Sennrich2026-03-13💬 cs.CL

MDER-DR: Multi-Hop Question Answering with Entity-Centric Summaries

The paper introduces MDER-DR, a novel Retrieval-Augmented Generation framework that combines a Map-Disambiguate-Enrich-Reduce indexing strategy with a Decompose-Resolve retrieval mechanism to significantly improve multi-hop question answering on Knowledge Graphs by preserving contextual nuance and enabling robust reasoning without explicit graph traversal.

Riccardo Campi, Nicolò Oreste Pinciroli Vago, Mathyas Giudici, Marco Brambilla, Piero Fraternali2026-03-13💬 cs.CL

Markovian Generation Chains in Large Language Models

This paper defines the iterative inference process of large language models as Markovian generation chains and demonstrates through experiments and modeling that such processes can either increase or reduce sentence diversity depending on factors like temperature and initial input, with outcomes ranging from convergence to a recurrent set to the production of novel sentences.

Mingmeng Geng, Amr Mohamed, Guokan Shang, Michalis Vazirgiannis, Thierry Poibeau2026-03-13💬 cs.CL