Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
This paper introduces CiteTracer, a multi-agent framework that reframes citation hallucination detection as a taxonomy-aligned field-level adjudication task to achieve high-accuracy verification of scientific references through structured extraction, multi-source evidence retrieval, and specialized agent reasoning.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are walking through a massive, bustling library where every book is a scientific paper. In this library, the most important thing isn't just the story inside the book, but the "breadcrumbs" it leaves behind—citations. These are the little notes that say, "I got this idea from that book over there." They are the map that lets other explorers find the original treasure. But lately, a new kind of librarian has arrived: a super-smart robot that can write stories faster than anyone else. The problem is, this robot sometimes gets so excited about telling a good story that it invents its own breadcrumbs. It might write down a book title that sounds real, an author who sounds famous, or a year that makes sense, but if you try to walk over to that shelf, the book simply isn't there. It's a "hallucination"—a convincing lie that looks real until you check the facts.
This is a big deal because science runs on trust. If a researcher can't find the source a paper claims to use, the whole chain of evidence breaks. Until now, tools designed to catch these fake breadcrumbs were like bouncers at a club with a simple rule: "You're in, or you're out." They would check if a book existed, but if the book existed with a slightly different name, or if the author's name was spelled weirdly, the bouncer would get confused and let a fake in or kick a real one out. They couldn't tell the difference between a typo, a nickname, and a complete fabrication.
Enter CITETRACER, a new team of digital detectives built by researchers Mingzhe Li, Zhiqiang Lin, and Shiqing Ma. Instead of just asking "Is this fake?", they created a much smarter system that asks, "What exactly is wrong with this?" They realized that citations have different parts—like the title, the author, the year, and the publisher—and a mistake in one part doesn't always mean the whole thing is a lie.
To solve this, the team built a 12-code rulebook. Think of it like a traffic cop who doesn't just say "Stop" or "Go," but can give you a specific ticket: "You were speeding," "You ran a red light," or "You forgot your license." Their system sorts citations into three main buckets: REAL (perfect or just a little messy), POTENTIAL (looks okay but needs a second look, like a nickname or a non-academic source), and HALLUCINATED (totally made up).
The detectives work in a four-step relay race. First, a Reference Extractor acts like a high-tech scanner that reads the messy text from a PDF and turns it into a clean, organized list of facts. Next, a Cascading Evidence Collector goes on a scavenger hunt. It starts by checking a quick memory cache, then follows direct links (like a DOI), then asks specialized academic databases, and finally, if needed, does a broad web search. It's like checking your own notes, then calling the author, then checking the library catalog, and finally asking the whole town.
Once they have the evidence, a Field Matcher compares the robot's claim against the real facts, piece by piece. If the title matches but the year is off, it flags just the year. Finally, if the case is tricky, a panel of Class-Specialist Judgers steps in. These are like expert judges who specialize in different types of errors. One judge handles the "Real but messy" cases, another handles the "Maybe it's real" cases, and a third handles the "Totally fake" cases.
The team tested their system on a massive playground of 2,450 fake and real citations they created themselves, plus 957 real-world examples of fake citations that had already been caught by conference organizers. The results were impressive: CITETRACER got it right 97.1% of the time on their test set. It was especially good at spotting the tricky "fake" ones (98.5% accuracy) and the "maybe" ones (95.8% accuracy), far beating other tools that just guessed "yes" or "no." When they tested it on the real-world mess of actual conference papers, it caught 97.1% of the fabricated references without giving up on any of them.
In short, the paper shows that by breaking the problem down into tiny, specific pieces and using a team of specialists instead of one generalist, we can catch the robot's lies with much higher precision. It turns a blurry "maybe fake" into a clear "here is exactly what is wrong," helping keep the library of science honest.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.