💬 NLP

CheckIfExist: Detecting Citation Hallucinations in the Era of AI-Generated Content

This paper introduces "CheckIfExist," an open-source web tool that addresses the growing issue of AI-generated citation hallucinations by providing real-time, multi-source validation of bibliographic references through a cascading architecture that verifies authenticity across major scholarly databases.

Original authors: Diletta Abbonato

Published 2026-02-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Diletta Abbonato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are writing a story, and you want to prove your points by quoting famous experts. You ask a super-smart, fast-talking robot (an AI) to help you find those quotes. The robot quickly hands you a list of sources. They look perfect: the names sound real, the book titles are catchy, and the dates make sense.

But here's the catch: The robot is lying.

It has invented these books and authors out of thin air. In the world of science, this is called a "citation hallucination." It's like the robot dreaming up a fake witness for a court case. If you don't check, you might accidentally put a fake witness on the stand, and your whole argument could collapse.

This paper introduces a new tool called CheckIfExist to solve this problem. Here is how it works, explained simply:

The Problem: The "Fake News" of Science

In the past, if a researcher made a mistake in a citation, it was usually just a typo (like writing "1995" instead of "1996"). But now, AI models can generate entirely fake references that look 100% real.

The Danger: These fake references are sneaking into top-tier scientific conferences. Reviewers are so busy they miss them.
The Consequence: If a scientist builds their research on a fake paper, it's like building a house on a foundation of sand. The whole structure is unstable.

The Solution: The "Fact-Checker" App

Think of CheckIfExist as a super-powered fact-checker or a digital bouncer for your bibliography.

It's a Web Tool: You don't need to install anything. You just go to a website.
The "Three-Headed Monster" Strategy:
Most tools check just one database (like checking a library catalog). But what if the book isn't in that specific library?
- CheckIfExist is smarter. It sends your reference to three different massive libraries at the same time:
  - CrossRef: The official registry for published papers.
  - Semantic Scholar: A giant AI-driven database of academic papers.
  - OpenAlex: A free, open database that gathers info from everywhere.
- The Analogy: Imagine you are trying to verify if a person really exists. You don't just ask one neighbor; you check the phone book, the voter registry, and the local school records. If all three say "Yes, they exist," you are sure. If they all say "No," or if the names don't match, you know something is fishy.
How It Spots the Fakes:
- The "Uncanny Valley" Test: The tool looks at the details. Does the author's name match the journal? Is the year right?
- The "Chimera" Detector: Sometimes AI mixes two real papers to make one fake one (e.g., taking the title of Paper A and the author of Paper B). The tool spots these mismatches instantly.
- The "LaTeX" Cleaner: Scientists often copy-paste text from their code files, which is full of messy formatting symbols (like \textit or \vspace). The tool automatically cleans this "digital dust" off the text so it can read the actual words.

What Happens When You Use It?

Instant Feedback: You paste a list of references (even a messy one), and within seconds, it tells you: "This one is real," "This one is fake," or "This one is suspicious."
The "Do-Over" Button: If a reference is fake or has a typo, the tool doesn't just say "No." It finds the real version of the paper and gives you a clean, perfect citation to copy and paste back into your work.
Free and Open: Unlike some expensive services that charge you a fee to check just a few papers, this tool is free and open to everyone. It's like a public library vs. a private club.

Why Does This Matter?

Science relies on trust. If we can't trust the sources scientists cite, we can't trust the science itself.

For Writers: It saves you from embarrassment and ensures your work is solid.
For Reviewers: It helps them catch liars (or confused AI) before a paper gets published.
For the World: It keeps the "library of human knowledge" from getting filled with fake books.

In short: CheckIfExist is a free, fast, and smart tool that acts as a guardian for scientific truth, ensuring that every citation in your paper is a real, existing fact, not a robot's daydream.

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into academic workflows has introduced a critical threat to bibliographic integrity: citation hallucination. Unlike traditional citation errors caused by human transcription mistakes, LLM hallucinations are systematic fabrications that generate plausible-sounding but non-existent references (e.g., fake authors, journals, or DOIs).

Severity: Recent analyses of top-tier AI conferences (NeurIPS 2025, ICLR) revealed that hallucinated citations have already passed peer review, with over 100 fake citations found in 53 NeurIPS papers.
Limitations of Current Tools: Existing reference managers (Zotero, Mendeley, EndNote) excel at organization and formatting but lack real-time validation capabilities. They store metadata without verifying authenticity. Commercial hallucination detectors often impose restrictive usage limits or high costs, creating a barrier for researchers in resource-constrained environments.
Consequence: The proliferation of unverified citations distorts bibliometric indicators, misdirects research, and undermines the self-correcting nature of science.

2. Methodology: System Architecture

The paper proposes CheckIfExist, an open-source, web-based tool designed for real-time, multi-source validation of bibliographic references.

A. Core Architecture

The system is built using React with TypeScript and employs a cascading validation architecture to query three complementary scholarly databases:

CrossRef: Primary source for DOI-registered works (140M+ records).
Semantic Scholar: Fallback source with enhanced author disambiguation (200M+ records).
OpenAlex: Aggregates data from multiple repositories (CrossRef, PubMed, ORCID, arXiv) to ensure broad coverage, particularly for open-access and non-English publications.

B. Preprocessing & Input Handling

LaTeX Cleaning: A preprocessing module automatically strips LaTeX formatting commands (e.g., \vspace, \textit) from user input, allowing researchers to paste raw text directly from manuscript source files without manual cleaning.
Input Modes: Supports both Quick Check (free-form text in APA/MLA/Chicago) and Batch Check (structured BibTeX or newline-separated lists).

C. Verification Algorithm

The core logic utilizes a multi-stage matching algorithm with cascading fallbacks:

Initial Query: The system queries CrossRef first. If no matches are found, it falls back to Semantic Scholar and OpenAlex.
String Similarity: Uses the Levenshtein distance metric to calculate normalized similarity scores between the input query and database candidates.
- Formula: $similarity(a, b) = 1 - \frac{lev(a, b)}{\max(|a|, |b|)}$
Author Verification: Extracts family names and checks for containment. It identifies "suspect authors" (capitalized tokens in the query that do not match any real author in the database).
Cross-Validation: If the initial score is low, the system cross-validates author lists across all three sources. Authors appearing in multiple databases receive a confidence bonus; those appearing in only one or none are flagged.
Confidence Scoring:
- A composite score is calculated based on title, author, journal, and year similarities.
- Penalties: Applied for title mismatches (-20), author mismatches (-20), and detected fake authors (-10 to -20).
- Bonuses: Multi-source confirmation adds up to +10 to the score.
- Thresholds: A score >50 indicates existence; >70-80 indicates high confidence.

3. Key Contributions

Specialized Validation Tool: Fills the gap between reference management (organization) and verification (authenticity). It is the first tool to offer unlimited, free, real-time validation against multiple authoritative databases.
Cascading Multi-Source Logic: By combining CrossRef, Semantic Scholar, and OpenAlex, the system achieves higher recall than single-database approaches while maintaining precision through cross-source author confirmation.
Automated Correction & Export: Unlike tools that only flag errors, CheckIfExist generates corrected metadata. It outputs:
- Validated APA 7th edition citations.
- Clean BibTeX records with automatically generated keys, ready for immediate re-import into LaTeX projects.
Open Source & Accessibility: Released under the MIT License, ensuring accessibility for all researchers regardless of institutional funding.

4. Results and Evaluation

Performance: The system returns validation results within seconds, processing entries sequentially with rate limiting (800ms intervals) to comply with API policies.
Accuracy: The multi-source cross-validation mechanism effectively detects "chimeric" hallucinations (blends of real and fake data) and fully fabricated entries.
Comparative Advantage: As shown in the paper's comparison table (Table 1), CheckIfExist is unique in offering immediate validation, hallucination detection, fake author detection, and batch verification simultaneously, features absent in major commercial reference managers.

5. Significance and Impact

Restoring Scholarly Integrity: Provides a necessary defense mechanism against the "market failure" where the low cost of generating fake citations (via AI) outweighs the high cost of manual verification.
Workflow Integration: Designed to fit seamlessly into existing research pipelines. Researchers can audit bibliographies before submission, reviewers can efficiently check manuscripts, and publishers can integrate validation into submission pipelines.
Scientometric Reliability: By filtering spurious references from datasets, the tool helps ensure the accuracy of large-scale bibliometric analyses and citation network studies.
Future-Proofing: As AI writing assistants become ubiquitous, CheckIfExist serves as a critical "human-in-the-loop" support tool, ensuring that the scientific record remains verifiable and trustworthy.

Availability: The tool is publicly available at https://zabbonat.github.io/References-Validation/.