🤖 AI

CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text

The paper introduces CiteCheck, a hybrid framework that combines scholarly retrieval with structured LLM verification to accurately detect citation hallucinations in scientific texts, achieving superior performance on a newly constructed physics benchmark compared to existing baselines.

Original authors: Khashayar Khajavi, Shaghayegh Sadeghi, Rise Adhikari, Alexander Tessier

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Khashayar Khajavi, Shaghayegh Sadeghi, Rise Adhikari, Alexander Tessier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a scientific report written by a very confident, very fast robot. The robot writes beautifully, but when it lists the books and papers it used as sources, it sometimes makes things up. It might invent a book title, mix up the author's name, or even create a whole paper that never existed. This is called a "citation hallucination."

The paper you provided introduces a new tool called CITECHECK to catch these mistakes. Think of CITECHECK not as a robot that guesses, but as a super-detective librarian with a specific job: to verify if a reference is real and if the details are correct.

Here is how CITECHECK works, broken down into simple steps:

1. The "Library Search" (Retrieval)

When CITECHECK sees a citation (like "Smith, 2023, The Physics of Stars"), it doesn't just trust the robot's memory. Instead, it acts like a librarian who immediately runs to the shelves (or the internet) to find the actual book.

It checks multiple databases (like CrossRef, Semantic Scholar, and arXiv) to find the real paper that matches the title.
If it can't find a match, it knows immediately that the citation is likely fake.

2. The "Spot the Difference" Game (Verification)

Once the librarian finds the real book, CITECHECK puts the robot's citation side-by-side with the real book's cover and details.

It uses a smart AI "judge" to compare them.
The Score: The judge gives the citation a score from 0 to 10.
- 10 (Exact Match): The robot got everything right.
- 6 or 7 (Minor Hallucination): The book exists, but the robot got the year wrong, misspelled the author's name, or changed the URL slightly. It's like ordering a pizza and getting the right toppings but the wrong crust.
- 0 or 1 (Major Hallucination): The book doesn't exist at all, or the robot is describing a completely different book. It's like ordering a pizza and getting a sandwich that doesn't exist.

3. The "Second Opinion" (Reviewer Pass)

Sometimes, the robot might be tricky. It might give you a real URL (like a link to a real paper) but pair it with a fake title or author. It's like giving you a real address but telling you the wrong person lives there.

CITECHECK has a "reviewer" step for these suspicious cases. A second AI looks closely to make sure the title and authors actually match the link, not just the link itself.

The "Training Ground" (The Benchmark)

To prove their tool works, the authors built a "training gym" for the system. They took 982 real physics citations and created two types of fake versions:

Minor Fakes: Real papers with small, sneaky errors (like changing "2023" to "2024").
Major Fakes: Completely made-up papers that sound plausible but don't exist.

The Results: The Detective Wins

When they tested CITECHECK against other big AI models (like GPT, Claude, and Gemini), the results were clear:

CITECHECK was the best. It correctly identified real vs. fake citations about 89% of the time.
Why it won: The other AI models tried to guess from their memory or by just reading the text. CITECHECK won because it actually went out and checked the facts against real databases before making a decision.
Even when the other AIs were given "cheat sheets" (examples of what to look for) or allowed to use the internet, they still couldn't beat CITECHECK's method of "Search first, then compare."

The Bottom Line

The paper claims that to stop AI from lying about its sources, you can't just ask the AI to "be honest." You need a system that retrieves the real evidence first and then uses a structured comparison to catch the lies. CITECHECK does exactly that, acting as a reliable fact-checker for scientific writing.

Technical Summary: CITECHECK

Problem Statement

Large Language Models (LLMs) are increasingly integrated into scientific workflows, including manuscript drafting and report generation. However, a critical reliability issue persists: LLMs frequently generate "citation hallucinations." These are references that appear stylistically legitimate but contain corrupted metadata (e.g., incorrect authors, years, or DOIs) or point to entirely non-existent papers. Such errors undermine the traceability, attribution, and reproducibility of scientific communication. Existing verification methods often rely on the model's internal parametric knowledge or simple web searches, which struggle to distinguish between subtle metadata drift and complete fabrication, or fail to ground the verification in structured scholarly evidence.

Methodology: The CITECHECK Framework

CITECHECK is a hybrid framework designed to detect citation hallucinations by grounding each reference in external scholarly sources rather than relying on the LLM's internal knowledge. The system operates through a structured pipeline consisting of four main stages:

1. Metadata Extraction

The system first parses a raw citation string into structured fields (title, authors, year, URL, DOI, arXiv ID). It employs a rule-based parser for standard patterns and invokes an LLM-based parser only when deterministic extraction fails. This step generates retrieval signals without determining validity.

2. Candidate Retrieval Cascade

CITECHECK attempts to locate the real publication corresponding to the citation using a waterfall cascade over multiple external sources:

Direct Lookup: If an arXiv identifier is present, a direct arXiv lookup is performed first.
Structured APIs: The system queries CrossRef, Semantic Scholar, and OpenAlex using title-based searches.
Web Search Fallback: If structured sources fail to return a reliable match, an LLM-assisted web search is triggered.
Selection Logic: Candidates are accepted based on normalized Levenshtein similarity between the cleaned citation title and the candidate title, with additional confidence thresholds (e.g., from CrossRef) and year-based ranking signals.

3. Structured LLM Verification

Once a candidate publication is retrieved, a verifier LLM compares the original citation metadata against the retrieved candidate's metadata. Unlike standard classification prompts, the verifier outputs:

A numeric score (0–10) representing the degree of agreement.
A confidence level and a short rationale.
Identification of specific discrepancies.
The system uses a numeric score to allow for calibrated decision boundaries rather than relying on the model's categorical label, which can be inconsistent.

4. Thresholding and Reviewer Pass

The numeric score is mapped to three labels using fixed thresholds ( $\tau_M$ and $\tau_E$ ) determined on a validation set:

EXACT: The citation faithfully describes the real publication ( $s \ge \tau_E$ ).
MINOR: The intended paper exists, but metadata fields are corrupted ( $\tau_M \le s < \tau_E$ ).
MAJOR: No matching publication exists, or the candidate is too dissimilar ( $s < \tau_M$ ).

A "Reviewer Pass" is applied for suspicious cases where an identifier (URL/arXiv ID) resolves to a real paper, but the title similarity is low. A second LLM judge re-evaluates these cases to ensure the citation text accurately describes the source, preventing false positives where a valid link is paired with fabricated metadata.

Dataset Construction

To evaluate the framework, the authors constructed a benchmark of 982 citations drawn from physics literature, covering nine subdomains (e.g., astrophysics, condensed matter, quantum computing).

Controlled Corruptions: The dataset includes valid citations, "minor hallucinations" (metadata drift such as altered authors or years while keeping the paper identity), and "major hallucinations" (fully fabricated references that remain topically plausible).
Generation: Corruptions were generated using GPT-4o-mini with specific temperature settings to simulate transcription errors (minor) and substantial fabrication (major).
Balance: The final benchmark is roughly balanced across the three classes (36.25% valid, 30.55% minor, 33.20% major).

Results

Evaluated on a held-out test set of 792 citations, CITECHECK achieved:

88.9% Accuracy
88.7 Macro-F1

Comparative Performance:

CITECHECK outperformed direct LLM baselines (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash) in both zero-shot and few-shot settings.
It surpassed the strongest baseline (Claude Sonnet 4.6 with web search and few-shot examples) by 5.8 macro-F1 points and 5.7 accuracy points.
Notably, CITECHECK achieved this while remaining zero-shot at the verification stage, demonstrating that retrieval-grounded verification is more effective than in-context supervision or prompting alone.
Ablation Studies: Removing the web-search fallback reduced macro-F1 by 7.0 points, and removing the reviewer pass reduced it by 4.0 points, confirming the necessity of both broad retrieval and targeted review for handling subtle metadata drift.

Significance and Claims

The paper claims that reliable citation verification requires a shift from relying on the LLM's internal knowledge to a pipeline that combines scholarly retrieval, structured LLM-based comparison, and calibrated decision rules.

Key takeaways emphasized by the authors include:

Retrieval is Essential: Even strong LLMs struggle to detect hallucinations without access to external evidence; web search significantly improves baseline performance but is insufficient on its own without structured comparison.
Structured Comparison over Prompting: Treating verification as a candidate retrieval followed by metadata comparison yields higher accuracy than direct prompting, particularly for "minor" hallucinations where the paper exists but metadata is corrupted.
Granular Detection: The framework successfully distinguishes between exact matches, minor metadata errors, and major fabrications, a nuance often lost in binary "real vs. fake" detection.

The authors conclude that CITECHECK provides a foundation for broader verification pipelines for AI-generated scientific text, though they acknowledge limitations regarding the current focus on physics and the need for future work on naturally occurring LLM-generated bibliographies and claim-level alignment.