💬 NLP

BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation

This paper introduces a comprehensive benchmark and error taxonomy to evaluate citation hallucinations in search-enabled LLMs, revealing that even frontier models struggle with recent papers due to reliance on parametric memory, and demonstrates that a two-stage mitigation strategy using the open-source `clibib` tool significantly improves BibTeX accuracy while minimizing regression.

Original authors: Delip Rao, Chris Callison-Burch

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Delip Rao, Chris Callison-Burch

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student writing a research paper. You ask a very smart, well-read AI assistant to find a specific book for your bibliography and write down all the details: the author's name, the title, the year it was published, and the page numbers.

You expect the AI to be perfect. But in reality, the AI often gets it wrong. It might invent a fake page number, mix up the author's middle name, or worse, cite a completely different book that sounds similar but isn't the one you asked for. This is called a "citation hallucination."

This paper is like a detective report investigating why these smart AI assistants keep making these mistakes, even when they have access to the internet, and how we can fix them.

The Problem: The "Smart but Forgetful" Librarian

The researchers tested three of the most advanced AI models available (think of them as the top three "super-librarians"). They asked these AIs to find citations for 931 different scientific papers.

They found three main problems:

The "Famous Book" Bias: The AIs were great at finding details for famous, old, or highly cited papers. It's like they had these books memorized in their heads. But when asked about a brand-new paper or a very obscure one, they started guessing.
The "Internet isn't Enough" Trap: Even when the researchers told the AIs, "Go look this up on the web," the AIs still made mistakes. It turns out that just because the AI found the right webpage doesn't mean it read the details correctly. It's like a student who finds the right textbook in the library but then writes down the wrong page number because they were daydreaming while reading.
The "All-or-Nothing" Mistake: The researchers discovered two ways the AIs fail:
- The "Wrong Book" Swap: The AI finds a different book entirely (e.g., it thinks you asked for "Harry Potter and the Chamber of Secrets" when you actually asked for "Harry Potter and the Goblet of Fire"). In this case, every detail (author, year, title) is wrong because it's talking about the wrong thing.
- The "Typo" Mistake: The AI finds the right book but gets a small detail wrong, like the page numbers or the volume number.

The Solution: The "Fact-Checker" Tool

Since the AIs are prone to daydreaming and guessing, the researchers built a new tool called clibib.

Think of clibib as a super-strict fact-checker or a digital notary.

When the AI tries to write a citation, clibib doesn't let the AI guess.
Instead, clibib takes the paper's ID number (like a DOI) and goes straight to the official publisher's database.
It grabs the exact correct details and hands them to the AI, saying, "Here is the truth. Just copy this."

The Experiment: One-Step vs. Two-Step

The researchers tested two ways to use this fact-checker:

1. The "One-Step" Method (The Distracted Student):
They told the AI: "Go search the web, find the paper, and then use the fact-checker tool to get the details."

Result: The AI still got confused. It would search, find the paper, but then forget to use the fact-checker properly or mix up the details while trying to do too many things at once. Accuracy went up a little, but not enough.

2. The "Two-Step" Method (The Editor):
They changed the process:

Step 1: Let the AI do its best guess first (using the internet).
Step 2: Take that guess and run it through the fact-checker (clibib) separately. Then, ask the AI to simply compare its guess with the fact-checker's truth and fix the mistakes.
Result: This worked amazingly well. By separating the "searching" from the "fixing," the AI stopped getting confused. The accuracy jumped from about 50% (half the citations were perfect) to nearly 80%.

The Big Takeaway

The main lesson of this paper is that AI is a great writer but a terrible librarian.

If you ask an AI to generate a bibliography from scratch, it will likely invent facts. Even if you give it internet access, it might still get the details wrong because it's trying to "remember" things rather than "read" them carefully.

The Fix: Don't trust the AI to generate the citation. Instead, let the AI find the paper, but then use a deterministic tool (like clibib) to pull the official data from the source, and have the AI simply act as a copy-paste editor to format it.

In short: If you want a perfect citation, don't ask the AI to create it. Ask the AI to find the paper, and then let a computer program grab the details for you.

1. Problem Statement

Large Language Models (LLMs) integrated with web search are increasingly used to automate scientific workflows, including literature reviews and manuscript preparation. However, these agents frequently produce BibTeX entries with pervasive field-level errors (hallucinations).

The Gap: Prior research evaluated base models without web search or relied on manual verification. This does not reflect current practice where "frontier" models use native search tools.
The Risk: Unlike free-text hallucinations, bibliographic errors are deterministic and verifiable. Fabricated or corrupted metadata (authors, titles, DOIs, venues) propagates into published manuscripts and downstream citation analyses, corrupting the scholarly record.
The Challenge: It remains unclear how much LLMs rely on parametric memory versus search retrieval when generating citations, and whether current search-enabled agents can reliably produce publication-ready metadata.

2. Methodology

The authors constructed a rigorous benchmark and evaluation pipeline to isolate the causes of citation errors and test mitigation strategies.

A. Benchmark Construction

Dataset: 931 papers across four scientific domains (AI, Medicine, Materials Science, Quantum Computing).
Stratification: Papers were divided into three citation tiers to test memory vs. search dependence:
1. Popular: High-citation papers (2010–2022), likely in training data.
2. Low-citation: Low-citation papers (2015–2025), less likely memorized.
3. Recent: Post-knowledge-cutoff papers (2025–2026), requiring search.
Ground Truth: A version-aware ground truth was constructed using the Zotero Translation Server, DBLP, PubMed, and CrossRef. This accounts for multiple citable versions of the same work (e.g., arXiv preprint vs. journal version).
Models Evaluated: Three frontier models with native search enabled:
- GPT-5 (OpenAI)
- Claude Sonnet 4.6 (Anthropic)
- Gemini 3 Flash (Google)

B. Evaluation Pipeline

Task: Known-item retrieval using natural language descriptions (e.g., "Title by First Author et al.") without providing structured identifiers (DOIs).
Error Taxonomy: A six-way classification for 9 evaluable fields (Author, Title, Year, Venue, Volume, Number, Pages, DOI, Entry Type):
- C: Correct
- M: Missing
- F: Fabricated (invented)
- P: Partially correct
- S: Substituted (correct paper, wrong version, or different paper)
- X: Not applicable
Scoring: Automated deterministic matching (Stage 1) followed by an LLM-as-judge (Stage 2) for fuzzy matches, validated by human spot-checks.

C. Mitigation Strategy: `clibib`

The authors developed clibib, an open-source Python tool that performs deterministic BibTeX retrieval from the Zotero Translation Server (aggregating CrossRef, DBLP, PubMed).

Integration Architectures Tested:
1. Single-Stage: The model uses clibib as a tool within a search loop to generate the entry.
2. Two-Stage: The model generates a baseline entry; a client-side clibib lookup is performed; a revision prompt asks the model to reconcile the baseline with the authoritative record.

3. Key Results

A. Baseline Performance (Search-Enabled)

Overall Accuracy: 83.6% of individual fields are correct.
Full Entry Accuracy: Only 50.9% of entries are fully correct (all fields correct).
Memory vs. Search: Accuracy drops significantly from Popular (92.7%) to Recent (65.0%) papers. This 27.7 percentage point drop reveals that even with search enabled, models rely heavily on parametric memory for well-known papers and struggle to parse search results for unfamiliar ones.
Error Modes:
1. Wholesale Entry Substitution: The model retrieves metadata for a completely different paper. Identity fields (Author, Title, Venue, Year) fail together.
2. Isolated Field Error: The model identifies the correct paper but hallucinates specific fields (e.g., wrong page numbers, formatting errors).

B. Mitigation Results

Single-Stage Integration: Improved accuracy by +4.4 pp (to 85.3%) but introduced a 4.8% regression rate (correct fields becoming wrong).
Two-Stage Integration (Optimal):
- Accuracy: Rose to 91.5% (+8.0 pp over baseline).
- Full Correctness: Increased from 50.9% to 78.3%.
- Regression: Drastically reduced to 0.8%.
- Insight: Separating the retrieval (deterministic lookup) from the revision (LLM reconciliation) prevents the model from overwriting correct data during the search process.

C. Field-Specific Findings

Most Accurate: Author names and Entry Types.
Least Accurate: Numerical fields (Volume, Number, Pages) and DOIs.
Correction Efficacy: clibib is highly effective at fixing Missing (49.2% correction) and Partial (47.5%) errors. It is less effective on Substituted errors (14.9%) because if the model queries the wrong paper, the database lookup cannot recover the correct one without the correct identifier.

4. Key Contributions

Benchmark: A version-aware benchmark of 931 papers across four domains and three citation tiers, designed to disentangle parametric memory from search dependence.
Evaluation Framework: A systematic evaluation of three search-enabled frontier models producing ~23,000 field-level observations, revealing that search tools do not fully compensate for knowledge gaps.
Tool (clibib): An open-source, deterministic retrieval tool that fetches authoritative BibTeX entries, serving as a robust mitigation mechanism.
Architectural Insight: Demonstration that integration architecture matters. A two-stage pipeline (Search $\to$ Baseline $\to$ Deterministic Lookup $\to$ Revision) significantly outperforms single-stage tool loops, reducing regression rates and improving accuracy independent of model capability.

5. Significance and Recommendations

Scientific Integrity: The study proves that LLM-generated citations cannot be trusted for publication without verification. Even with search, half of the entries contain errors.
Mitigation Strategy: The most effective approach is to bypass LLM-based metadata construction for structured fields. Authors and agents should use deterministic tools (like clibib or direct database exports) to retrieve metadata, using the LLM only for natural language tasks.
Verification Protocol: A lightweight check of three "identity fields" (Author, Venue, Year) is sufficient to detect "wholesale substitution" errors, the most damaging failure mode.
Future Work: The paper calls for standardized evaluation methodologies and expanded database coverage to handle recent publications and niche domains (e.g., workshop papers lacking DOIs).

Conclusion: While search-enabled LLMs improve citation accuracy, they remain unreliable for generating publication-ready BibTeX. The proposed two-stage integration with deterministic retrieval tools offers a viable path to near-perfect accuracy, ensuring the integrity of the scientific record.