Do Deployment Constraints Make LLMs Hallucinate Citations? An Empirical Study across Four Models and Five Prompting Regimes

This empirical study demonstrates that deployment-motivated prompting constraints significantly exacerbate citation hallucinations across four large language models, with no model achieving a citation existence rate above 47.5% and a substantial portion of unverifiable outputs being fabricated, thereby underscoring the critical need for post-hoc verification in academic and software engineering contexts.

The Gist

Imagine you are asking a very smart, well-read assistant to write a short essay for you. You ask them to include a list of "real books" they used to write it. The assistant hands you a beautiful, perfectly formatted list of references. They look professional: they have authors, titles, publication years, and even ISBN numbers.

But here's the catch: most of those books don't actually exist.

This is exactly what researchers Chen Zhao, Yuan Tang, and Yitian Qian investigated in their paper, "Do Deployment Constraints Make LLMs Hallucinate Citations?" They wanted to see if putting "rules" on these AI assistants makes them lie more about their sources.

Here is the breakdown of their study using simple analogies.

The Setup: The "Closed-Book" Exam

The researchers treated the AI models like students taking a closed-book exam. They couldn't look up answers on the internet; they had to rely entirely on what was in their memory (training data).

They gave four different "students" (two famous, expensive AI models and two open-source ones) 144 different writing prompts. They asked them to write academic paragraphs and include a list of references.

To make things tricky, they gave the students five different sets of rules (constraints):

1. Baseline: Just write normally.
2. Temporal: "Only use books published between 2020 and 2025." (Like asking for only the latest news).
3. Survey: "Write a broad overview covering many different topics." (Like asking for a 10-page summary of a whole library).
4. Non-Disclosure: "Don't say you memorized these books from your training data." (A rule often used in corporate settings).
5. Combo: All the above rules combined.

The Investigation: The "Fact-Checker"

After the AI generated thousands of citations, the researchers didn't just take their word for it. They built an automated fact-checking machine. This machine checked every single citation against two massive databases of real academic papers (Crossref and Semantic Scholar).

They categorized the results into three buckets:

• Existing: The book/paper is real and matches the citation.
• Fabricated: The book/paper is a complete lie (fake title, fake author, fake year).
• Unresolved: The machine couldn't tell. It looked plausible, but the machine couldn't find it in the database. (The researchers found that about half of these "unsure" ones were actually lies too).

The Big Findings

1. The "Perfectly Fake" Problem

The most shocking discovery was that no AI model got a majority of citations right. Even the best model only had about 47% of its citations verified as real.

When the researchers added the "Temporal" rule (only recent papers), the AI's performance tanked.

• The Analogy: Imagine asking a student, "Tell me about the history of the internet, but only use books published last week." The student, desperate to follow the rule, invents fake books that look like they were published last week.
• The Result: The AI followed the rule perfectly (the dates were correct), but the books didn't exist. The "format" was perfect, but the "substance" was zero.

2. The "Rich vs. Poor" Student Gap

The researchers compared the expensive, proprietary models (like GPT-4o and Claude) against the free, open-source ones (like LLaMA and Qwen).

• The Analogy: The expensive models are like students who went to a massive, private library with millions of books. The open-source models are like students who only had access to a small public library.
• The Result: The "Rich" students did better, but they still lied a lot. The "Poor" students lied even more. When the rules got harder (like the "Survey" rule asking for a broad overview), the gap between them got huge. The expensive models could still find some real books; the open-source models mostly made them up.

3. The "Unresolved" Trap

About 36% to 61% of the citations fell into the "Unresolved" bucket.

• The Analogy: Imagine a student hands you a reference that looks like a real book, but the library catalog is missing that specific entry. Is the book real but lost? Or did the student invent it?
• The Danger: The researchers found that nearly half of these "mystery" citations were actually fake. This is dangerous because if you just look at the list, it seems trustworthy, but it's actually a trap.

4. The "Combo" Disaster

When they combined all the rules (Recent + Broad + Secretive), the results were terrible.

• The Result: The open-source models collapsed almost entirely (near 0% real citations). Even the expensive models struggled, though they managed to keep a tiny sliver of real citations.
• The Twist: Even though the quality of the citations was terrible, the AI kept generating more of them to fill the quota. It was like a factory churning out more and more fake products just to meet the production target.

Why Should You Care?

This paper is a warning label for anyone using AI to write academic papers, technical reports, or software documentation.

• Don't trust the list: Just because an AI gives you a list of citations that looks perfect (with DOIs and years), it doesn't mean they are real.
• Rules make it worse: If you ask an AI to follow strict rules (like "only use 2024 papers"), it is more likely to hallucinate (lie) to satisfy you.
• The "Unresolved" is a red flag: If you can't easily verify a source, assume it might be fake.

The Bottom Line

The researchers conclude that prompt engineering alone cannot fix this. You can't just tell the AI "don't lie."

If you want reliable citations, you need to treat AI-generated lists as drafts only. You must manually check every single reference against a real database before you use it in any serious work. The AI is a great writer, but it is a terrible librarian.

Technical Summary

Here is a detailed technical summary of the paper "Do Deployment Constraints Make LLMs Hallucinate Citations? An Empirical Study across Four Models and Five Prompting Regimes."

1. Problem Statement

Large Language Models (LLMs) are increasingly used for drafting academic text and supporting evidence synthesis in Software Engineering (SE). However, they frequently suffer from citation hallucination: generating bibliographic references that appear structurally complete (with authors, venues, DOIs, and years) but correspond to non-existent works.

While prior research has established that hallucination occurs, it remains unclear how real-world deployment constraints affect citation verifiability. Specifically, the study investigates how the following constraints impact the reliability of generated citations in a "closed-book" setting (no external retrieval):

• Temporal constraints: Restricting citations to a specific publication year window.
• Survey-style breadth: Requiring a synthesis of literature across multiple categories.
• Non-Disclosure policies: Instructing models not to claim access to memorized training data.

2. Methodology

Experimental Design

• Models Evaluated:
  • Proprietary: Claude Sonnet (Anthropic) and GPT-4o (OpenAI).
  • Open-Weight: LLaMA 3.1–8B (Meta) and Qwen 2.5–14B (Alibaba).
• Dataset: A curated set of 144 claims (questions) spanning six academic domains, including 24 specifically in Software Engineering & Computer Science (SE & CS).
• Prompting Regimes (5 Conditions):
  1. Baseline: Standard academic paragraph with 5 citations.
  2. Temporal: 5 citations restricted to a specific 5-year window (ending in 2025).
  3. Survey: 8 citations organized into 3–4 approach categories (simulating literature review breadth).
  4. Non-Disclosure: 5 citations with instructions to avoid claiming access to training data.
  5. Combo: Combination of all three constraints (Temporal + Survey + Non-Disclosure) with 8 citations.
• Total Runs: 2,880 runs generating 17,443 individual citations. All models used deterministic decoding (temperature = 0).

Verification Pipeline

The authors developed an automated, deterministic pipeline to verify citations against Crossref and Semantic Scholar.

1. Parsing: Extracts structured fields (title, authors, venue, year, DOI).
2. Retrieval: Performs lookups via DOI (Crossref) and title searches (Semantic Scholar/Crossref).
3. Scoring: Calculates a weighted similarity score ($s$):
   $$s = 0.60 \cdot t + 0.20 \cdot a + 0.15 \cdot y + 0.05 \cdot v$$
   Where $t$ is fuzzy title similarity, $a$ is author overlap, $y$ is year agreement, and $v$ is venue similarity.
4. Labeling (Three-Way Taxonomy):
   • Existing: Score $\ge$ 0.85 (Verified match).
   • Unresolved: 0.60 $\le$ Score < 0.85 (Ambiguous; could be real but unconfirmed or partially fabricated).
   • Fabricated: Score < 0.60 or no candidate found.
   • Note: The "Unresolved" category is critical, as manual audits revealed ~46% of these were actually fabricated.

3. Key Contributions

1. Curated Dataset: A benchmark of 144 claims across six domains, specifically including SE & CS, designed to test citation reliability under constraints.
2. Automated Verification Framework: A deterministic pipeline with a three-way labeling taxonomy (Existing, Unresolved, Fabricated) that distinguishes clearly fake citations from those that are merely hard to verify. The pipeline achieved a Cohen's $\kappa$ of 0.63 against human labels.
3. Empirical Analysis of Constraints: A systematic evaluation of how specific deployment constraints (Temporal, Survey, Non-Disclosure) degrade citation quality, revealing that format compliance does not guarantee factual accuracy.

4. Key Results

Overall Performance

• No model achieved a majority of verifiable citations. The highest existence rate observed was 0.475 (Claude Sonnet under Survey conditions).
• Unresolved Dominance: Across all models and conditions, 36–61% of citations fell into the "Unresolved" category. Manual audits suggest nearly half of these are actually fabricated, indicating a massive "high-risk" pool of citations that automated checks cannot easily dismiss.

Impact of Constraints

• Temporal Constraints (Steepest Decline): Restricting citations to a recent 5-year window caused the most severe drop in verifiability.
  • Example: GPT-4o's existence rate dropped from 0.235 (Baseline) to 0.019 (Temporal).
  • Models maintained format compliance (correct year fields) but generated non-existent papers within that window.
• Survey-Style Prompting: Widened the gap between proprietary and open-weight models.
  • Proprietary models (Claude) improved slightly (0.381 $\to$ 0.475), while open-weight models (Qwen) degraded significantly (0.090 $\to$ 0.020), with fabrication rates reaching 54.7%.
• Non-Disclosure Policies: Did not eliminate errors but redistributed them. It reduced the number of "obviously wrong" citations (Fabricated) but increased the "Unresolved" category by suppressing DOIs, making errors harder to detect.
• Combo Condition: Produced the worst outcomes. Three of the four models dropped to near-zero existence rates.

Proprietary vs. Open-Weight Gap

• A significant and persistent gap exists between proprietary and open-weight models across all conditions.
• At Baseline, proprietary models had existence rates of ~0.38 and 0.23, while open-weight models were at ~0.07 and 0.09.
• This gap widened under Survey conditions ($\Delta = +0.310$), suggesting proprietary models have better training data coverage for academic literature.

Domain Specifics (SE & CS)

• The SE & CS domain exhibited an existence rate of 0.132, comparable to the cross-domain average.
• The proprietary–open-weight gap persisted within the SE domain, with Claude Sonnet reaching 0.349 while open-weight models remained below 0.10.

5. Significance and Implications

• Compliance $\neq$ Accuracy: Models can perfectly adhere to formatting constraints (e.g., specific year ranges, DOI presence) while generating entirely hallucinated content. Format-level checks are insufficient for validation.
• The "Unresolved" Danger: A binary (Real/Fake) evaluation metric is dangerous because it ignores the large "Unresolved" category, which contains a high proportion of fabricated citations.
• SE Workflow Risks: Since Systematic Literature Reviews (SLRs) rely on accurate citations, using LLMs for drafting these reviews without rigorous post-hoc verification poses a high risk of propagating false evidence into the engineering literature.
• Recommendations:
  • For Developers: Implement post-hoc verification pipelines against multiple databases (Crossref, Semantic Scholar, DBLP) and treat "Unresolved" citations as high-risk.
  • For Researchers: Never trust LLM-generated citations as final. They must be manually verified, and persistent identifiers (DOIs) should be required.
  • Future Directions: Prompt engineering alone cannot solve this; reliable generation requires Retrieval-Augmented Generation (RAG) or built-in verification architectures.

Conclusion

The study concludes that deployment constraints consistently worsen citation hallucination, often masking the failure behind format compliance. The "Unresolved" category represents a critical blind spot in current evaluation methods. For the Software Engineering community, the practical takeaway is clear: LLM-generated reference lists must be treated as drafts requiring independent, automated, and manual verification before inclusion in any scholarly or technical output.