How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

This study utilizes a massive 172-billion-token evaluation across diverse models, context lengths, and hardware to reveal that while model selection is the primary determinant of accuracy, hallucination rates in document Q&A rise significantly with context length and vary non-linearly with temperature, highlighting that grounding ability and fabrication resistance are distinct capabilities.

The Gist

Imagine you hire a team of super-smart librarians (the AI models) to answer questions based only on a specific set of books you give them. You want them to be perfect: if the answer isn't in the books, they should say, "I don't know." If the answer is there, they should find it instantly.

But here's the scary truth this paper reveals: Even the best librarians sometimes make up stories. They might confidently tell you a fact that isn't in the books, just because it sounds plausible. This is called "hallucination."

The authors of this paper, Kamiwaza AI, didn't just ask a few questions and guess. They ran a massive experiment—172 billion tokens worth of testing (that's like reading the entire internet 10 times over)—to see exactly how often these AI librarians lie, and why.

Here is the breakdown of their findings, translated into everyday language:

1. The "Library Size" Problem (Context Length)

Imagine you give a librarian a single pamphlet. They can probably answer your questions perfectly. Now, imagine you give them a library with 200,000 books.

• The Finding: As the library gets bigger, the librarians get confused and start making things up.
• The Analogy: It's like trying to find a specific needle in a haystack. If the haystack is small, you find it. If the haystack is the size of a mountain, you might just grab a random piece of hay and pretend it's the needle because you're overwhelmed.
• The Data: At a small library size (32K), the best AI lies about 1% of the time. But at a massive library size (200K), every AI starts lying more than 10% of the time. Some models, which were great at small libraries, completely collapse and lie 70% of the time in huge libraries.

2. The "Smartness" vs. "Honesty" Trap

You might think, "If a librarian is really smart, they won't lie." The paper proves this is false.

• The Finding: Being good at finding facts (Grounding) is totally different from being good at refusing to make up facts (Fabrication Resistance).
• The Analogy: Think of a model like a Llama 3.1 70B. It's like a genius who can find any book in the library instantly (90%+ accuracy at finding facts). But, it's also a compulsive storyteller who will invent a whole new chapter if you ask about something that isn't there. It's "smart but untrustworthy."
• The Contrast: Other models, like the GLM 4.5, are slightly less "flashy" at finding things but are incredibly disciplined. They rarely invent things. The paper shows that honesty is a training habit, not just a result of being big and smart.

3. The "Temperature" Knob (The Chaos Dial)

AI models have a setting called "Temperature."

• Low Temp (0.0): The model is a robot. It picks the most logical next word every time.
• High Temp (1.0): The model is a jazz musician. It takes risks and tries different words.
• The Old Rule: Everyone thought, "For facts, always set the dial to 0.0 (Robot Mode)."
• The New Discovery: This is dangerous advice!
  • The "Robot" Glitch: When set to 0.0, especially with huge libraries, the AI sometimes gets stuck in a loop, repeating the same sentence forever (like a broken record). This happens 48 times more often at 0.0 than at higher settings!
  • The "Jazz" Benefit: Surprisingly, turning the dial up a little (to 0.4 or 0.7) actually helps the AI stop lying in some cases and prevents it from getting stuck in loops.
  • Takeaway: Don't just set it to zero. You need to tune it carefully.

4. The Hardware Myth

People often worry: "If I run this AI on an NVIDIA chip, will it lie less than on an AMD or Intel chip?"

• The Finding: No. It doesn't matter.
• The Analogy: It's like asking if a car drives better on a Honda engine or a Ford engine. If the driver (the AI model) is the same, the car performs the same. The paper tested three different hardware giants and found zero difference in how much the AI lied. You can pick your hardware based on price, not fear of hallucinations.

5. The "Best Librarian" Ranking

The paper tested 35 different AI models. Here is the hierarchy they found:

• The Gold Standard: GLM 4.5 is the current champion. It lies only about 1.2% of the time at small library sizes.
• The "Big but Risky" Models: The massive Llama models (like the 405B or 70B) are very popular, but they lie 25% to 50% of the time. They are great at finding facts but terrible at knowing when not to answer.
• The "Small but Honest" Models: Some smaller models (like MiniMax) are surprisingly honest and reliable.

The Big Picture: What Should You Do?

If you are a business owner trying to use AI to answer questions from your documents:

1. Don't trust the biggest model. Size doesn't equal honesty. Pick a model family known for being "honest" (like GLM or MiniMax) rather than just the one with the most parameters.
2. Watch your library size. If you feed the AI too much text at once, it will start hallucinating. Break big documents into smaller chunks.
3. Don't set the temperature to zero blindly. It might make the AI get stuck in loops or lie more. Try a middle setting (0.4 or 0.7).
4. Accept that lies happen. Even the best AI will lie about 1% of the time. You need a safety net (human review or other checks) to catch those lies.

In short: AI is getting smarter at finding information, but it hasn't learned to be perfectly honest yet. The more text you give it, the more likely it is to make things up. Choose your tools wisely, and don't let the "biggest" model fool you.

Technical Summary

Here is a detailed technical summary of the paper "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms."

1. Problem Statement

The paper addresses a critical gap in enterprise AI deployment: the lack of reliable, large-scale measurement of hallucination rates (fabrication) in Large Language Models (LLMs) when answering questions grounded in provided documents. Existing evaluation methods suffer from three fundamental flaws:

• Data Contamination: Static benchmarks often contain data the models have seen during training, inflating performance scores.
• LLM-as-Judge Bias: Using LLMs to evaluate other LLMs introduces systematic biases and unreliability.
• Statistical Insufficiency: Most studies rely on single-run results with small datasets, lacking the statistical power to draw confident conclusions.

The core question is: How much do LLMs actually hallucinate in document Q&A, and how do factors like context length, temperature, and hardware influence this?

2. Methodology: RIKER

The study utilizes RIKER (Retrieval Intelligence and Knowledge Extraction Rating), a "ground-truth-first" evaluation methodology that inverts the traditional paradigm.

• Paradigm Inversion: Instead of extracting ground truth from existing documents (which requires human annotation), RIKER generates documents from a known relational database of facts.
• Deterministic Scoring: Because the ground truth is defined before document generation, every question has a verifiable answer. This allows for deterministic scoring without human annotators or LLM judges.
• Contamination Resistance: Corpora are regenerated with different random seeds, ensuring models cannot have memorized the test data.
• Question Taxonomy:
  • Single-document (L01–L04): Tests retrieval and comprehension.
  • Aggregation (L05–L10): Tests cross-document synthesis.
  • Hallucination Probes (L11–L12): Questions about entities or facts that do not exist in the corpus. Any specific answer is a definitive fabrication.

3. Experimental Design

The study represents a massive scale-up from previous work, involving:

• Scale: Over 172 billion tokens of evaluation across 4,264 runs.
• Models: 35 open-weight models spanning 7 families (GLM, Llama, Qwen, DeepSeek, MiniMax, Granite) and sizes ranging from 1B to 480B parameters.
• Variables:
  • Context Lengths: 32K, 128K, and 200K tokens.
  • Temperatures: 0.0, 0.4, 0.7, and 1.0 (8 runs per configuration).
  • Hardware: NVIDIA H200, AMD MI300X, and Intel Gaudi3.

4. Key Findings & Results

A. Hallucination Rates are Non-Trivial and Context-Dependent

• The "Floor" is Non-Zero: Even the best-performing model (GLM 4.5) at 32K context fabricates answers 1.19% of the time. Top-tier models generally sit at 5–7%, while the median model hallucinates roughly 25% of the time on trap questions.
• Context Length Degradation: Fabrication rates rise steeply with context length.
  • At 128K, the best fabrication rate rises to ~3.2%, and the median increases significantly.
  • At 200K, no model stays below a 10% fabrication rate. The best model (Qwen3 Next 80B-A3B) still hallucinates 10.25% of the time.
  • Some models (e.g., GLM 4.6) suffer "catastrophic collapse," jumping from ~7% fabrication at 32K to 71.6% at 200K.

B. Model Selection Dominates All Other Factors

• Family over Size: Model family is a stronger predictor of fabrication resistance than parameter count. The GLM and MiniMax families consistently show low fabrication rates across sizes, whereas the Llama 3.x family exhibits high fabrication rates regardless of size (e.g., Llama 3.1 70B hallucinates at 49.5%, similar to the 8B variant).
• Accuracy Gap: There is a 72-percentage-point gap in overall accuracy between the best and worst models at 32K. No amount of temperature tuning can bridge this gap.

C. Grounding and Fabrication are Decoupled

• Distinct Capabilities: A model can be excellent at finding existing facts (Grounding) but terrible at refusing to answer non-existent ones (Fabrication).
• Example: Llama 3.1 70B has a 90.18% grounding score but a 49.50% fabrication rate. It is highly skilled at retrieval but equally willing to invent facts. This decoupling implies that standard retrieval benchmarks are insufficient for assessing trustworthiness.

D. Temperature Effects: Nuance over Dogma

• Accuracy: Setting temperature to 0.0 yields the best overall accuracy in only 60% of cases. For Grounding and Fabrication specifically, higher temperatures (0.4–1.0) often produce better results (reducing fabrication in 53% of cases).
• Coherence Loss (Infinite Loops): This is a critical finding. T=0.0 dramatically increases the risk of infinite generation loops, especially at long context lengths.
  • At 200K, T=0.0 can cause coherence loss rates 48x higher than T=1.0 (e.g., GLM 4.7).
  • Practitioners must balance the slight accuracy gain of T=0.0 against the massive risk of query failure.

E. Hardware Independence

• Consistency: Results across NVIDIA H200, AMD MI300X, and Intel Gaudi3 were statistically indistinguishable. Hardware choice does not meaningfully affect model fidelity or hallucination rates, provided the serving framework (vLLM) is consistent.

5. Significance and Implications

• Deployment Reality Check: Enterprise systems cannot rely on LLMs for 100% factual accuracy. Safeguards (e.g., citation verification, human-in-the-loop) are mandatory, as even the best models hallucinate ~1–7% of the time under optimal conditions.
• Evaluation Reform: Current benchmarks focusing solely on "retrieval" or "grounding" are dangerous. They miss the critical "fabrication" capability. Evaluation must include negative constraints (questions about non-existent entities) to measure trustworthiness.
• Context Limits: Advertised context windows (e.g., 200K) often exceed usable capacity. Performance degrades significantly before the token limit is reached, and fabrication rates explode.
• Configuration Strategy:
  • Model Choice: Prioritize model families with proven low fabrication (e.g., GLM, MiniMax) over simply choosing the largest model.
  • Temperature: Avoid blanket T=0.0 policies. A moderate temperature (0.4 or 0.7) often offers a better trade-off between accuracy, fabrication reduction, and preventing infinite loops.
  • Testing: Always test at the actual deployment context length, not just short benchmarks.

Conclusion

The study concludes that hallucination in document Q&A is substantial and unavoidable. While model selection and context management are the primary levers for mitigation, the fundamental decoupling of grounding and fabrication resistance suggests that reducing hallucinations requires targeted training interventions rather than just scaling model size. The authors release all 172 billion tokens of data to support reproducibility and future benchmarking.