💬 NLP

Parametric Knowledge is Not All You Need: Toward Honest Large Language Models via Retrieval of Pretraining Data

This paper addresses the issue of LLM hallucinations by proposing a robust evaluation benchmark that accounts for pretraining data and introducing a novel method to enhance model honesty through the retrieval of that pretraining data.

Original authors: Christopher Adrian Kusuma, Muhammad Reza Qorib, Hwee Tou Ng

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christopher Adrian Kusuma, Muhammad Reza Qorib, Hwee Tou Ng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a brilliant but overconfident student named "LLM" who has read billions of books during their training. This student is great at answering questions, but they have a major flaw: they don't know what they don't know. When asked a question about a topic they've never seen, instead of saying, "I've never heard of that," they confidently make up a story. In the world of AI, this is called hallucination.

The paper you shared argues that to make AI trustworthy, we need to teach it the difference between "I know this" and "I don't know this." Here is a breakdown of their solution, using simple analogies.

The Problem: The "Black Box" Blind Spot

Usually, when researchers try to test if an AI is honest, they treat the AI like a black box. They ask it questions and see what it says, but they don't know exactly what books the AI read to learn its facts.

The Flaw: If the AI gets a question wrong, researchers assume the AI "doesn't know" the answer. But maybe the AI did read the answer in its training books; it just forgot or got confused.
The Result: Previous tests were unfair because they couldn't tell if the AI was lying (hallucinating) or just having a bad day (forgetting).

The Solution: The "Open Book" Approach

The authors used a special, open-source AI model called Pythia. The key advantage? We have the entire library of books this AI read.

The Analogy: Imagine giving the student a specific list of every single book they studied. Now, when you ask a question, you can actually check the library to see: "Did this student read the answer to this question?"
The New Benchmark: They created a new test (called TIP-TRIVIAQA) where they know exactly which questions the AI should know (because the answer is in its training books) and which it shouldn't (because the answer is nowhere in its books). This creates a fair playing field to measure honesty.

The Method: The "Librarian" System (RETAIN)

To fix the AI's honesty, the authors didn't just tell it to "be honest." They built a three-part system called RETAIN that acts like a smart librarian:

The Retriever (The Searcher):
When you ask a question, this agent immediately goes into the AI's "library" (its pretraining data) to look for the answer. It's like a librarian pulling books off the shelf to see if the answer is there.
The Answerability Classifier (The Gatekeeper):
This agent looks at the books the librarian found. It asks: "Does this book actually contain the answer?"
- If Yes: It passes the book to the next agent.
- If No: It stops the process and tells the AI, "We don't have the answer in our library. Say 'I don't know'."
The Responder (The Speaker):
This is the AI that actually talks to you. It only speaks if the Gatekeeper gives it the green light and a relevant book to read from. If the Gatekeeper says "No," the Responder simply says, "I don't know."

Why This Works

The paper found that this method is much better than previous tricks.

Old Way: Just telling the AI "If you aren't sure, say 'I don't know'" didn't work well. The AI still tried to guess.
RETAIN Way: By physically retrieving the source material first, the AI has a "reality check." It can see, "Oh, the answer isn't in my books," so it honestly admits it doesn't know.
Bonus: When the answer is in the books, the AI gives a much more accurate answer because it's reading the source material right then, rather than relying on its memory alone.

The Results

When they tested this new system:

It was much better at saying "I don't know" when the answer wasn't in its training data.
It was much better at answering correctly when the answer was in the data.
It outperformed all other methods they tested, proving that giving the AI access to its own "source code" (the books it read) is the key to making it honest.

In short: The paper shows that to stop AI from making things up, you shouldn't just tell it to be honest. Instead, you should give it a tool to check its own library first. If the answer isn't there, it should be honest enough to say, "I haven't read that."

Technical Summary: Parametric Knowledge is Not All You Need

Problem Definition
Large Language Models (LLMs) frequently suffer from "hallucinations," generating factually incorrect responses because they lack awareness of their own knowledge boundaries. Current methods to improve LLM "honesty"—defined as the ability to refuse answering questions outside one's knowledge (self-knowledge) and to answer correctly when knowledge exists (self-expression)—face significant evaluation challenges. Existing benchmarks often treat LLMs as black boxes, estimating knowledge boundaries indirectly. This leads to flawed evaluations where a model's failure to answer is attributed to a lack of knowledge (a self-knowledge issue) when it may actually be an inconsistency in expression (a self-expression issue). Furthermore, without access to the specific training data, it is impossible to definitively determine if a model should know an answer, making robust benchmarking difficult.

Methodology
The authors propose a two-pronged approach: a new evaluation benchmark and a novel inference-time method called RETAIN (Retrieval-Enhanced Training-Aware Inference).

Benchmark Construction (TIP-TRIVIAQA):
- Leveraging Pythia, a truly open LLM with publicly available pretraining data, the authors construct a benchmark where the "ground truth" of knowledge is known.
- They utilize a two-stage search process to determine if a question is "answerable" (the answer exists in the pretraining corpus) or "unanswerable" (it does not):
  - Token-Based Search: Retrieves documents containing specific keywords from the question and gold answer. An LLM judges relevance.
  - Vector-Based Search: For questions not resolved by token search, semantic similarity is used to retrieve documents, followed by relevance judgment.
- This creates a dataset (TIP-TRIVIAQA) where questions are explicitly categorized based on the model's actual exposure to the information during pretraining.
The RETAIN Method:
- RETAIN is an agentic framework comprising three components:
  - Retriever: Encodes the input question and retrieves top- $k$ documents from the model's own pretraining corpus using dense retrieval.
  - Answerability Classifier: A fine-tuned model that determines if the retrieved documents contain sufficient information to answer the question. If no relevant document is found, the system triggers a refusal ("I don't know").
  - Responder: A fine-tuned model that generates the gold answer using the retrieved document as context, but only if the classifier deems the question answerable.

Key Contributions

A Robust Benchmark: The paper introduces the first benchmark for LLM honesty that explicitly accounts for the model's knowledge boundary by identifying relevant documents within its pretraining data, moving beyond black-box evaluation.
A Novel Inference Method: The authors propose RETAIN, which leverages retrieval over the model's own pretraining corpus at inference time. This allows the model to distinguish between questions it has seen (and can answer) and those it has not (and should refuse).
Empirical Validation: The study demonstrates that simply augmenting prompts with documents from the pretraining data improves both answer accuracy and the model's ability to recognize its limitations.

Results
Experiments conducted on the TIP-TRIVIAQA benchmark using Pythia-12b-deduped show that RETAIN significantly outperforms existing baselines (including SFT, Best-of-N, DPO, and R-Tuning).

Performance: RETAIN achieves an ExactMatch-F1 score of 58.57 and a PartialMatch-F1 of 62.23, substantially higher than the next best baseline (R-Tuning at 44.16 PM-F1).
Generalization: On the HoneSet dataset (containing questions humans cannot answer), RETAIN achieves an 87.63% refusal rate, significantly outperforming all baselines, indicating the method generalizes beyond the specific benchmark.
Ablation: The results confirm that the combination of retrieval, answerability classification, and a trained responder is necessary for optimal performance; removing any component degrades the F1 score.

Significance
The paper claims to be the first to leverage pretraining data to construct an LLM honesty benchmark and to perform retrieval over a model's own pretraining corpus for question answering. The authors argue that parametric knowledge alone is insufficient for honesty; instead, models require a mechanism to access and verify their training data to accurately perceive their knowledge limits. By utilizing the transparency of open models like Pythia, this work provides a pathway toward more reliable, interpretable, and honest AI assistants that can explicitly acknowledge when they do not know an answer, thereby reducing hallucinations in critical domains.