💬 NLP

Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

This paper introduces CitePretrain, a retrieval-free framework that enhances Large Language Models' ability to verifiably attribute answers to their pretraining sources through a two-stage process featuring "Active Indexing," which significantly improves citation precision and robustness compared to passive baselines.

Original authors: Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly unreliable student named LLM (Large Language Model). This student has read millions of books, articles, and websites. They are incredibly smart and can answer almost any question you ask.

However, there's a problem: LLM loves to make things up.

If you ask, "Who wrote Pride and Prejudice?" they might say, "Jane Austen, according to The Daily Star," even though they made up the newspaper name. This is called hallucination.

Currently, to fix this, we usually attach a search engine to the student. When they answer, the search engine goes out, finds the real book, and says, "Here is the proof!" But this is slow, expensive, and sometimes the search engine brings back the wrong book or gets confused.

This paper introduces a new way to train the student so they can cite their own memory without needing a search engine. They call it CitePretrain.

Here is the simple breakdown of how it works, using some fun analogies:

1. The Problem: The "Passive" Student

Imagine you give the student a stack of 10,000 books. You tell them, "Read these."

The Old Way (Passive Indexing): You just tape a sticky note with the book's title on the back cover. You hope the student memorizes that "Book A" is on "Page 100."
The Result: The student is okay at memorizing exact quotes. But if you ask, "What was the main idea of Book A?" they might get the idea right but forget which book it came from, or they might mix it up with Book B. They are too passive; they just copy the title without really understanding the connection.

2. The Solution: The "Active" Student (Active Indexing)

The authors propose a new training method called Active Indexing. Instead of just handing the student the books, they turn the library into a giant, interactive scavenger hunt.

They use two main tricks to train the student:

Trick A: The "Source-to-Fact" Drill (Forward)

Imagine a game where the teacher points to a specific book (e.g., The History of Space) and says, "Tell me everything you know about rockets from this specific book."

Why it works: The student learns to open that specific mental "file" and pull out facts only from that source. They learn to say, "According to The History of Space, rockets use liquid fuel."

Trick B: The "Fact-to-Source" Drill (Backward)

Now, the teacher asks a question: "How do rockets work?"
The student answers, "They use liquid fuel."
Then the teacher asks, "Which book did you just read that in?"

Why it works: This forces the student to work backward. They can't just guess; they have to trace their thought process back to the specific document. It's like a detective solving a case: "I know the answer, so which file cabinet must I have pulled it from?"

3. The "Synthetic" Library

To make this training super effective, the authors didn't just use the original books. They used AI to create millions of new practice questions based on those books.

They took one fact and asked it in 50 different ways (e.g., "Who built the White House?" vs. "When was the White House constructed?").
They mixed facts from different books to create complex questions.
The Analogy: It's like a personal trainer who doesn't just make you lift a weight once. They make you lift it with different grips, different speeds, and different weights, so your muscles (the model's memory) get strong enough to handle any situation.

4. The Results: A Super-Reliable Student

When they tested this new "Active Student" (using models like Qwen):

Accuracy: They got the facts right.
Citations: They correctly pointed to the exact book title for every fact.
Improvement: They were up to 30% better at citing correctly than the old "Passive" students.
Bonus: Even when the search engine (external retrieval) failed or gave bad info, this student could still give a correct answer with a citation because their internal memory was so well-organized.

The Big Picture Takeaway

Think of the old way as a Librarian who has to run to the shelves every time you ask a question. It's slow and they might grab the wrong book.

This new method turns the Librarian into a Walking Encyclopedia. They have memorized not just the content of the books, but exactly which book every fact came from. They can answer you instantly, without running to the shelves, and they can point to the exact shelf location (the citation) to prove they aren't making it up.

Why does this matter?

Speed: No waiting for a search engine.
Trust: You can verify their answers immediately.
Safety: If the internet goes down or the search engine is broken, this model still works and tells you where its info came from.

In short, the paper teaches AI models to be honest scholars who know exactly where their knowledge comes from, rather than confident liars who guess and hope for the best.

1. Problem Statement

Large Language Models (LLMs) often generate hallucinated or unverifiable citations when asked to provide sources for their answers. Current solutions rely on Retrieval-Augmented Generation (RAG), where an external retriever fetches documents at inference time to ground the model's output. While effective, RAG introduces significant latency, infrastructure dependencies, and vulnerability to retrieval noise (e.g., missing or conflicting documents). Furthermore, RAG does not reveal what the model internally knows, limiting explainability.

The core problem addressed is: Can LLMs be trained to reliably attribute answers to documents seen during pretraining without relying on external retrieval at inference time?
Existing attempts (e.g., simply appending document IDs to text) fail because they rely on verbatim memorization. Real-world facts are often paraphrased or synthesized across multiple documents, and simple ID appending does not teach the model to associate non-verbatim facts with their correct source identifiers.

2. Methodology

The authors propose a two-stage training framework and a novel data augmentation strategy called Active Indexing.

A. Benchmark: CitePretrain-Bench

To evaluate this capability, the authors constructed a benchmark mixing:

Real-world corpora: Wikipedia, Common Crawl, and arXiv papers.
Novel documents: Unseen synthetic documents (RepliQA) to test the ability to learn and cite new knowledge.
Tasks: Both short-form (single fact) and long-form (multi-fact synthesis) QA tasks requiring citations.

B. Training Framework

Continual Pretraining (Index Learning): The model learns to map factual content to unique document identifiers (titles).
Instruction Tuning: The model learns to generate answers with citations in a closed-book setting.

C. Active Indexing Strategy

The paper introduces Active Indexing, which moves beyond "Passive Indexing" (merely appending IDs to text). Active Indexing uses synthetic data augmentation to create bidirectional, generalizable bindings between facts and sources:

Forward Augmentation (Source $\to$ Fact):
- Goal: Teach the model to recall facts given a document ID.
- Method: Extract entities from a document and generate diverse Question-Answer (QA) pairs conditioned on the document title. This forces the model to internalize the mapping $ID \to \text{Facts}$ and handle varied linguistic formulations of the same fact.
Backward Augmentation (Fact $\to$ Source):
- Goal: Teach the model to attribute a fact to its source.
- Method: Create clusters of related chunks from different documents. Generate QA pairs that require synthesizing information across these chunks. The model must generate the answer and explicitly cite the specific document titles that support each claim. This trains the $Fact \to ID$ mapping and cross-document reasoning.

3. Key Contributions

CitePretrain-Bench: A comprehensive benchmark for evaluating internal citation capabilities across diverse domains (Wikipedia, Web, Scientific) and task types (short/long-form), including unseen documents.
Active Indexing: A novel continual pretraining technique that uses bidirectional synthetic data augmentation (Forward and Backward) to teach models to associate facts with document identifiers robustly, even when facts are paraphrased or synthesized.
Retrieval-Free Attribution: Demonstrating that LLMs can achieve high citation precision without external retrieval, serving as a robust fallback when retrieval is unavailable or noisy.
Hybrid Citation System: A framework combining internal citations (parametric memory) with external retrieval, showing that the two are complementary and that a hybrid approach outperforms either method alone across varying retrieval qualities.

4. Experimental Results

Experiments were conducted on Qwen-2.5 (3B, 7B, 14B) and Llama-3 models.

Performance Gains: Active Indexing significantly outperforms Passive Indexing and Instruction-only baselines.
- Citation Precision: Achieved gains of up to 30.2% (e.g., on RepliQA, precision jumped from ~2.4% in Passive Indexing to ~32.6% with Active Indexing on Qwen-7B).
- Answer Correctness: The method also improved factual accuracy, likely due to exposure to diverse fact formulations.
Ablation Studies:
- Bidirectionality: Combining Forward and Backward augmentation yields the best results; they are complementary.
- Scaling: Performance continues to improve as the amount of augmented data increases, showing no saturation even at 16 $\times$ the original token count.
- Model Size: Larger models (7B $\to$ 14B) show monotonic gains, but even proprietary models like GPT-4.1 struggle with internal citation without this specific training.
Robustness:
- Semantic Shortcuts: The model does not merely "guess" titles based on semantic similarity; it learns genuine fact-to-ID associations, performing well even on "Hard" examples where titles are not semantically distinctive.
- Retrieval Noise: In hybrid settings, internal citations act as a critical fallback when external retrieval is sparse or noisy, while external retrieval helps when internal knowledge is incomplete.

5. Significance and Implications

Trustworthiness & Explainability: This work provides a pathway to "traceable" AI, where outputs can be linked back to training data without external tools, aligning with regulatory requirements (e.g., EU AI Act) for transparency.
Efficiency: It eliminates the inference-time overhead of retrieval systems, making citation generation faster and infrastructure-independent.
Paradigm Shift: It challenges the reliance on RAG for all grounding tasks, suggesting that with the right training (Active Indexing), models can internalize the "retrieval" function, turning complex RAG pipelines into unified, end-to-end models.
Future Directions: The paper highlights the potential for hybrid systems that dynamically switch between internal memory and external retrieval based on confidence, offering a robust solution for real-world deployment.

In summary, CitePretrain demonstrates that by explicitly training models to associate facts with document identifiers through diverse, bidirectional synthetic data, LLMs can reliably generate verifiable citations from their parametric memory, reducing hallucination and dependency on external infrastructure.