💬 NLP

Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

This paper demonstrates that for closed-book question answering, internalizing documents into LoRA adapters via high-quality data curation (such as canonicalizing answers and removing trivia) is more critical to performance than architectural capacity or hyperparameter tuning, ultimately enabling a 4-bit Gemma-4 model to outperform traditional retrieval-based baselines.

Original authors: Joan Figuerola Hurtado

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Joan Figuerola Hurtado

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who knows a lot about the world, but if you ask it about a specific, brand-new book you just bought, it has no idea what's inside. Usually, to answer questions about that book, we have to use a "search engine" method: we find the right page, copy the text, and paste it into the robot's prompt so it can read it and answer. This is called Retrieval-Augmented Generation (RAG). It works, but it's like having to carry a heavy library card catalog for every single question you ask. It's slow, and sometimes the search engine grabs the wrong page.

What if, instead of searching, we could teach the robot to memorize the book so well that it could answer questions without ever looking at the text again? This is called "internalizing" knowledge. The paper you are about to read explores exactly this: taking a specific set of documents and baking them directly into the robot's brain using a special, lightweight training technique called LoRA. The big question is: how do we do this effectively? Is it about making the robot's brain bigger? Is it about using a smarter search algorithm? Or is it something much simpler?

The Great Memory Bake-Off

This paper is a deep dive into a specific corner of artificial intelligence where researchers try to teach a language model to answer questions about a specific collection of documents without ever seeing those documents during the actual test. Think of it like a student taking a "closed-book" exam. They can't look at their notes; they have to rely entirely on what they studied and memorized beforehand.

The researchers used a model called Gemma-4-e4b (a 4-bit version, which is like a compressed, efficient version of a larger brain) and tried to "bake" the knowledge of up to 99 documents directly into its weights using LoRA adapters. These adapters are like little, removable training modules that teach the model new facts without rewriting its entire brain. The goal was to see if this "internalized" memory could beat the traditional method of searching for the answer (RAG) in terms of speed and accuracy.

The Big Surprise: It's Not About the Size of the Brain

The researchers ran about 100 different training experiments, trying everything from tiny training sets (one document) to larger ones (99 documents). They expected that the key to success would be "capacity"—basically, making the training module bigger or more complex to hold more information. They thought, "If we have more documents, we just need a bigger brain to hold them, right?"

They were wrong.

The paper found that once the training module (the LoRA adapter) was big enough to hold the information, making it even bigger didn't help much. The real secret sauce wasn't the size of the brain or the complexity of the architecture. It was the quality of the study notes.

Here is the most vivid finding: The researchers took a dataset of 15 documents. When they used messy, long-winded, trivia-filled study notes generated by an AI, the robot's closed-book accuracy was a disappointing 57.7%. It was like studying for a test using a textbook that had too many footnotes and irrelevant jokes.

But then, they did a simple "curation pass." They went through the data and did two things:

They shortened the correct answers to be short, punchy phrases (1–6 words).
They threw out all the trivia and random facts that didn't matter.

Suddenly, the accuracy skyrocketed to 85.7%. That is a 28-point jump just by cleaning up the data. This single change was more powerful than any fancy architectural tweak or increasing the training size. The paper argues that if your study notes are cluttered, no amount of "brain power" will save you.

The "Too Big to Fit" Problem

While data quality was the star, the paper did find one rule about size: The training module must grow as the library grows.

When they tried to fit 50 documents into a module designed for 25, the performance crashed. It was like trying to stuff 50 suitcases into a backpack meant for 25; everything got jumbled, and the robot started mixing up facts (a phenomenon called "mis-binding"). To fix this, they had to double the size of the training module (the "rank").

However, they also discovered a tricky side effect: when they made the module bigger, they had to be more gentle with how they taught it. If they used the same "teaching speed" (learning rate) for a huge module as they did for a small one, the robot forgot everything. They had to slow down the teaching process to get the big module to learn properly. This was a common mistake they almost made, thinking the module was just "too full" when really, they were just teaching it too fast.

Beating the Search Engine

Finally, the researchers wanted to know: Is this "memorization" approach actually better than the standard "search and read" method?

They set up a real-world race on a 15-document corpus.

The Search Engine (BM25-RAG): This method found the right page only 53% of the time on the first try. Even when it found the right page, the robot only got the answer right 58.9% of the time.
The "Perfect" Search (Oracle): Even if they gave the robot the exact right page to read (a realistic best-case scenario), it only got the answer right 65.6% of the time.
The Internalized Robot: The robot that had memorized the documents directly got the answer right 84.2% of the time.

Not only was the memorized robot more accurate, but it was also much faster. The search method took nearly 3 seconds per question, while the internalized robot took between 0.3 and 0.8 seconds. It was like comparing a librarian who has to walk to the stacks, find a book, and read it, versus a person who has the book's contents already memorized in their head.

The Takeaway

This paper is a case study in debugging how we train AI. The authors admit they made mistakes along the way, initially blaming the size of the model or the length of the text when the real culprit was messy data.

The main lesson is clear: Data quality is king. If you want an AI to remember a specific set of documents without needing to search for them, don't just throw more computing power at the problem. Instead, clean up your training data. Make the answers short, remove the fluff, and ensure the facts are clear. Once the data is clean and the training module is big enough, the robot can learn the material better than it can find it in a library.

While the paper notes that this is based on a single "seed" (one specific random starting point) and smaller datasets (up to 99 documents), the results are a strong signal that for closed-book tasks, a well-curated, simple approach beats complex, messy ones every time.

Technical Summary: Data Quality over Capacity in Closed-Book QA via LoRA

Problem Statement

The standard approach for enabling Large Language Models (LLMs) to utilize untrained corpora is Retrieval-Augmented Generation (RAG), which retrieves relevant document chunks and injects them into the prompt. However, RAG introduces retrieval as a failure mode, consumes context-window budget for every query, and caps the amount of information informing a single answer.

This paper investigates an alternative: knowledge internalization. The goal is to bake a document corpus directly into the weights of a model using a lightweight LoRA (Low-Rank Adaptation) adapter. This enables "closed-book" question answering where the model answers questions about the corpus without any document text in the prompt, eliminating retrieval steps and context limits. The study specifically addresses how to scale this from single documents to a corpus of nearly 100 documents on a 4-bit quantized Gemma-4-e4b model.

Methodology

Task and Data Generation

The authors generate closed-book question-answer pairs where gold answers are grounded verbatim in the source documents.

Data Source: Training data is self-generated by a strong LLM reading extracted chunks of the target documents. No dedicated QA-teacher model is used.
Structure: Each claim includes a verbatim source quote, 8 training paraphrase-questions, and 2 held-out questions.
Training Recipe: The base model is gemma-4-e4b-it-4bit. LoRA is applied only to MLP projections across all 42 layers. Training occurs on Apple Silicon via MLX in two stages resumed into a single adapter:
1. Stage A (CPT-like): Completion-loss training on raw chunks and multi-level paraphrases.
2. Stage B (SFT): Closed-book QA supervision with answer-token loss only.

Evaluation Metrics

Correctness: Defined by exact substring match (either direction) or token-F1 $\ge$ 0.74.
Reporting: Recall is reported separately from overall accuracy. Overall accuracy includes an "abstention" slice for questions about facts never seen during training.
Baseline: A real BM25-RAG pipeline (retriever + base reader) and a "realistic oracle" (feeding the true gold chunk to the base reader) are established on a 15-document slice.

Key Results

1. Data Quality is the Dominant Lever

The most significant finding is that once adapter capacity is adequate, the quality of training data outweighs architectural changes, rank, or learning rate tuning.

The Curation Effect: On a 15-document corpus, switching from self-generated claims (which over-mined high-entropy trivia and produced long answer clauses) to a curated dataset (shortening gold answers to 1–6 word canonical spans and dropping trivia) increased closed-book recall from 55.6% to 84.2% and overall accuracy from 57.7% to 85.7%.
Comparison to Architecture: This +28 point gain from data curation exceeded any architectural intervention tested, including EntiGraph-style synthetic pretraining and trainable KV-cache cartridges (which performed poorly).
Verification: Re-scoring predictions under stricter rules confirmed the gain was not a scoring artifact; the benefit of shorter answers actually widened under exact match criteria.

2. Capacity Scaling and Learning Rate Coupling

The study identifies a trend where adapter rank must grow with corpus size, but this is entangled with learning rate requirements.

Capacity Saturation: On a nested corpus (15 $\to$ 25 $\to$ 50 $\to$ 100 documents), a fixed rank (64) caused recall to collapse (from 71% at 25 docs to 50% at 50 docs). Doubling the rank to 128 restored performance.
Learning Rate Artifact: When scaling to 100 documents with rank 256, reusing the learning rate from rank 128 caused a regression in recall. The authors initially misdiagnosed this as a sequence-length issue or capacity limit. Direct measurement revealed the true cause: larger ranks require gentler learning rates. Halving the learning rate (4e-5 $\to$ 2e-5) recovered the loss floor and improved recall by 6 points.

3. Internalization vs. Retrieval

On the 15-document corpus, the internalized adapter significantly outperformed retrieval baselines:

Internalized Adapter: 84.2% recall.
BM25-RAG: 58.9% recall (BM25 retrieves the gold chunk only 53% of the time at rank 1).
Realistic Oracle: Even when the base reader was given the true gold chunk (oracle), it only achieved 65.6% recall.
Latency: The internalized adapter operated at a fraction of the latency (300–800 ms) compared to the RAG pipeline (~2,895 ms).

Contributions and Significance

The paper frames its findings as a case study in empirically debugging LLM training, highlighting three specific misdiagnoses corrected during the process (confusing learning rate issues with capacity limits, sequence length issues, and data quality issues).

Primary Contributions:

Data Curation Superiority: Demonstrated that a single curation pass (shortening answers, removing trivia) yields a larger performance gain (+28 points) than complex architectural changes or capacity increases.
Scaling Laws: Established a "corpus-scaling ladder" showing that rank must grow with corpus size, and crucially, that larger ranks necessitate lower learning rates to avoid performance regression.
Feasibility of Internalization: Provided a real-world baseline showing that internalization can outperform both standard RAG and "oracle" RAG on accuracy and latency for small-to-medium corpora, provided data quality is high.

Limitations and Threats to Validity:
The authors explicitly note several constraints:

Single-Seed: All experiments use seed 42; sub-2-point differences may be noise, and multi-seed replication is needed for load-bearing deltas.
Metric Bias: The deterministic scorer favors the style-matched adapter over paraphrasing RAG readers, meaning RAG accuracy is a lower bound on its knowledge.
Scale: The 99-document corpus is far below the target of tens of thousands of documents; results regarding specific rank/learning rate constants may not extrapolate linearly.
Quantization: The use of a 4-bit base model may set a loss floor attributed to data quality in higher-precision settings.

The paper concludes that while capacity acts as a "hard gate" below which no data intervention helps, once adequate capacity is reached, training-data quality (specifically answer brevity and trivia removal) is the most reliable and impactful lever for closed-book internalization.