💬 NLP

Can a Language Model Learn Facts Continually in Its Weights?

This paper demonstrates that while broad training data enables language models to store new facts in their weights more effectively than bare statements, continual weight-based learning inevitably causes catastrophic forgetting and interference, making context the only reliable channel for preserving and composing knowledge across sequential updates.

Original authors: Charles O'Neill

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Charles O'Neill

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who knows a lot of things. You want to teach it new, made-up facts every day—like "Zorvathine is a metal that melts when it gets cold." The big question is: Can we write these facts directly into the robot's brain (its "weights") so it remembers them forever, even after we teach it a hundred more things?

This paper tries to answer that by playing a game of "write, forget, and check" with a model called Qwen3. Here is what they found, using some fun analogies.

The Two Ways to Teach a Robot

The researchers discovered that how you teach the robot matters more than how many times you repeat the lesson.

The "Bare Statement" Method (The Bad Way): Imagine you just whisper the fact to the robot once: "Zorvathine melts below -10°C." You do this over and over. The robot gets really good at reciting that exact sentence. It's like a parrot that can repeat a phrase perfectly but doesn't understand what it means.
The "Study" Method (The Good Way): Imagine you teach the robot using a whole textbook. You show it the fact, then ask it questions, give it different ways to say the same thing, and show it how the fact contradicts what it usually thinks. The robot learns the idea behind the fact, not just the words.

The Result:
When the robot learned via the "Study" method, it could actually use the fact to solve new puzzles. When it learned via the "Bare Statement" method, it could only recite the sentence. If you asked it a tricky question like, "What happens if I cool Zorvathine?", the "Study" robot said, "It melts!" The "Bare Statement" robot just stared blankly or gave the wrong answer, even though it had memorized the sentence perfectly.

The "Overwriting" Problem: The Library of Lost Books

Now, here is the tricky part. The researchers taught the robot 20, then 100 new facts, one after another. They wanted to see if the old facts survived.

The "Bare Statement" Facts: These vanished almost instantly. After just 20 new lessons, the robot forgot 99% of them. It was like writing a note on a sticky pad and then immediately sticking a new, bigger note right on top of it. The old note is still there physically, but you can't see it anymore.
The "Study" Facts: These were tougher. After 20 new lessons, about 46% of them were still usable. After 100 lessons, they settled at a plateau of 25–28%. They didn't disappear completely, but they got harder to find.

The Big Surprise:
The researchers checked the robot's brain (its "weights") even after it seemed to have forgotten a fact. They found that the "memory" of the fact was still there! The robot hadn't erased the data; it just lost the address to find it.

Think of it like a library. The books (facts) are still on the shelves. But the librarian (the robot) has been given a new map that points to the newest books. When you ask for an old book, the librarian looks at the new map, sees the old book's spot is now labeled "Newest Fact," and hands you the wrong book. The old book is still there, but the robot can't reach it anymore.

Why Can't We Just "Fix" the Address?

The researchers tried many clever tricks to help the robot remember the old facts while learning new ones:

They tried to "penalize" the robot if it forgot.
They tried to "project" the new lessons away from the old ones.
They tried to use a "frozen teacher" (a perfect copy of the robot before it started learning) to guide the new lessons.

What Worked:
Using a frozen teacher helped the robot keep its general smarts (like math or logic) from getting worse. It kept the robot from going crazy. But it did not fix the problem of finding the old facts. The old facts were still lost in the shuffle.

What Didn't Work:
No matter how carefully they measured the "conflict" between new and old facts, they couldn't stop the new lessons from blocking the old ones. The paper suggests that local measurements (looking at just the next step) can predict what happens immediately, but they fail to predict what happens after a long series of lessons. It's like trying to predict a traffic jam by looking at the car right in front of you; you can't see the accident happening three miles down the road.

The Final Verdict: Context is King

So, can we write facts into a robot's brain and keep them forever?

If you want the robot to use the fact: You must teach it with broad, diverse examples (the "Study" method), not just simple repetition.
If you want the fact to survive 100 new lessons: The paper suggests that writing it into the brain is the wrong tool. Even the best methods lose more than half the facts after 20 updates.

The most reliable way to keep a fact usable is to give it to the robot in the prompt (the "context").
Imagine the robot has a whiteboard next to it. If you write the fact on the whiteboard every time you ask a question, the robot can use it perfectly, even after learning 100 other things. The whiteboard doesn't get overwritten; it just gets updated.

The Bottom Line:
The robot's brain is great at storing content, but it's terrible at creating a permanent "address" for that content that survives new updates. If you need a fact to be composed, combined with other knowledge, or remembered after many new lessons, keep it in the context (the prompt), not in the weights (the brain). The brain stores the what, but the context provides the where.

Technical Summary: Can a Language Model Learn Facts Continually in Its Weights?

Problem Statement

Continual learning aims to enable language models (LMs) to acquire new facts after initial training and retain them through subsequent updates, effectively writing knowledge directly into model weights. While context-based learning (in-context learning) allows immediate fact usage, it is transient. The central question addressed by this work is whether weight-based continual learning can support the accumulation of usable knowledge. Specifically, the paper investigates three critical requirements for successful continual writing:

Creation: Does each write create knowledge that is usable beyond simple recitation?
Preservation: Do general abilities survive the accumulation of updates?
Reachability: Can earlier facts remain reachable after later writes?

Previous observations suggest that fine-tuning learns facts slowly, often fails at multi-hop reasoning or reversals compared to in-context usage, and suffers from distributional drift. However, a unified account of why some writes produce usable knowledge, why some survive, and the nature of "forgotten" facts has been lacking.

Methodology

The study utilizes Qwen3-4B models (with LoRA adapters and full fine-tuning variants) and a controlled evaluation framework involving 247 invented facts about non-existent entities (e.g., "Zorvathine is a metal that melts when cooled below −10°C"). These facts contradict the model's priors to ensure answers rely on the new knowledge.

Evaluation Design

Five Question Types: To distinguish between recitation and genuine use, facts are tested via:
1. Recall: Direct statement of the fact.
2. Paraphrase: Rephrasing the fact.
3. Application: One-step use of the fact.
4. Composition: Combining the fact with unstated world knowledge.
5. Counterfactual: Choosing the written fact over the model's default.
Anchors: Performance is measured against a floor (original model without the fact) and a ceiling (original model with the fact in the prompt).
Dual Grading Policy: Answers are scored under a strict policy (requiring the specific conclusion) and a lenient policy (crediting facts that entail the conclusion). The difference, termed the entailment gap, measures the "recitation-to-use" gap.
Continual Protocol: Models undergo sequential writes (20 to 100 facts) where adapters are merged after each fact. Retention is measured by re-evaluating earlier facts after subsequent writes.

Experimental Conditions

The study varies:

Training Data Breadth: "Bare-statement" (fact in trivial framings) vs. "Study" (24 diverse paraphrases, Q&A, implications, and contrasts).
Objectives: Supervised Fine-Tuning (SFT), Offline Context Distillation, and Online Context Distillation (Forward/Reverse KL).
Update Methods: LoRA vs. Full Fine-Tuning.
Teachers: Frozen original model vs. accumulated (moving) model for distillation.

Key Contributions and Results

1. Training Data Breadth Determines Knowledge Usability

The breadth of training data is the primary causal factor in creating usable knowledge.

Recitation vs. Use: Bare-statement training produces high recitation (97% recall) but fails at application and counterfactuals (21–23% accuracy). It leaves a large entailment gap (27.4 points), indicating the model memorizes the statement but cannot derive conclusions.
Broad Data Effect: "Study" training (diverse restatements) reduces the entailment gap to 5.4 points without explicitly showing conclusions. It significantly improves application (+21 points), composition (+18 points), and counterfactual use (+29 points) compared to bare-statement training.
Scale Independence: This effect holds across model scales (4B and 8B) and update methods (LoRA and Full FT).

2. Knowledge Type Predicts Survival in Continual Writing

The nature of the written knowledge dictates its survival rate under sequential updates.

Retention Disparity: After 20 sequential writes, facts written via bare-statement training retain only 1% accuracy, whereas study-trained facts retain 46%.
Correlation: There is a strong negative correlation ( $\rho = -0.526$ ) between the initial entailment gap and retention; methods that create usable knowledge (low gap) survive better.
Long-term Plateau: Even with study training, retention plateaus at 25–28% after 100 writes, indicating that while broad data helps, it does not solve the reachability problem entirely.

3. Forgetting Destroys Access, Not Storage

The paper demonstrates that "forgetting" is a failure of retrieval (access) rather than erasure of the stored trace.

Storage Persistence: Reconstructed checkpoints show that "forgotten" facts (those failing all questions) retain 57–67% of the log-probability lift produced by their original write.
Address Capture: In bare-statement conditions, 70% of wrong answers to forgotten facts contain the content of the most recently written fact. This suggests later writes redirect the "address" (the query routing) to new content, overwriting the path to the old fact.
Context Recovery: When the statement of a forgotten fact is supplied in the prompt, accuracy recovers to 77–80% (study condition), proving the content is still present but unreachable via the model's internal query mechanism.
Joint Use Failure: Two individually usable written facts cannot be reliably used together (32% accuracy) because the model lacks a self-retrieval mechanism to access both simultaneously, unlike when both are in the prompt (91% accuracy).

4. General Abilities and Distributional Drift

General capability loss is highly correlated with distributional drift from the original model.

KL Correlation: Capability damage correlates strongly with KL divergence from the original model ( $\rho = 0.83$ to $0.946$).
Frozen Teacher: Distilling against a frozen original model preserves general capabilities (near-zero cost) and retains facts better than distilling against a moving, accumulated model.
Drift Penalty: Adding a KL penalty to the current base improves retention and capability but does not reduce the measured KL divergence from the original, suggesting the penalty prevents degenerate collapse rather than reversing drift.

5. Interference is Caused by Incoming Writes

Causal tests isolate the source of interference.

Incoming vs. Stored: The method of the incoming write determines retention. Study-based incoming writes improve retention by 37.6 points compared to bare-statement incoming writes, regardless of how the earlier fact was stored.
Local Control Failure: Linearized predictors can forecast the immediate effect of an update ( $\rho = 0.795$ ) but fail to predict long-term forgetting ( $\rho = -0.258$ ). Local interventions (gradient projection, activation patching) failed to improve retention significantly.

Significance and Claims

The paper concludes that continual learning in weights is currently insufficient for facts that must be composed or survive extensive later writes.

The "Question-Keyed" Nature of Weights: Written knowledge is "question-keyed," meaning it is only reachable via the specific questions used during training. The model lacks a general "address" or handle to retrieve facts on demand. Later writes redirect these question keys to new content.
Context as the Reliable Channel: While broad data can create usable knowledge and frozen teachers can preserve general capabilities, no tested intervention ensures earlier facts remain reachable. Therefore, context remains the reliable channel for facts requiring composition or long-term retention.
Practical Guidance: To maximize the utility of weight-based updates, practitioners should use broad training data (diverse restatements) and distill against a frozen original model. However, for robust, composable knowledge, facts should be kept in context or external retrieval systems rather than relying solely on weight updates.

The work reframes catastrophic forgetting not as data erasure, but as a loss of access routes, where the "address" of a fact is captured by subsequent updates. It highlights a fundamental limitation in current weight-based continual learning: the ability to store content does not equate to the ability to create a persistent, retrievable address for that content.