🤖 machine learning

When Does Continual Learning Require Learning

This paper argues that continual learning for large language models must be reframed around increasing competence amidst spatial and temporal changes, demonstrating through a unified evaluation protocol that no single method excels across all scenarios and that the optimal strategy—whether updating model weights or using external scaffolding—depends fundamentally on the specific pattern of environmental change.

Original authors: Anne Harrington, Nayan Saxena, Michael Murphy, Anastasia Borovykh, Zeyu Yun, Sridhar Kamath, Ara Eindra Kyi, Trevor Darrell, Jitendra Malik, Yutong Bai

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anne Harrington, Nayan Saxena, Michael Murphy, Anastasia Borovykh, Zeyu Yun, Sridhar Kamath, Ara Eindra Kyi, Trevor Darrell, Jitendra Malik, Yutong Bai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who knows everything up to a certain date. You want to keep teaching it new things as the world changes, without it forgetting the old stuff or getting confused. For a long time, scientists thought the biggest problem was just "forgetting"—like a student who studies for a history test and then instantly forgets how to do math. But this paper, titled "When Does Continual Learning Require Learning," argues that the real challenge is much more interesting: it's about figuring out how the robot should learn when the world changes in different ways.

The authors set up a giant playground with four different types of "world changes" to test eight different learning strategies. They didn't just guess; they ran real experiments using a specific robot brain (Qwen3-8B) and watched exactly what happened. Here is what they found, broken down into the four ways the world can change.

1. The "New Subject" Challenge (Space)

Imagine your robot friend is great at fixing cars, but suddenly you ask it to cook Italian food, and then later, to perform surgery. These are totally different skills.

The Test: They taught the robot three totally unrelated tasks: using tools, doing finance math, and biology.
The Result: Some methods, like Prompt Optimization (telling the robot new instructions via a chat message), were like a sprinter. They learned the new task super fast. But as soon as they moved to the next task, they degraded heavily on the previous ones. It was like a sprinter who runs a 100-meter dash in record time but then trips and falls immediately after.
The Winner: Methods that actually rewrote the robot's brain (called "Distillation-based" methods like SDFT and SDPO) were slower but much steadier. They managed to accumulate knowledge over time. Notably, SDFT was the only method that ended up with scores on all three tasks that were at or above where the robot started from scratch, successfully keeping the old skills while adding the new ones.

2. The "Fact Update" Challenge (Time: Discrete)

Now imagine the robot knows that "The capital of France is Paris." But then, in a weird twist of fate, the capital actually changes to Lyon (let's pretend!). The robot needs to update just that one fact without forgetting that Paris used to be the capital, or that London is the capital of the UK.

The Test: They gave the robot a stream of Wikipedia updates where specific facts changed month by month.
The Result: The "sprinters" (Prompt Optimization) were great at learning the new fact immediately. They wrote "Lyon" into their instructions and were done. But in doing so, they accidentally messed up other facts that hadn't changed, like the capital of Germany. They were too eager to update.
The Surprise: The methods that rewrote the brain (SDFT and SDPO) actually struggled here. They tried to blend the new fact with the old one and ended up degrading the accuracy of facts that were supposed to stay stable.
The Real Hero: One specific method using Reinforcement Learning (GRPO) was the only weight-update method that could update the fact to "Lyon" without its stable facts moving in the opposite direction. However, it's worth noting that Context Compression methods (like Cartridges) actually preserved the stable facts the cleanest of all, even if they didn't learn the new facts very well.

3. The "Noisy Trend" Challenge (Time: Drift)

Imagine you are trying to predict if a stock will go up or down based on a 10,000-word financial report. The rules of the market change slowly over years. Sometimes a phrase means "buy," and five years later, the same phrase means "sell." The signal is weak and noisy.

The Test: They fed the robot financial reports from 2015 to 2020, one year at a time, and asked it to predict the future.
The Result: The "sprinters" (Prompt Optimization) tried to find simple rules, like "if the word 'risk' appears, sell." But these rules were just lucky guesses for that specific year. When the market changed, the robot failed completely.
The Winner: The Distillation methods (SDFT) were very good at this, sustaining their ability to predict future years while keeping past performance steady. However, the Context Compression method (Cartridges) was actually the most stable method tested, hovering around the same accuracy throughout the entire chain without degrading or overfitting. It was like a wise old sailor who knows the ocean changes slowly, rather than a sailor who panics at every wave.

4. The "Agent" Challenge (Time: Accumulation)

Finally, imagine the robot is playing a game where it has to do a series of steps, like "Create a label, rename it, then email it." The catch? The robot's own actions change the game state for the next step. If it messes up step 1, step 2 becomes impossible.

The Test: They made the robot play a chain of web tasks (like managing emails) where the length of the chain grew from 1 step to 10 steps.
The Result: Without any learning, the robot would crash and burn after just a few steps. But when they let the robot learn from its past attempts (either by updating its brain or by keeping a "playbook" of notes), it got much better.
The Catch: Even with learning, the robot still got worse as the chains got longer. By the time it reached 10 steps, success rates dropped significantly. It suggests that while learning helps, there's a limit to how much a robot can hold in its memory when the game gets complicated.

The Big Takeaway

The paper suggests that there is no single "magic bullet" for teaching robots to learn forever.

If the world changes by giving you new skills, you need to rewire the brain slowly and steadily (specifically using methods like SDFT).
If the world changes by updating a specific fact, you need a method that can edit the brain surgically (like GRPO) or use external memory to avoid breaking other things.
If the world changes by slow, noisy trends, you need a method that ignores the noise and finds the stable pattern (like Cartridges or SDFT).
If the world changes by accumulating state (like a long game), you need experience, but even then, it gets harder the longer the game goes.

The authors didn't just guess this; they measured it across thousands of examples. They found that different types of change require fundamentally different ways of learning. It's not just about "learning more"; it's about knowing how to learn when the rules of the game shift.

Technical Summary: When Does Continual Learning Require Learning

Problem Definition

The paper argues that the current framing of continual learning (CL) in Large Language Models (LLMs) is incomplete. Traditionally, CL is defined as the problem of mitigating catastrophic forgetting—learning task B without losing task A. The authors contend this view is insufficient for modern LLM deployment, where models must increase competence as the world changes. They propose a redefinition: continual learning is the problem of increasing model competence as the world changes.

To analyze this, the authors disentangle environmental change along two axes:

Space (Domain Shift): The model encounters new domains or tasks (e.g., moving from tool use to finance).
Time (Temporal Drift): The task remains fixed, but the underlying data distribution shifts. This is further subdivided into:
- Slow Trends: Gradual shifts (e.g., yearly financial filings) requiring alignment with persistent structures.
- Discrete Fact Changes: Sudden updates to specific knowledge (e.g., Wikipedia revisions) requiring the rewriting of specific beliefs without corrupting stable knowledge.
- Agentic Accumulation: The environment's state drifts as a direct consequence of the agent's own prior actions (e.g., multi-step web navigation), creating a "frame problem" where the model must decide which beliefs are outdated.

Methodology

The authors introduce a unified, mechanism-agnostic protocol to evaluate CL methods on equal footing. Instead of tailoring benchmarks to specific update mechanisms, they recast widely used LLM benchmarks as sequential problems.

The Protocol:

A learner processes stages $T_1, \dots, T_K$ in fixed order.
Between stages, an update operator $\theta_k = U_k(\theta_{k-1}, D^{\text{tr}}_k)$ is applied.
$U_k$ is unrestricted: it can modify weights, edit system prompts, write to external memory, or perform no update.
Evaluation: A "forgetting matrix" is generated by evaluating the model at stage $i$ $i$ on the test sets of all stages $j$ $j$ . This yields two metrics:
- Backward Transfer (BWT): Measures if later stages damage earlier ones.
- Forward Transfer (FWT): Measures if prior stages prepare the model for future, unseen stages.

Evaluated Methods:
Eight methods across four families were tested on a Qwen3-8B backbone:

Prompt Optimization: GEPA (reflect-and-mutate) and ACE (markdown playbook).
Offline Supervised Learning: SFT (full fine-tuning) and SDFT (self-distillation).
Online Reinforcement Learning: GRPO (group relative policy optimization) and SDPO (sequential DPO).
Context Compression: Cartridges (learned KV cache adapters) and In-place TTT (test-time training).

Experimental Settings:

Domain Shift: Sequential training on ToolUse, FinQA, and SciKE-Bio.
Discrete Fact Update (TempWiki): Learning new facts from Wikipedia diffs while monitoring stable facts to detect "catastrophic memorizing."
Noisy Temporal Drift: Predicting stock movements from 10-K filings (2015–2020), where signals are weak and regimes shift.
Agentic Tasks: Sequential chains on WebArena-Infinity (e.g., Gmail) where state persists across steps.

Key Results

The study reveals consistent trade-offs across method families, suggesting no single approach handles all regimes:

Prompt-Based Methods (GEPA, ACE):
- Strengths: Fit quickly to new stages and achieve strong backward accuracy.
- Weaknesses: Suffer from severe catastrophic forgetting on future tasks. In the TempWiki setting, they overwrite stable knowledge to fit new facts. In the 10-K task, they overfit to transient heuristics, failing to generalize forward.
- Agentic: ACE improves over zero-shot baselines by accumulating procedural guidance.
Distillation-Based Methods (SDFT, SDPO):
- Strengths: Accumulate knowledge stably over time with minimal degradation of past capabilities. SDFT is the only method to maintain or exceed zero-shot baselines across all domain shift tasks.
- Weaknesses: Struggle to rapidly incorporate new information or update outdated facts. In the TempWiki setting, they degrade stable facts while trying to interpolate new data. They show strong forward transfer in noisy temporal drift (10-K) but fail to learn discrete fact updates effectively.
Context Compression (Cartridges, In-place TTT):
- Strengths: Improve efficiency and memory management. They are less sensitive to domain shifts than prompt methods.
- Weaknesses: Do not substantially improve the ability to learn new tasks or accumulate capabilities. In the 10-K task, they neither acquire nor degrade, failing to absorb slow-moving trends.
Online Reinforcement Learning (GRPO, SDPO):
- Strengths: Adapt most effectively to discrete knowledge updates (TempWiki), with GRPO showing the sharpest gains in learning new facts while preserving stable knowledge better than other weight-update methods.
- Weaknesses: Highly sensitive to noisy reward signals. In the 10-K task, GRPO degrades significantly on both past and future years due to weak, noisy rewards.
Agentic Accumulation:
- Both prompt-based (ACE) and weight-based (SFT) methods compound experience and beat zero-shot baselines at every chain length. However, absolute success still decays as the chain lengthens, indicating that accumulated state eventually overwhelms the model's ability to ground actions.

Key Contributions

Unified Framework: Formalizes continual learning as increasing competence over time, defining change along space (domain) and time (drift, facts, agentic state) axes.
Mechanism-Agnostic Protocol: Introduces a protocol allowing fair comparison of prompt, weight, and architectural updates on a common backbone.
Sequential Benchmark Suite: Adapts standard benchmarks (FinQA, 10-K, TempWiki, WebArena) into sequential tasks to evaluate CL under realistic conditions.
Empirical Trade-off Analysis: Demonstrates that different environmental change patterns require fundamentally different update behaviors. Prompting is fast but unstable; distillation is stable but slow to update; RL adapts well to facts but fails on noise; compression aids efficiency but not learning.

Significance and Claims

The paper claims that continual learning is not a single capability. The authors argue that the field's focus on "mitigating forgetting" is a misalignment with the practical needs of deployed LLMs, which must adapt to a changing world.

The primary significance lies in the finding that different patterns of environmental change require fundamentally different update behaviors.

When the world changes via discrete facts, learning inside model weights (specifically via RL or targeted updates) is necessary.
When the world changes via slow trends, stability-anchored methods (like distillation) that preserve past structure are superior.
When the world changes via agentic accumulation, external scaffolding (playbooks) or fine-tuning can help, but both face limits as state complexity grows.

The authors conclude that understanding where each method succeeds and fails is critical for designing stronger continual learning systems. They do not propose a new algorithm but rather a diagnostic framework to guide the selection of update mechanisms based on the specific nature of the environmental change.

Limitations: The authors note that their conclusions are based on a single model (Qwen3-8B) and a specific subset of environmental changes. They caution that results may vary for larger models or reasoning-enabled models, and that real-world deployment involves even more open-ended and uncertain environments than those tested.