🤖 machine learning

Understanding Generalization and Forgetting in In-Context Continual Learning

This paper introduces the first theoretical framework for in-context continual learning, demonstrating that standard attention mechanisms in Transformers inevitably cause intertask interference and systematic bias when processing sequential heterogeneous tasks, thereby explaining fundamental limits on generalization and the occurrence of forgetting in long prompts.

Original authors: Guangyu Li, Meng Ding, Lijie Hu

Published 2026-05-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guangyu Li, Meng Ding, Lijie Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Super-Student" Who Never Takes Notes

Imagine a brilliant student (the Large Language Model) who has already read a massive library of books during their training. They are incredibly smart, but they have a unique rule: They are not allowed to change their brain or take notes.

When you give them a new test, you can't teach them new facts. Instead, you have to whisper the answers to previous questions right next to the new question in the same conversation. This is called In-Context Learning (ICL). The student looks at the examples you provided in the chat and figures out the pattern on the fly.

This paper asks a specific question: What happens if you give this student a long conversation containing a mix of different types of tests?

For example, imagine a single chat window where you first ask them to solve math problems, then switch to translating French, then switch to writing poetry, all without stopping the conversation. Does the student get better at math because they saw more examples? Or does the French translation confuse their math skills?

The Core Discovery: The "Mixing Bowl" Problem

The researchers built a mathematical model to understand how the student's "brain" (specifically, the Attention Mechanism) processes this long, mixed-up conversation.

They discovered that the student's brain works like a mixing bowl. When the student looks at the conversation to answer a question, they don't just look at the math examples; they look at everything in the bowl (the math, the French, the poetry) and blend them together to form an answer.

Here are the three main things they found:

1. The "Too Much Context" Trap (Generalization)

The Intuition: You might think, "If I give the student more examples of the math problem, they will get better at math."
The Reality: This is only true if the examples are all about math. If you give them 10 math examples followed by 10 French examples, the student gets confused. The French examples act like "noise" in the math bowl.
The Analogy: Imagine trying to taste a specific spice in a soup. If the soup only has that spice, adding more of it makes the flavor stronger. But if you start adding huge amounts of chocolate and broccoli into the same pot, adding more spice doesn't help; it just makes the soup taste weird and messy.
The Result: The paper shows that adding more context (more examples) doesn't always help. If the tasks are different, adding more examples can actually make the student worse at the current task because the "wrong" information from previous tasks gets mixed in.

2. The "Catastrophic Amnesia" (Forgetting)

The Intuition: Since the student never deletes anything from the chat history, they should remember everything perfectly, right?
The Reality: Even though the information is still there, the student "forgets" how to use it.
The Analogy: Imagine the student is trying to solve a math problem at the very end of a 100-page document. The math examples are on page 1. The French examples are on page 50. The student's brain is designed to pay attention to the most recent things they read. As they read the French and Poetry sections, those new words "drown out" the math words from page 1. The math info is still on the page, but the student's attention mechanism has shifted its focus so heavily to the new stuff that the old math info becomes useless.
The Result: The student forgets the first tasks not because the data is gone, but because the way they weigh the information changes. The more different tasks you add, the more the student forgets the earlier ones.

3. The Order Matters (Sequence Sensitivity)

The Intuition: It shouldn't matter if you ask for Math then French, or French then Math.
The Reality: It matters a lot.
The Analogy: Think of it like painting a wall. If you paint a wall blue (Task A) and then immediately paint over it with red (Task B), the final color is a muddy purple. If you paint red first and then blue, you get a different shade of purple. The order in which you present the tasks changes how the "colors" (information) mix together.
The Result: The paper proves mathematically that the order of tasks creates a specific "bias." If you put similar tasks together (e.g., Math then Algebra), the student does well. If you put very different tasks together (Math then Poetry), the student gets confused and performs poorly.

The "Bias-Variance-Interference" Recipe

The authors broke down the student's mistakes into three ingredients:

Variance: The student is just guessing because they didn't see enough examples of the current task. (Fix: Add more examples of the same task).
Bias: The student is confused because the examples they did see were from a different task. (Fix: Don't mix different tasks).
Interference: The student is actively mixing the wrong tasks together, creating a systematic error that gets worse the longer the conversation gets.

The Bottom Line

This paper explains why Large Language Models sometimes struggle when you give them long, complex prompts with many different types of questions.

It's not a bug; it's a feature of the design. The mechanism that allows them to learn from examples in real-time (Attention) is the exact same mechanism that causes them to forget old tasks when new, different tasks arrive.
More isn't always better. Just because you can put more text in the prompt doesn't mean the model will understand better. If you mix too many different "flavors" of tasks, the model gets overwhelmed and makes systematic errors.
The "Forgetting" is structural. The model doesn't "delete" old info; it just stops paying attention to it because the new info is louder in the mixing bowl.

In short: If you want a Large Language Model to do a great job on a specific task, keep the conversation focused. If you mix too many different topics into one long chat, the model will inevitably get confused and forget how to do the earlier tasks.

Technical Summary: Understanding Generalization and Forgetting in In-Context Continual Learning

Problem Statement

While Large Language Models (LLMs) demonstrate remarkable In-Context Learning (ICL) capabilities, adapting to new tasks via prompts without parameter updates, existing theoretical frameworks primarily analyze ICL in single-task settings. In real-world scenarios, prompts often contain sequences of heterogeneous tasks. This raises a critical question: Do LLMs implicitly perform continual learning during inference, and if so, how do they manage generalization and forgetting across sequential tasks?

Current research lacks a theoretical understanding of how shared attention mechanisms process multiple sequential tasks within a single prompt. Specifically, it is unclear whether LLMs can accumulate and reuse task-relevant information over long contexts or if they suffer from interference and catastrophic forgetting similar to traditional training-time continual learning, despite the absence of parameter updates.

Methodology

The authors propose the first theoretical framework for In-Context Continual Learning (ICCL). The core of their approach involves:

Modeling Setup:
- They model a pretrained Transformer processing a sequence of $T$ tasks concatenated into a single prompt.
- The framework focuses on linear regression tasks where task parameters ( $w_t$ ) are sampled from a distribution.
- They analyze two attention mechanisms:
  - Linear Self-Attention (LSA): Aggregates information uniformly across all tokens.
  - Masked Linear Self-Attention (MSA): Employs a causal mask to ensure information flows only forward, respecting task order and boundaries.
Theoretical Derivation:
- The authors derive explicit error expressions for model predictions under sequential task prompts.
- They decompose the prediction error into a Bias-Variance-Interference framework.
- They define Generalization Error ( $G_t$ ) as the prediction accuracy on the current task's query and Forgetting ( $F_t$ ) as the increase in error on a previously seen task after processing subsequent tasks.
Empirical Validation:
- Experiments were conducted using decoder-based GPT-2 architectures (standard and tiny configurations) trained on linear regression.
- Theoretical predictions were validated against empirical Mean Squared Error (MSE) across varying context lengths ( $M$ ), task similarities, and task orderings.
- The framework was further tested on non-linear tasks (sparse linear regression, ReLU networks) and real-world LLMs (Qwen2.5-1.5B-Instruct) using heterogeneous text classification tasks (SST-2 and AG News).

Key Contributions

1. First Theoretical Formalization of ICCL

The paper formalizes ICCL as a setting where a pretrained Transformer processes sequential tasks via shared attention without parameter updates. This explicitly bridges the gap between ICL and core continual learning concepts like task order, historical accumulation, and interference.

2. Bias-Variance-Interference Decomposition

The authors provide an explicit decomposition of prediction error, revealing that:

Variance arises from finite in-context samples and decreases as context length ( $M$ ) increases.
Bias stems from the misalignment between the current task and historical tasks.
Interference is a structural property of attention aggregation, where historical contexts induce systematic bias.
Crucially, they show that increasing context length or task history does not universally improve performance; it can degrade generalization when tasks are misaligned due to increased bias.

3. Theoretical Grounding for Empirical Phenomena

The framework explains several observed phenomena in real models:

Order-Dependent Forgetting: Forgetting is not merely a loss of information but arises from the reweighting of task contributions by the attention mechanism. Past tasks receive negative coefficients in the final prediction, actively offsetting their representations.
Task Similarity Effects: Positive transfer occurs when historical tasks are aligned with the target task. Conversely, learning future tasks that are statistically similar to the current one amplifies forgetting, while orthogonal tasks preserve performance.
Limits of Long Contexts: The benefit of longer contexts plateaus or reverses when task means are misaligned, as the irreducible bias from mean misalignment persists regardless of context length.

Results

Generalization: In multi-task settings, increasing the number of in-context examples ( $M$ ) initially reduces error (variance reduction) but can eventually increase error if the added context introduces systematic bias from dissimilar tasks.
Forgetting: Forgetting is proven to be an intrinsic consequence of attention-based aggregation. Even without parameter updates, the model "forgets" earlier tasks because the attention mechanism aggregates historical contexts with fixed, decaying weights, leading to systematic interference.
Task Ordering: The order of tasks significantly impacts performance. Sequences where later tasks are misaligned with earlier ones induce stronger interference.
Real-World Validation: Experiments on Qwen2.5-1.5B-Instruct confirmed that introducing a second heterogeneous task (Task B) after Task A causes severe negative transfer (dropping Task B accuracy by >15% at low $M$ ) and catastrophic forgetting of Task A (accuracy dropping from 0.934 to 0.472), validating the theoretical predictions of irreducible mean misalignment.

Significance and Claims

The paper claims to provide the first theoretical framework for understanding in-context continual learning. Its primary significance lies in:

Identifying Fundamental Limits: It demonstrates that attention-based continual inference has inherent limits. Even without parameter updates, LLMs accumulate, interfere, and sometimes forget task-specific information over long contexts.
Explaining Mechanisms: It offers a mathematical explanation for why standard attention mechanisms induce inter-task interference and systematic bias, leading to order sensitivity and performance degradation in long prompts.
Guidance for Practice: The results provide concrete guidance for prompt design and multi-task evaluation, suggesting that simply increasing context length or reusing historical prompts is not a panacea and can be detrimental if task alignment is not considered.

The authors emphasize that their work explains when and why in-context generalization succeeds or fails, highlighting that the "parameter-free" nature of ICL does not preclude the fundamental challenges of continual learning, such as interference and forgetting.