💬 NLP

Transformers Remember First, Forget Last: Dual-Process Interference in LLMs

By adapting cognitive psychology paradigms to test 39 large language models, this study reveals that transformers universally exhibit a dominant proactive interference effect where early information overrides recent data—a pattern distinct from human memory that suggests separate, capacity-dependent mechanisms for retrieval and consolidation.

Original authors: Sourav Chattaraj, Kanak Raj

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sourav Chattaraj, Kanak Raj

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Old Guard" vs. The "New Kid"

Imagine you are trying to remember a list of facts about a patient's blood pressure.

Fact 1: 120 (Morning)
Fact 2: 135 (Noon)
Fact 3: 128 (Afternoon)
Fact 4: 118 (Evening)

If a doctor asks, "What was the blood pressure this morning?" (asking for the old fact), that is a Retroactive test.
If the doctor asks, "What is the blood pressure right now?" (asking for the new fact), that is a Proactive test.

In human brains, the "New Kid" usually wins. If you learn something new, it often pushes the old stuff out of your head. This is called Retroactive Interference.

But here is the shocker: The researchers tested 39 different AI models (LLMs) and found they do the exact opposite.

Humans: Forget the old to make room for the new.
AI: Clings desperately to the old and forgets the new.

The AI models act like a stubborn librarian who refuses to throw away the first book they ever shelved, even if a newer, more important book arrives. They protect the "First Impression" so fiercely that they can't remember what happened five minutes ago.

The Two Types of Memory Trouble

The researchers used a classic psychology test (like a memory game) to see how AI handles conflicting information. They found two distinct ways AI fails, which are totally unrelated to each other.

1. The "Blank Page" Failure (Retroactive Interference)

The Scenario: You ask the AI about the first value (the morning blood pressure).
The Problem: The AI looks at the new updates (noon, afternoon, evening) and gets confused. It's like trying to read a page in a book where someone has scribbled over the original text with a thick marker.
The Result: The AI often just says, "I don't know," or gives a random guess. It's a passive failure. The memory is just gone or buried.
The Good News: Bigger AI models (with more "brain power") are much better at this. If you give the AI more space to store facts, it can keep the old facts safe from the new scribbles.

2. The "Ghost in the Machine" Failure (Proactive Interference)

The Scenario: You ask the AI about the latest value (the evening blood pressure).
The Problem: The AI knows the evening value, but the morning value keeps popping up in its head and shouting, "No! It was 120!"
The Result: The AI confidently gives you the old answer (120) instead of the new one (118). It's an active failure. The old memory is so strong it's actively blocking the new one.
The Bad News: Making the AI bigger does not help. Even the biggest, most powerful AI models still struggle with this. It's a fundamental design flaw in how they "pay attention."

Why Does This Happen? (The "Flashlight" Analogy)

Imagine the AI's attention mechanism is a flashlight shining on a long hallway of information.

How Humans Work: The flashlight moves smoothly down the hallway. When you get to the end, the light is brightest on the newest items. The old items at the start fade into the dark.
How AI Works: The flashlight is broken. It shines super bright on the very first item in the hallway and never really turns down the brightness. As new items appear, the light stays stuck on the first one.
- Because the light is so bright on the first item, the AI thinks, "This must be the most important thing!"
- When you ask for the newest item, the AI is still blinded by the brightness of the oldest item.

This is why the AI is great at remembering the beginning (it's stuck there) but terrible at remembering the end (it can't look away from the start).

Key Takeaways for the Real World

Size Matters (But Only for Some Things): If you need an AI to remember a long history (like a legal case file or a medical history), bigger models are better. They have more "shelf space" to keep old facts safe.
Bigger Doesn't Fix Everything: If you need an AI to track real-time updates (like a stock ticker or a live chat), bigger models won't help. They will still get stuck on the first thing they heard.
The "Reasoning" Trap: The smartest, most "reasoning" AI models (the ones that think step-by-step) are actually worse at remembering the latest news. They are so good at analyzing the past that they get stuck in it.
No Hallucinations: Interestingly, when the AI gets this wrong, it rarely makes things up. It usually just picks the wrong fact from the list it was given. It's a memory mix-up, not a lie.

The Bottom Line

AI models are not like human brains. They have a "Primacy Bias"—they love the beginning of a story so much they forget the ending.

If you are building an app that needs to track changes over time, you can't just buy the biggest, smartest AI and hope for the best. You have to design your system to help the AI look at the new stuff, because its natural instinct is to stare at the old stuff.

1. Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities in few-shot learning and reasoning, their ability to manage memory interference in long contexts remains poorly understood. Specifically, it is unclear how LLMs handle conflicting information when multiple updates occur for the same semantic entity within a single context window.

The paper addresses two critical questions:

Interference Direction: Do LLMs suffer more from Retroactive Interference (RI) (where new information disrupts recall of old information) or Proactive Interference (PI) (where old information blocks the recall of new information)?
Mechanism: Do RI and PI stem from a unified "memory capacity" or distinct computational mechanisms?

This is distinct from "catastrophic forgetting" (weight updates during training); the study focuses on in-context interference (competition between items during a single forward pass).

2. Methodology

The authors adapted classical interference paradigms from cognitive psychology (specifically the AB-AC paired-associate task) to test 39 diverse LLMs.

Experimental Design:
- Stimuli: Models were presented with 46 semantic category-value pairs (e.g., "Visual Art: Impressionism").
- Updates: For each category, $N$ interleaved updates were introduced, assigning new values to the same category (e.g., "Visual Art: Baroque"). $N$ ranged from 3 to 300.
- Tasks:
  - Retroactive Interference (RI): Query the initial value (e.g., "What was the first value for Visual Art?").
  - Proactive Interference (PI): Query the most recent value (e.g., "What is the current value for Visual Art?").
- Control: Both tasks used identical stimulus sequences; only the query target differed, isolating interference from search difficulty.
Models Tested: 39 LLMs spanning 1B to 2.5T parameters, including open-weight (Llama, Mistral, Qwen) and proprietary (GPT-4o, Claude-3.5, o1, o3) architectures, with context windows ranging from 8K to 2M tokens.
Metrics:
- RIES (Retroactive Interference Endurance Score): Area under the accuracy curve across interference levels.
- PIES (Proactive Interference Endurance Score): Equivalent metric for PI.
- Error Analysis: Classification of failures into retrieval failure, same-key intrusion, cross-key intrusion, hallucination, and partial matches.

3. Key Contributions

Inverted Interference Profile: The study establishes that all 39 tested LLMs exhibit Proactive Interference > Retroactive Interference (PI > RI). This is the inverse of human memory, where RI typically dominates (new information overwrites old).
Dual-Process Evidence: RI and PI are shown to be uncorrelated ( $R^2 = 0.044$ ) and scale differently, suggesting they engage distinct computational mechanisms rather than a unified memory capacity.
Failure Mode Taxonomy: The paper distinguishes between passive failures (RI: inability to retrieve) and active failures (PI: intrusion of earlier values), with minimal hallucination in both cases.

4. Key Results

A. Universal PI > RI Asymmetry

Finding: Every model tested showed significantly higher resistance to RI than PI.
Magnitude: The mean difference in endurance scores was 69.5 points (Cohen's $d = 1.73$ , $p < 0.0001$ ).
Human Contrast: Humans typically show RI > PI (recent info disrupts old). LLMs show the opposite: Transformer attention creates a "primacy bias," protecting early encodings at the cost of recent information.

B. Scaling Laws and Architecture

Model Size vs. RI: Parameter count strongly predicts RI resistance ( $R^2 = 0.49$ ). Larger models (>500B) maintain significantly better initial recall than smaller models.
Model Size vs. PI: Parameter count does not predict PI resistance ( $R^2 = 0.06$ , n.s.).
Context Length: Context window size showed no correlation with either RI or PI resistance. A 100B model with 32K context outperformed a 10B model with 1M context.
Reasoning Models: Chain-of-thought models (e.g., o1, o3) showed an 18% improvement in RI resistance but performed poorly on PI (e.g., o1 ranked #1 for RI but #35 for PI), suggesting reasoning aids consolidation but hinders recency access.

C. Error Analysis & Failure Modes

RI Failures (Passive): Predominantly retrieval failures (51%). The model cannot access the initial value; it is effectively "overwritten" or inaccessible.
PI Failures (Active): Predominantly same-key intrusions (56%). The model actively retrieves an earlier value from the correct category, blocking the recent one.
Hallucination: Extremely low (<1%) for both, indicating the issue is position confusion, not fabrication.
Metacognition: Models explicitly refused to answer in 8.6% of PI failures (sensing competition) but 0% of RI failures (confidently returning the wrong, most-attended value).

D. Architectural Variations

O-Series (Reasoning): Extreme asymmetry (8.5x difference), optimizing historical accuracy at the cost of recency.
Claude Models: Minimal asymmetry (~1.05x), suggesting a balanced memory design.
Decay Patterns: RI decay varies by architecture (exponential, polynomial, log-quadratic), while PI decay follows a universal log-linear decline, further supporting distinct mechanisms.

5. Significance and Implications

Theoretical Implications

Dual-Process Mechanism: The results parallel the consolidation–retrieval distinction in cognitive science.
- RI (Consolidation): Capacity-dependent. Larger models use more orthogonal vectors (reducing "superposition"), allowing initial encodings to survive updates.
- PI (Retrieval): Architecture-constrained. The self-attention mechanism inherently accumulates attention on early tokens ("attention sinks"), creating a structural bias that parameters cannot override.
Transformer Bias: The "primacy protection" is a feature of causal attention, where early tokens accumulate attention from all subsequent positions, while later tokens only attend backward.

Practical Implications

Model Selection:
- For Historical Memory: Applications requiring recall of initial states (e.g., legal document analysis, medical history) should prioritize larger models and reasoning architectures (e.g., O-series).
- For Real-Time Tracking: Applications requiring the latest state (e.g., dialogue, live monitoring) should avoid reasoning-heavy models and prefer architectures with balanced designs (e.g., Claude), as standard scaling does not fix PI.
Context Length Myth: Increasing context window size alone does not solve interference; parameter count (representational capacity) is the primary driver for interference resistance.

Future Directions

The authors suggest moving toward ecologically valid benchmarks (narratives, evolving documents) and mechanistic validation (attention probing) to confirm that early tokens accumulate disproportionate attention. They also propose targeted mitigations like recency-weighted attention or positional debiasing.