🤖 machine learning

Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

This paper introduces a head-level metric called differential circuit vulnerability to demonstrate that while supervised fine-tuning adapts models to new tasks faster, reinforcement learning better preserves internal computational circuits, thereby offering a mechanistic explanation for its superior resistance to catastrophic forgetting.

Original authors: Jeanmely Rojas Nunez, Viraj Sawant, Nathan Allen, Nomgondalai Amgalanbaatar, Yannis Zongo, Vasu Sharma, Maheep Chaudhary

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeanmely Rojas Nunez, Viraj Sawant, Nathan Allen, Nomgondalai Amgalanbaatar, Yannis Zongo, Vasu Sharma, Maheep Chaudhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Why Do AI Models "Forget"?

Imagine you have a brilliant student (the AI model) who already knows how to write poetry, solve math problems, and tell jokes. You want to teach them a new skill: Science.

You have two ways to teach them:

SFT (Supervised Fine-Tuning): You give them a textbook and say, "Memorize these answers exactly."
RL (Reinforcement Learning): You let them try to answer science questions, and you give them a "thumbs up" or "thumbs down" based on how good the answer is, letting them figure out the best way to learn.

The Problem: When you teach the student the new skill, they often start forgetting their old skills (like poetry or math). This is called Catastrophic Forgetting.

The Discovery: Recent studies showed that the "Thumbs Up/Down" method (RL) is better at keeping the old skills alive than the "Memorize the Textbook" method (SFT). But why? This paper digs inside the AI's brain to find the mechanical reason.

The Analogy: The Library of Thought Circuits

Think of the AI's brain not as a single block of memory, but as a massive library of tiny, specialized workers (called "circuits" or "attention heads").

Some workers are great at math.
Some are great at grammar.
Some are great at telling jokes.

When the AI learns something new, it has to rearrange these workers. The question is: Does it fire everyone and hire new ones, or does it keep the old team and just give them new instructions?

The Experiment: What Happened?

The researchers took a specific AI (Qwen2.5-3B) and taught it to answer science questions using both methods. Then, they looked inside the "library" to see which workers were still working and how they were behaving.

1. The "Textbook" Method (SFT) = The Over-Optimizer

What it does: It learns the new science task very fast. It gets high scores quickly.
The Cost: To get those high scores, it aggressively fires the old workers and replaces them with a tiny, specialized team of "Science Experts."
The Result: The library shrinks. It keeps only about 52% of the original workers. The old workers (poetry, math, jokes) are pushed out. The AI becomes a great scientist but a terrible poet.
Analogy: It's like a general contractor who, to build a new kitchen, rips out the entire house's wiring and plumbing to make it fit the new design perfectly. The kitchen is perfect, but the lights in the bedroom don't work anymore.

2. The "Thumbs Up/Down" Method (RL) = The Careful Renovator

What it does: It learns the new science task slower. It takes more time to get to the same high score.
The Benefit: Instead of firing the old team, it gently guides them. It keeps almost all the original workers (about 72%) and just teaches them how to apply their existing skills to science.
The Result: The library stays large and diverse. The AI learns science, but it also remembers how to tell jokes and do math.
Analogy: It's like a contractor who builds the new kitchen by carefully rearranging the existing furniture and wiring. It takes longer to finish, but the lights in the bedroom still work, and the house feels the same.

The Key Findings (The "Mechanistic" Proof)

The paper introduced a new way to measure this called "Differential Circuit Vulnerability." Here is what they found:

SFT is a "Compressor": It squeezes the AI's brain into a small, specialized shape. It creates a few "Super-Workers" who do everything for the new task, but in doing so, it breaks the delicate networks that handled the old tasks.
RL is a "Distributed Adaptor": It spreads the new learning across the whole brain. No single worker is overwhelmed. The original "circuits" (the pathways the AI used for its old skills) stay intact and continue to function.
The Trade-off:
- SFT: Fast learning, but you lose your old personality and skills.
- RL: Slower learning, but you keep your old personality and skills.

Why This Matters

The paper concludes that RL is more robust against forgetting because it preserves the internal "machinery" of the AI.

When we use SFT, we are essentially overwriting the AI's internal code to fit a new shape, which causes the old code to break. When we use RL, we are tweaking the existing code, allowing the AI to grow new skills without deleting the old ones.

In short: If you want an AI that learns fast but forgets everything else, use the "Textbook" method. If you want an AI that learns steadily and keeps its original personality and skills, use the "Thumbs Up/Down" method.

Technical Summary: Mechanistic Origins of Catastrophic Forgetting

Problem Statement
Fine-tuning large language models (LLMs) for new downstream tasks frequently induces catastrophic forgetting, where gains on a new objective come at the expense of prior capabilities. While recent behavioral studies suggest that Reinforcement Learning (RL) preserves prior capabilities more effectively than Supervised Fine-Tuning (SFT)—attributing this to policy-gradient updates remaining closer to the base policy in terms of distributional drift (measured by KL divergence)—the internal mechanistic reasons for this difference remain unexplored. The central question addressed is whether RL's retention advantage is mirrored by a stronger preservation of the internal computational circuits (attention heads, MLP layers, and residual pathways) that implement model capabilities.

Methodology
The authors investigate this question using Qwen2.5-3B-Instruct adapted to a scientific question-answering task. The experimental pipeline consists of three phases:

Behavioral Reproduction: The study reproduces the known retention gap between SFT and RL. A base model ( $\pi_{base}$ ) is fine-tuned via completion-only cross-entropy loss to create an SFT model ( $\pi_{SFT}$ ). This SFT model is then refined using Dr.GRPO (a variant of Group Relative Policy Optimization) to create an RL model ( $\pi_{RL}$ ). Behavioral drift is quantified using the expected KL divergence on retention benchmarks.
Circuit Identification: The authors employ Differential Binary Masking (DBM) to identify internal computational circuits at the attention-head level. DBM learns a mask over attention heads that interpolates between base and counterfactual activations. By optimizing these masks to increase the probability of target answers while maintaining sparsity, the method isolates sparse causal subnetworks (circuits) responsible for specific behaviors.
Cross-Model Circuit Comparison: The study introduces differential circuit vulnerability, a head-level measure comparing how much a circuit degrades under different training objectives. Key metrics include:
- Circuit Faithfulness: The ratio of behavior recovered by the extracted subgraph to the full model's behavior.
- Head-Level Mask Shifts ( $\Delta m_h$ ): Comparing mask values of fine-tuned models against the base model to identify preserved, amplified, or weakened heads.
- Necessity and Sufficiency: Interventions measuring the log-probability drop when a head is disrupted (necessity) and the behavior recovered when only that head is active (sufficiency).

Key Contributions

Differential Circuit Vulnerability: The paper introduces a novel metric to quantify the relative susceptibility of internal computational subgraphs to degradation under different training objectives (SFT vs. RL).
Mechanistic Explanation of Forgetting: The work extends the behavioral account of RL's superiority over SFT to the mechanistic level, demonstrating that RL preserves a larger fraction of the base circuit compared to SFT.
Trade-off Characterization: The study characterizes a fundamental trade-off in post-training: SFT adapts more rapidly to the target task but produces substantial circuit disruption, whereas RL preserves the base circuit at the cost of slower task adaptation.

Results

Circuit Retention Trajectories: Over two training epochs, SFT and RL diverge sharply. SFT retains only 63.5% of base attention heads after Epoch 1, declining to 59.0% by Epoch 2. In contrast, RL retains 69.8% after Epoch 1 and recovers to 72.5% by Epoch 2, maintaining a 13.5 percentage-point advantage in base circuit retention.
Structural Compression vs. Distributed Adaptation: DBM identifies a base circuit of 297 heads. The SFT model undergoes structural compression to roughly 265 heads (46.0% of total), whereas the RL model retains approximately 296 heads (51.4%). SFT concentrates new-task behavior into a smaller set of "critical specialists" (high necessity and sufficiency), while RL maintains a distributed architecture overlapping closely with the base model.
Causal Engagement: RL maintains consistently higher Differential Causal Mediation (DCM) scores (e.g., 15.8 vs. 10.4 at Epoch 1), indicating that the preserved circuit in RL carries stronger causal signal regarding model behavior compared to SFT.
Output vs. Internal Metrics: Notably, the RL model exhibits a larger output-space KL divergence than the SFT model yet preserves more prior capabilities and internal circuit structure. This suggests that output-space metrics may not reliably predict internal forgetting.

Significance
The paper claims that circuit preservation offers a mechanistic explanation for why RL is more robust to catastrophic forgetting than SFT. The findings suggest that continual adaptation should be viewed not merely as parameter optimization, but as the selective preservation and modification of circuits encoding skills. The results indicate a trade-off where SFT acts as a "compressor" that aggressively reconfigures circuits for rapid specialization, while RL acts as a "distributed adaptor" that conservatively reuses existing representations. This perspective provides a foundation for understanding the internal causes of forgetting and suggests that future adaptation methods could benefit from combining efficient learning with selective circuit preservation.

Limitations
The authors note that the study is limited to a single model architecture (Qwen2.5-3B-Instruct) and a specific task set (scientific QA). The circuit analysis focuses primarily on attention heads, excluding MLP layers and residual-stream features. Generalizability to other architectures (e.g., Gemma, Mistral, Llama) and broader capability domains remains to be validated.