🤖 AI

How to Teach Large Multimodal Models New Skills

This paper demonstrates that selectively tuning specific components of Large Multimodal Models, such as self-attention projections or MLP gates, effectively mitigates catastrophic forgetting during sequential skill learning by controlling output distribution shifts, outperforming full fine-tuning and existing mitigation methods without requiring complex mechanisms like replay or auxiliary parameters.

Original authors: Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multi-talented assistant named "The Model." This assistant can describe photos, read medical scans, tell time, and identify birds. They are incredibly smart, but they were trained on a massive, general dataset. Now, you want to teach them a very specific new skill: counting objects in a picture.

The problem? When you try to teach them this new skill, they often start acting weird. They might get great at counting but suddenly forget how to describe a scene normally, or they might start answering every question with a number, even when you just asked for a description. This is called "Catastrophic Forgetting." It's like a student who studies so hard for a math test that they forget how to speak their native language.

This paper asks a simple question: How do we teach this assistant a new skill without making them forget everything else?

Here is the breakdown of their discovery, using some everyday analogies.

1. The Problem: The "Over-Writer"

The researchers found that when you try to teach the model a new skill by updating its entire brain (all its parameters), it's like handing a student a red marker and telling them to rewrite their entire textbook to learn one new chapter. They might learn the new chapter perfectly, but they accidentally scribble over the old chapters, making them unreadable.

In technical terms, the model starts to bias its output. If you teach it to count, it gets obsessed with numbers. It starts thinking, "Oh, you asked me a question? I must give you a number!" even if you asked for a description.

2. The Discovery: The "Specialized Tools"

The team realized the model's brain isn't a single blob; it's made of different tools. They tested updating different parts of the brain to see which one caused the least damage.

They found two "magic tools" that work perfectly:

Tool A: The "Traffic Director" (Self-Attention Projections)

The Analogy: Imagine the model is a busy airport. The Self-Attention layers are the air traffic controllers. They decide which plane (piece of information) talks to which other plane. They don't write the flight plans; they just route the traffic.
The Result: When the researchers only updated the "Traffic Controllers," the model learned the new skill (counting) very well. But because they didn't touch the "flight plan writers," the model didn't get obsessed with numbers. It could still fly its old routes perfectly.
The Catch: It learned the new skill well, but not quite as fast as if they had rewritten the whole book.

Tool B: The "Fact Writer" (MLP Gate & Up)

The Analogy: The MLP (Feed-Forward Network) is the part of the brain that actually stores facts and writes the answers. It's like a librarian who pulls books off the shelf.
The Problem: Usually, when you teach the librarian a new fact, they get so excited they start shouting that fact at everyone, even when you just want to ask for the weather.
The Fix: The researchers found a way to update the librarian's "search skills" (Gate & Up) but froze their "shouting skills" (the Down projection).
The Result: The librarian learned the new fact perfectly but didn't start screaming it at everyone. They could still answer normal questions normally. This was the best balance: they learned the new skill almost as well as the "full rewrite" method, but they forgot almost nothing.

3. The "Recovery" Phenomenon

One of the coolest findings was that forgetting isn't always permanent.

The Analogy: Imagine the model gets "stuck" in a mindset. After learning to count, it thinks, "Everything is a number!"
The Magic: If you then teach it a different skill, like reading a medical report (which requires words, not numbers), the model "wakes up." The obsession with numbers fades, and it remembers how to describe things again.
The Lesson: The information wasn't actually deleted; it was just temporarily buried under a new, narrow habit. Changing the task can dig it back up.

4. The Solution: "Selective Tuning"

Instead of trying to use complex, expensive methods to save the old data (like keeping a backup of every book the student ever read), the researchers suggest a simple rule:

Don't rewrite the whole book. Just update the specific pages that handle the new skill.

If you want stability: Update only the "Traffic Controllers" (Self-Attention).
If you want maximum learning with minimal damage: Update the "Search Skills" of the Librarian (MLP Gate & Up) but leave the "Shouting" part frozen.

Why This Matters

Teaching AI new skills is usually expensive and risky. You might spend millions of dollars training a model, only to have it forget how to be helpful in general.

This paper gives us a simple, cheap, and effective recipe: Be surgical. Don't change the whole brain. Just tweak the specific parts that handle the new task. This way, the AI can learn to count, read medical reports, or identify birds, all while remembering how to be a helpful, general assistant.

In short: To teach a super-intelligent AI a new trick without making it forget its old ones, don't force it to relearn everything. Just show it how to use its existing tools in a slightly new way.

1. Problem Statement

Large Multimodal Models (LMMs) like LLaVA and Qwen-VL are powerful general-purpose agents but often lack proficiency in specific domains (e.g., medical imaging, precise counting, OCR). The standard approach to adapt them is sequential fine-tuning on new tasks. However, this typically leads to catastrophic forgetting, where learning a new skill degrades performance on previously learned tasks and, crucially, on broad, held-out general benchmarks.

The paper addresses two core questions:

How can we teach LMMs new skills without erasing their prior general abilities?
What is the underlying mechanism of forgetting in LMMs, and can it be mitigated without complex techniques like experience replay or model expansion?

2. Methodology

Experimental Setup

The authors constructed a rigorous evaluation suite involving:

Models: Three distinct LMM families: LLaVA-OneVision, LLaVA-NeXT (LLaMA-3 based), and Qwen2.5-VL.
Target Tasks: Five sequential skills spanning different formats:
1. Bird Classification (CUB200)
2. Counting (PixmoCount)
3. Medical VQA (PathVQA)
4. OCR Reading (TextVQA)
5. Time Reading (TimeClock)
Held-out Benchmarks: Eight diverse general vision-language benchmarks (e.g., AI2D, ChartQA, DocVQA, MMStar) to measure "general ability."
Protocol: Sequential fine-tuning without rehearsal (no mixing of old data).

Mechanistic Hypothesis

The authors hypothesize that "forgetting" is not necessarily a loss of internal knowledge (memory erasure) but rather a bias in the output token distribution.

Self-Attention (SA): Acts as a router, organizing context.
MLP (Feed-Forward): Acts as a key-value memory that writes features back into the residual stream, directly influencing the next-token distribution.
Hypothesis: Fine-tuning the MLP heavily biases the model toward the specific answer format of the current task (e.g., outputting numbers for counting), causing it to fail on general tasks that require diverse outputs.

Proposed Tuning Recipes

Based on the hypothesis, the authors propose two simple, parameter-selective tuning strategies:

SA Proj. (Self-Attention Projections): Update only the $W_Q, W_K, W_V, W_O$ matrices in the self-attention layers. This updates routing without directly altering the output distribution mechanism.
MLP (Gate & Up): Update only the $W_{gate}$ and $W_{up}$ matrices in the MLP, while freezing the $W_{down}$ projection. This allows the model to activate new concepts (feature detection) but prevents the "write-back" from aggressively shifting the final output logits.

Comparison Baselines

The study compares these recipes against:

Full-model fine-tuning.
Tuning specific components (Vision Encoder, Projector, Full LLM, Full MLP).
Standard Continual Learning (CL) methods: Learning without Forgetting (LwF), LoRA, Mixture-of-Experts (MoE), and Weight-space Interpolation (WiSE-FT).

3. Key Findings & Results

A. The "Recovery" Phenomenon

A surprising discovery is that forgetting is often reversible. When a model is fine-tuned on a second specialized task (e.g., PathVQA) after a first task (e.g., PixmoCount), performance on the held-out benchmarks often recovers partially or fully. This suggests the initial "forgetting" was a temporary distribution shift rather than permanent knowledge loss.

B. Output Distribution Shift as the Root Cause

Using a Counting-Bias Probe, the authors measured the likelihood of the model emitting numeric tokens on general captioning tasks.

Correlation: There is a strong negative correlation ( $\rho = -0.84$ ) between the rise in numeric token probability and the drop in held-out benchmark performance.
Mechanism: Tuning the full MLP causes a massive spike in numeric token bias during counting tasks. Tuning only SA Proj. or MLP (Gate & Up) keeps this bias near baseline levels.

C. Performance Comparison (LLaVA-OneVision Results)

The authors report $\Delta$ Learning (improvement on target) vs. $\Delta$ Held-out Forgetting (drop on general benchmarks):

Method	Target Learning	Held-out Forgetting	Overall Trade-off
Full LLM	+31.8	-23.3	High learning, severe forgetting
Full MLP	+31.1	-15.7	High learning, high forgetting
SA Proj.	+24.9	-0.6	Excellent balance
MLP (Gate & Up)	+30.5	-2.1	Best balance
Vision Encoder	+9.6	-10.8	Poor learning, high forgetting

SA Proj. achieves near-maximal learning with almost zero forgetting.
MLP (Gate & Up) achieves learning comparable to full MLP tuning but with drastically reduced forgetting (only -2.1 vs -15.7).
Both methods significantly outperform full-model tuning and standard CL baselines (LoRA, LwF, MoE) in the learning-stability trade-off.

D. Generalization

These results hold across LLaVA-NeXT and Qwen2.5-VL.

On Qwen2.5-VL, SA Proj. tuning actually resulted in positive transfer (improvement) on held-out tasks.
On LLaVA-NeXT, while forgetting was higher overall, the selective tuning methods still reduced forgetting by more than 50% compared to full tuning.

E. Efficiency

Selective tuning is computationally efficient.

SA Proj. processes 1.46 samples/sec (faster than LoRA at 1.27 samples/sec) because it avoids the overhead of adapter modules.
It requires no replay buffers, no auxiliary parameters, and no per-stage weight blending.

4. Significance and Contributions

Redefining Forgetting: The paper provides a mechanistic explanation that "catastrophic forgetting" in LMMs is largely a distributional shift (bias toward specific answer formats) rather than a loss of representational capacity.
Simple, Robust Recipes: It identifies that restricting updates to Self-Attention Projections or MLP Gate/Up layers is sufficient to learn new skills while preserving general capabilities. This is simpler and more effective than complex CL architectures.
Reversibility: It demonstrates that forgetting in sequential learning can be naturally recovered by subsequent training on diverse tasks, implying that "forgetting" is often just a temporary state of the output distribution.
Practical Impact: These findings offer a low-cost, high-efficiency strategy for practitioners to adapt expensive LMMs to new domains without the need for massive datasets, replay buffers, or retraining from scratch.

Conclusion

The paper concludes that the key to teaching LMMs new skills without forgetting lies in controlling the output distribution shift. By selectively tuning components that influence routing (SA Proj.) or feature activation without aggressive write-back (MLP Gate & Up), models can acquire new specialized skills while maintaining their broad, general-purpose intelligence.