💬 NLP

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

The paper proposes AWARe, a fine-tuning method that mitigates catastrophic forgetting in Multimodal Large Language Models by dynamically freezing parameters critical to prior knowledge based on activation patterns, thereby preserving upstream capabilities while achieving superior downstream performance without modifying model architectures.

Original authors: Juncheng Liao, Jinfan Lv, Guoming Wang, Jupeng Zheng, Ling Xiao, Siliang Tang

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Juncheng Liao, Jinfan Lv, Guoming Wang, Jupeng Zheng, Ling Xiao, Siliang Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot friend who has spent years reading every book in the library and looking at every painting in the world. This robot, a Multimodal Large Language Model (MLLM), is amazing at understanding both pictures and words. But here's the tricky part: if you try to teach this robot a new, specific skill—like how to solve a puzzle game or write a poem about a specific cat—it often forgets everything else it knew. It's like if you learned to play a new song on the piano so intensely that you suddenly forgot how to tie your shoes. This "forgetting" is called catastrophic forgetting, and it's a huge headache for scientists who want to keep these AI models useful and knowledgeable as they learn new things.

The paper you're about to read tackles this problem with a clever new method called AWARe. Think of the robot's brain as a massive city with billions of tiny workers (neurons) doing different jobs. When the robot learns something new, it usually tells all the workers to change their habits, which causes chaos and makes them forget their old jobs. AWARe suggests a smarter approach: instead of asking everyone to change, it listens to the workers to see who is currently shouting the loudest (activating the most) when the robot looks at its old, familiar tasks. It then tells those specific, busy workers, "You stay exactly the same! Keep doing your old job." Only the quieter, less busy workers are allowed to change their habits to learn the new task. This way, the robot learns the new skill without losing its old memories.

The Problem: The "Overwrite" Glitch

Multimodal Large Language Models are like super-powered students who have already graduated from a massive university of images and text. They can reason, explain, and chat about almost anything. But when we try to "fine-tune" them—teach them a new, specific job like identifying medical conditions or answering questions about a specific video game—they often suffer from catastrophic forgetting.

Imagine you are trying to learn a new dance move. If you practice it so hard that you forget how to walk, that's catastrophic forgetting. In the AI world, this happens because the process of learning the new task involves updating the model's internal "weights" (the knobs and dials that control how it thinks). When the model adjusts these knobs for the new task, it accidentally twists the ones that were holding up its old knowledge, causing the old skills to crumble.

The Solution: AWARe (Activation-Weighted Adaptive REtention)

The authors propose a method called AWARe to fix this. Instead of blindly changing the whole brain, AWARe acts like a smart librarian who knows exactly which books are most important to keep on the shelf.

Here is how it works, step-by-step:

The "Listening" Phase (Profiling): Before teaching the robot anything new, the researchers let it look at a small set of "calibration" examples (like a few old photos or questions it already knows well). They don't just look at the robot's brain structure; they watch how the brain lights up. They measure the activation—how much energy each tiny worker (neuron) uses when processing these familiar tasks.
The "Freeze" Decision: The researchers found that some neurons are super active and critical for the robot's old knowledge, while others are more relaxed. AWARe identifies the "loud" neurons (the top 30% that are most active) and says, "You are essential! Freeze." This means their settings are locked and cannot be changed.
The "Adapt" Phase: The remaining neurons (the quiet 70%) are left unlocked. When the robot learns the new task, only these unlocked neurons are allowed to change. The locked neurons keep the robot's old memories safe and sound.

Why This is a Big Deal

The paper shows that this method is a game-changer for two main reasons:

It's a "No-Brainer" Upgrade: Unlike other methods that require building new, complex add-ons to the robot's brain or storing huge libraries of old data (which takes up a lot of memory), AWARe works with the robot's existing brain. It doesn't change the architecture; it just changes which parts are allowed to move. It's like rearranging the furniture in a room without building new walls.
It Works Wonders: In their experiments, the team tested AWARe on models like LLaVA-v1.5 and Qwen2.5-VL.
- When teaching the model to answer questions about diagrams (IconQA), AWARe kept the model's old knowledge (like answering general questions about images) at 98.4% of its original strength, while still learning the new task very well.
- In a continuous learning test where the model had to learn five different tasks in a row (like medical questions, driving, and finance), AWARe outperformed all other methods, keeping the average score 3.18 points higher than the next best method.

What the Paper Says (and Doesn't Say)

The authors are very clear about what they found and what they didn't.

What Works: They proved that using activation (how much a neuron is working) is a better way to decide what to freeze than just looking at the weight's size or picking neurons at random. If you pick neurons randomly, the model forgets its old skills almost immediately. If you pick based on how "loud" they are, the model stays smart.
What Doesn't Work (or isn't needed): They showed that you don't need to freeze the entire brain. Freezing just the top 30% of the most active neurons in specific parts of the brain (the self-attention layers and the connector that links images to text) is enough. Freezing the "MLP" parts (the deep thinking layers) actually hurts performance, suggesting those layers are meant to be flexible.
The "Magic" Calibration Set: One of the coolest findings is that you don't need the robot's original training data to figure out which neurons to freeze. You can use a general, random set of questions (like the MMMU dataset) to do the profiling. Even with this generic data, AWARe works almost as well as if they had used the specific old data. This makes the method very practical for real-world use.
Efficiency: The method is incredibly efficient. In their best setup, they only updated about 17.5% of the total parameters. This means the computer doesn't have to work nearly as hard as it would if it had to update the whole model.

The Bottom Line

AWARe suggests that the secret to keeping an AI from forgetting isn't to stop it from learning, but to be selective about what it changes. By listening to which parts of the brain are working hardest on old tasks and protecting them, we can teach these models new tricks without losing their old wisdom. It's a simple, effective, and surprisingly gentle way to keep our digital friends smart, stable, and ready for whatever comes next.

Technical Summary: AWARe

Problem Statement

Multimodal Large Language Models (MLLMs) possess strong generalization and reasoning capabilities derived from large-scale pre-training. However, fine-tuning these models on downstream tasks often triggers catastrophic forgetting (CF), where the optimization for new objectives degrades previously acquired upstream capabilities. This issue is particularly severe in MLLMs due to the complex interplay between modalities and the high dimensionality of the parameter space. Existing mitigation strategies, such as experience replay or regularization-based methods, often incur prohibitive computational overhead or struggle to scale to billions of parameters. Furthermore, Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, while reducing trainable parameters, do not inherently prevent the erosion of pre-trained knowledge, as they can still disrupt reliable upstream features. The core challenge is enabling MLLMs to adapt to downstream tasks while robustly preserving pre-trained capabilities without incurring excessive costs or architectural modifications.

Methodology: AWARe

The authors propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning framework that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. The method is inspired by the principle that a small fraction of weights associated with large activation magnitudes are disproportionately important for preserving model behavior (a concept observed in Activation-aware Weight Quantization and pruning studies).

AWARe operates in two phases:

Knowledge Profiling:
- The method utilizes a small calibration set ( $S_{cal}$ ) sampled from upstream tasks. If upstream data is unavailable, a general-purpose benchmark (e.g., MMMU) serves as a proxy.
- For a target linear layer with weight matrix $W$ , the model performs a forward pass on the calibration set to obtain an activation tensor $A$ .
- Saliency Estimation: The method computes a saliency score for each output neuron through a three-step aggregation process:
  1. Compute the L2-norm of activations along the sequence length dimension.
  2. Apply per-sample L2-normalization across the hidden dimension to ensure comparability and prevent bias from outlier-scale activations.
  3. Average the normalized values across the batch to generate a final saliency vector $s$ .
- This process identifies neurons that consistently capture structural features across diverse upstream tasks.
Constrained Fine-tuning:
- Based on the saliency scores, a retention ratio $\rho$ (e.g., 30%) is used to identify the top- $\rho$ fraction of most salient output neurons globally across target layers.
- A binary gradient mask $M_{grad}$ is constructed where rows corresponding to high-saliency neurons are set to 0 (frozen), and others are set to 1 (trainable).
- During downstream optimization, parameters associated with salient regions are held constant via masked gradient updates: $W^{(t+1)} = W^{(t)} - \eta \cdot (\nabla_W L_{down} \odot M_{grad})$ .
- Target Selection: AWARe specifically applies this strategy to the linear projection layers within self-attention mechanisms ( $q\_proj, k\_proj, v\_proj$ ) and the multimodal projector ( $mm\_projector$ ). Other components, such as MLP blocks and the output projection, remain entirely frozen. This selection is based on findings that updating self-attention projections causes less forgetting of pre-existing knowledge compared to MLP blocks.

Key Contributions

AWARe Framework: The introduction of a simple, activation-based method for preserving critical upstream knowledge during downstream adaptation in MLLMs. It repurposes activation saliency from compression and merging contexts to forgetting mitigation.
Efficiency and Simplicity: The approach requires no architectural changes, introduces no extra adapter modules, and updates only a small subset of parameters (approximately 17.5% in the optimal setting). It avoids the need for replay buffers or second-order statistics.
Comprehensive Validation: The authors demonstrate through ablation studies and benchmarks that activation-based saliency is an effective criterion for parameter selection, outperforming static weight magnitude or random selection.

Experimental Results

The authors evaluated AWARe on the LLaVA-v1.5-7B backbone and validated its transferability to Qwen2.5-VL-7B.

Single-Task Downstream Adaptation: On IconQA (VQA) and COCO-Caption (Image Captioning), AWARe achieved a superior balance between Knowledge Retention (R) (stability of upstream tasks) and Learning Efficiency (E) (performance on the new task).
- Compared to state-of-the-art baselines (including LoRA, DoRA, Orth-Reg, Model Tailor, LoRASculpt, and SPIDER), AWARe maintained high retention rates (e.g., $R \approx 98.4$ on IconQA) while achieving competitive downstream performance.
- Notably, using a general-purpose calibration set (MMMU) instead of specific upstream data yielded results comparable to using the full upstream dataset, demonstrating robustness.
Continual Instruction Tuning: On the MLLM-DCL benchmark (a sequence of five tasks: Remote Sensing, Medical, Autonomous Driving, Science, Finance), AWARe achieved the highest average performance across all tasks and the best final performance after the last task. It improved overall averages by 3.18 and 1.77 points over the strongest baseline (DISCO) in average and last-task metrics, respectively.
Ablation Studies:
- Selection Strategy: Activation-based selection significantly outperformed random selection and weight-norm-based selection.
- Retention Ratio: A 30% retention ratio (freezing the top 30% of salient neurons) provided the optimal stability-plasticity trade-off.
- Training Targets: Applying AWARe to both self-attention projections and the multimodal projector yielded the best results. Freezing the MLP blocks entirely was crucial, as fine-tuning them led to severe performance collapse on upstream tasks.

Significance and Claims

The paper claims that AWARe offers a practical and effective solution to catastrophic forgetting in MLLMs by leveraging dynamic, data-dependent activation patterns rather than static weight priors. The significance of the work lies in:

Architecture Agnosticism: It operates without modifying model architectures, ensuring compatibility with existing inference engines.
Cost-Effectiveness: By freezing high-saliency parameters and updating only a small fraction, it significantly reduces computational overhead compared to full fine-tuning or replay-based methods.
Robustness: The method remains effective even when upstream task data is restricted, relying on general-purpose calibration sets.
Stability-Plasticity Trade-off: AWARe successfully balances the preservation of core general capabilities with the flexibility required for specialized task adaptation, addressing a critical bottleneck in the practical deployment of MLLMs.

The authors acknowledge limitations, noting that the quality of the calibration data can influence the selected frozen regions and that extremely large distribution shifts may require tuning the retention ratio. However, the proposed framework establishes a new baseline for efficient and stable multimodal fine-tuning.