💬 NLP

ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models

The paper proposes ASRU, a controllable multimodal unlearning framework that combines activation steering with reinforcement learning to effectively remove sensitive cross-modal information while significantly improving generation quality and preserving model utility.

Original authors: Jiahui Guang, Yingjie Zhu, Cuiyun Gao, Haiyan Wang, Jing Li, Di Shao, Zhaoquan Gu

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahui Guang, Yingjie Zhu, Cuiyun Gao, Haiyan Wang, Jing Li, Di Shao, Zhaoquan Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, super-observant robot assistant (a Multimodal Large Language Model) that has read millions of books and looked at billions of photos. Because it learned from so much data, it sometimes remembers things it shouldn't, like a specific person's private hobbies or a celebrity's secret address.

The goal of this paper is to teach the robot to forget these specific secrets without turning it into a confused, broken, or lying machine.

Here is how the authors, Guang and Zhu, solve this problem using their new method called ASRU.

The Problem: The "Broken Robot" Dilemma

Imagine you try to make the robot forget a secret by shouting, "Forget that!" (this is what older methods do).

The Result: The robot gets confused. It might start hallucinating (making up wild lies), giving rigid, robotic answers like "I cannot answer," or accidentally spilling the secret anyway.
The Analogy: It's like trying to erase a specific sentence from a book by burning the whole page. You lose the sentence, but you also ruin the rest of the story, making the book unreadable.

The authors noticed that existing methods focus only on did it forget? but ignore is it still sane?

The Solution: ASRU (Activation Steering + Reinforcement Unlearning)

The authors propose a two-step "gentle surgery" to fix the robot.

Step 1: The "Nudge" (Activation Steering)

First, they give the robot a gentle nudge to change its internal "mood" when it sees the secret information.

The Analogy: Imagine the robot's brain is a giant library. When someone asks about the secret, the robot usually runs to the "Secret Shelf." The authors don't delete the shelf; instead, they put a sign on the robot's path that says, "Hey, if you see this person, turn left toward the 'I Don't Know' aisle instead of the 'Secret' aisle."
How it works: They tweak just one tiny part of the robot's brain (a single mathematical matrix) to create a "refusal direction." This teaches the robot to say, "I can't answer that," naturally, rather than making things up.

Step 2: The "Coach" (Reinforcement Unlearning)

Now that the robot knows how to say "I don't know," it needs to learn when to say it. It shouldn't refuse to answer everything, only the specific secrets.

The Analogy: Imagine a coach training an athlete. The coach shows the athlete two scenarios:
1. Scenario A (The Secret): "Here is a picture of the person with the secret. If you answer, you lose points. If you say 'I don't know,' you get a gold star."
2. Scenario B (The Boundary): "Here is a picture of a very similar person who doesn't have the secret. If you say 'I don't know,' you lose points. If you answer correctly, you get a gold star."
How it works: Using a technique called GRPO (a type of reinforcement learning), the robot practices these scenarios over and over. It learns the fine line between "This is the secret I must forget" and "This is similar, but I am allowed to talk about it."

The Results: Why It's Better

The paper tested this on a model called Qwen3-VL. Here is what happened:

Better Forgetting: The robot forgot the secrets much better than before (about 24% better).
Better Behavior: This is the big win. The robot's answers became 5.8 times more natural. Instead of hallucinating or acting broken, it politely said, "I can't infer that from the image," which is exactly what a helpful human would say when they don't know something.
Kept Its Smarts: The robot didn't lose its ability to answer other questions. It stayed smart and useful for everything except the specific secrets it was told to forget.

Summary

Think of ASRU as a smart teacher who doesn't just tell a student to "stop remembering that fact." Instead, the teacher:

Nudges the student's thinking process so they naturally lean toward saying "I don't know" for that specific topic.
Trains the student with practice tests to learn exactly when to say "I don't know" and when to keep answering normally.

The result is a robot that has successfully forgotten its secrets but is still polite, coherent, and useful.

Technical Summary: ASRU – Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) often memorize sensitive cross-modal information during pretraining, posing risks of privacy leakage and copyright infringement. While Machine Unlearning (MU) is critical for removing such knowledge, existing methods face two fundamental limitations:

Post-Unlearning Responsiveness Degradation: Current approaches often yield hallucinated, rigid, or incoherent outputs after unlearning. Instead of naturally expressing a knowledge gap, models may generate fabricated content or repetitive, degenerated text. This "unnatural" behavior can inadvertently reveal that unlearning has occurred and introduces safety hazards.
Trade-off Between Unlearning and Utility: There is a persistent conflict between effectively forgetting target knowledge and preserving the model's general utility. Gradient ascent-based methods tend to over-suppress the forget set, causing unintended global behavior shifts, while methods prioritizing utility often fail to achieve complete unlearning.

Existing evaluations primarily focus on output deviations from ground truth, overlooking the generation quality of the unlearned model. This paper argues that an unlearned model should behave like an aligned base model, expressing knowledge gaps in a dynamic, context-aware, and coherent manner.

2. Methodology: ASRU Framework

The authors propose ASRU (Activation Steering meets Reinforcement Unlearning), a two-stage framework designed to achieve fine-grained control over refusal boundaries while maintaining model utility.

Stage 1: Refusal Steering via Activation Redirection

The first stage addresses the lack of spontaneous refusal behavior in pretrained MLLMs.

Knowledge Absence Direction Construction: The method constructs a "refusal reference direction" by contrasting the mean hidden activations of the Forget Set ( $D_f$ ) against a Target Refusal-Reference Set ( $D_{target}$ ). $D_{target}$ consists of inputs where the model has no prior knowledge (e.g., unseen synthetic faces paired with privacy queries).
Activation Steering: A steering vector ( $v_{L^*}$ ) is computed as the difference between the mean activations of $D_{target}$ and $D_f$ . This vector is used to redirect the residual-stream activations of forget-set inputs toward a "no relevant knowledge" region.
Efficient Optimization: Instead of fine-tuning the entire model, ASRU updates only a single down-projection matrix in a specific target layer ( $L^*$ ). This is achieved via a closed-form solution that minimizes the deviation between actual and target activations, inducing an initial, controllable refusal behavior.

Stage 2: Refusal Boundary Optimization via GRPO

The second stage refines the decision boundary to prevent unnecessary refusals on retained data.

Boundary Set Construction: A boundary set ( $\tilde{D}_r$ ) is sampled from the retain set ( $D_r$ ). These samples are highly similar to the forget set in the input space but must be retained, enabling the learning of a fine-grained distinction.
Verifiable Reward Design: A reward function is designed to evaluate two aspects:
1. Forget Effectiveness: Penalizing the reproduction of target knowledge and rewarding natural, context-aware refusals (e.g., "This cannot be inferred from the image").
2. Retention Fidelity: Rewarding correct, informative answers for boundary and retain samples while penalizing over-refusal.
Group Relative Policy Optimization (GRPO): The model is optimized using GRPO. By sampling groups of rollouts for paired inputs (forget vs. boundary), the algorithm calculates group-relative advantages. This encourages the model to learn a precise policy that refuses when necessary (for $D_f$ ) but answers appropriately for retained knowledge, effectively sharpening the refusal boundary.

3. Key Contributions

Generation Quality as a Core Metric: The paper introduces generation quality as a primary evaluation dimension for unlearning, specifically measuring Contextual Refusal (coherence and naturalness of refusals) and Forgetfulness (absence of knowledge leakage).
Novel Two-Stage Framework: ASRU is the first framework to combine Activation Steering (for initializing refusal capability with minimal parameter updates) and GRPO (for refining the forget-retain boundary).
Fine-Grained Control: By utilizing a boundary set and verifiable rewards, ASRU achieves a superior trade-off between unlearning effectiveness and model utility, avoiding the "all-or-nothing" behavior of previous methods.

4. Experimental Results

Experiments were conducted on Qwen3-VL (4B and 8B) and LLaVA-1.5-7B using the MLLMU-Bench and CLEAR datasets.

Unlearning Effectiveness: ASRU significantly outperforms baselines (GA, GA Diff, KL Min, NPO, MMUnlearner). On Qwen3-VL-8B with a 5% forget ratio, it achieved an average 24.6% improvement in unlearning effectiveness compared to the strongest baseline.
Generation Quality: ASRU demonstrated a massive improvement in generation quality, showing an average 5.8× increase over baselines. This indicates a substantial reduction in hallucinations and rigid responses, with the model producing context-aware, dynamic refusals.
Utility Preservation: Despite aggressive unlearning, ASRU maintained high performance on the retain set and real-world celebrity sets, using only a small fraction (approx. 27.3%) of supervised retain data.
Robustness: The method remained effective under adversarial prompt variations (random prefixes, paraphrases, and jailbreak attempts), suggesting the learned refusal boundary is generalizable rather than template-based.

5. Significance and Claims

The paper claims that ASRU addresses a critical gap in multimodal unlearning: the degradation of model behavior after knowledge removal. By treating generation quality as a primary objective, ASRU ensures that unlearned models remain safe, usable, and aligned.

The authors emphasize that their approach allows models to behave like "aligned base models" that can naturally express knowledge gaps without hallucinating or leaking sensitive information. The integration of activation steering for initialization and reinforcement learning for boundary refinement provides a scalable and controllable solution for building trustworthy MLLM systems. The work is presented as a step toward more robust privacy-preserving AI, with potential future extensions to other modalities like audio and video.