🤖 AI

Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models

This paper introduces attribute-level unlearning as a finer-grained alternative to profile-level deletion for Multimodal Large Language Models (MLLMs), proposes a comprehensive benchmark to evaluate the challenge of selectively forgetting specific attributes while preserving identity-related information, and presents CLRP, a lightweight, training-free framework that effectively achieves this selective forgetting through causal localization and retain-aware projection.

Original authors: Junkai Lin, Junkai Chen, Siqi Hou, Yuhao He, Ruiqi Liu, Chenhan Jin, Shengze Xu, Tieyong Zeng

Published 2026-08-04

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Junkai Lin, Junkai Chen, Siqi Hou, Yuhao He, Ruiqi Liu, Chenhan Jin, Shengze Xu, Tieyong Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can look at a picture and tell you a story about it. This robot has read millions of books and seen billions of photos, so it knows a lot. But sometimes, it remembers things it shouldn't, like a person's secret birthday or their home address. In the world of artificial intelligence, there's a growing need to teach these robots to "forget" specific secrets without making them forget everything else. This process is called "machine unlearning." Think of it like editing a diary: you want to cross out one embarrassing sentence about your summer vacation but keep all the other fun stories about your friends and family intact. If you just tear out the whole page, you lose everything. If you try to erase the ink with a chemical that dissolves the paper, you ruin the whole book. The challenge is finding a way to surgically remove just the bad memory while keeping the rest of the robot's brain working perfectly.

This paper tackles a tricky new version of that problem. Instead of asking the robot to forget an entire person (like "forget everything about my friend Bob"), the researchers asked it to forget just one specific thing about Bob, like his favorite color, while still remembering his name, his job, and what he looks like. They found that the robot's brain is messy; the memory of Bob's favorite color is tangled up with the memory of Bob's face. Trying to cut out the color often accidentally cuts out the face, too. To solve this, the authors built a new test called AMU-Bench to measure how well robots can do this fine-tuned forgetting. They then invented a clever, training-free tool called CLRP (Causal Localization and Retain-Aware Projection). Think of CLRP as a pair of "memory glasses" that the robot wears only when answering questions. These glasses don't change the robot's brain; they just tweak the way the robot looks at its own thoughts in that exact moment. By using a technique called "activation patching" (which is like swapping a specific Lego brick in a tower to see which part of the tower wobbles), CLRP finds the exact spot in the robot's brain where the secret is stored. Then, it applies a mathematical filter that mutes that specific secret while leaving the rest of the story loud and clear.

The researchers tested this on three different robot brains (LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-3B) and found that their method works much better than previous attempts. In their main tests, where they asked the robots to forget 5% of the data, CLRP was able to drop the robot's ability to guess the secret from 20.00% down to just 4.00%. Even better, while other methods made the robot forget the good stuff too, CLRP actually helped the robot remember the non-secret facts better, boosting its accuracy on those questions from 14.77% to 17.79%. The paper suggests that this approach is a promising way to give robots the ability to be selective about what they keep and what they let go, without needing to retrain them from scratch. They also discovered that forgetting numbers (like a phone number) is easier for robots than forgetting long descriptions (like a hobby), but keeping the rest of the person's story safe is the hardest part of all.

Technical Summary: Toward Fine-Grained Forgetting: Attribute Unlearning for Multimodal Large Language Models

Problem Definition

Multimodal Large Language Models (MLLMs) possess strong vision–language capabilities but risk memorizing and disclosing sensitive information. While existing machine unlearning research primarily focuses on profile-level deletion (removing all information about a specific entity, identity, or character), practical privacy requests are often finer-grained. Users may wish to delete a specific attribute (e.g., a birth date, address, or organizational affiliation) while retaining non-sensitive information about the same identity.

Current profile-level unlearning benchmarks (e.g., MLLMU-Bench, FIUBench) do not explicitly test the preservation of individual non-target attributes within a selected entity. This creates a significant gap: existing methods often fail to achieve attribute-level selectivity. Because target attributes and retained attributes share the same identity and visual evidence, they often reside in nearby regions of the representation space and traverse overlapping causal pathways. Consequently, interventions broad enough to remove the target attribute frequently cause collateral degradation to non-target attributes of the same identity, while overly conservative interventions fail to suppress the target.

Methodology: Causal Localization and Retain-Aware Projection (CLRP)

To address the challenge of isolating specific attribute directions while preserving shared identity representations, the authors propose CLRP, a lightweight, training-free framework. CLRP operates without updating model parameters and consists of two primary components:

Causal Layer Localization:
- The method uses activation patching to identify the specific transformer layer that causally mediates the disclosure of the target attribute.
- For each target example, a target query (seeking the sensitive attribute) is paired with a same-identity retain query (seeking non-target information).
- The system caches activations from the target run and patches them into the retain run. The layer yielding the maximum causal patching effect—where the target attribute's evidence is most strongly restored—is selected as the intervention point ( $\ell^*$ ).
Retain-Aware Contrastive Projection:
- Once the intervention layer is identified, CLRP estimates a low-dimensional subspace representing the target attribute.
- Crucially, this estimation is retain-aware. It constructs statistics from both target-conditioned representations and same-identity retain representations (plus neutral prompts).
- The method computes a projection subspace that captures variation prominent in the target but suppressed in the retain representations. This is achieved by whitening the target covariance matrix relative to the retain covariance matrix and selecting the leading eigenvectors.
- A projection operator attenuates this specific subspace at the selected layer during inference, effectively removing the target attribute direction while preserving the shared identity representation required for non-target facts.

Key Contributions

The paper makes three primary contributions:

Definition of Attribute-Level Unlearning: The authors formally define attribute-level MLLM unlearning as a setting where designated attributes must become inaccessible while non-target facts of the same identity remain available. This imposes stricter locality requirements than profile-level deletion.
AMU-Bench Construction: The authors introduce the Attribute-level Multimodal Unlearning Benchmark (AMU-Bench). Unlike previous benchmarks, it partitions deletion targets into long-text (descriptive), numeric (dates, salaries), and short-text (categorical) categories. It systematically evaluates performance across multiple forget ratios (5%, 10%, 15%) and distinct question types (cloze, classification, generation) on widely used MLLMs (LLaVA-1.5, Qwen2.5-VL).
CLRP Framework: The proposal of a training-free method that combines causal localization with retain-aware subspace projection. The method is designed to navigate the entanglement between target and retained attributes.

Experimental Results

The authors evaluated CLRP against four baselines (Gradient Ascent, Gradient Difference, KL Minimization, and Preference Optimization) on LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-3B.

Forgetting Difficulty: The evaluation revealed that attribute type significantly impacts forgetting difficulty. Numeric attributes were consistently the easiest to suppress (often reaching near-zero accuracy), whereas long-text and short-text attributes retained substantially more residual information across all baseline methods.
Performance Gains: CLRP demonstrated superior performance in balancing forgetting and retention.
- In the 5% forget ratio results for LLaVA-1.5-7B and Qwen2.5-VL-3B, CLRP reduced Attribute Forget Set cloze accuracy by up to 16.00 percentage points (e.g., from 20.00% to 4.00%) compared to the strongest baseline.
- Simultaneously, it improved Attribute Retain Set cloze accuracy by up to 3.02 percentage points (e.g., from 14.77% to 17.79%), demonstrating better preservation of same-profile non-target facts.
Trade-off Analysis: Existing methods exhibited unstable trade-offs, often degrading the retention of same-profile attributes when attempting to forget targets. CLRP achieved the best or tied-best results in 8 out of 9 model–attribute conditions for Attribute Forget Set accuracy and showed improved retention scores, indicating that stronger target forgetting did not come at the cost of whole-profile degradation.

Significance and Claims

The paper claims that AMU-Bench exposes limitations in current profile-level unlearning approaches, specifically the difficulty of preserving same-profile knowledge while removing specific attributes. The authors assert that CLRP provides an effective solution for this fine-grained task by leveraging causal localization to pinpoint intervention layers and retain-aware projection to distinguish between target-specific and shared identity representations.

The significance of this work lies in its shift from coarse-grained entity deletion to fine-grained attribute unlearning, a setting that better aligns with practical privacy requests. The results suggest that selective forgetting is feasible in MLLMs without retraining, provided the intervention is causally localized and constrained by retain-aware geometry. The authors conclude that their approach establishes state-of-the-art performance in balancing the suppression of designated attributes with the preservation of non-target information within the same profile.