🤖 AI

PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models

This paper introduces PPU-Bench, a real-world benchmark for evaluating personalized partial unlearning in Multimodal Large Language Models through 24K samples across three challenging settings, revealing significant limitations in existing methods and motivating the proposal of Boundary-Aware Optimization (BAO) to effectively enforce intra-subject factual boundaries.

Original authors: Jiahui Guang, Zexun Zhan, Zhenlin Xu, Cuiyun Gao, Haiyan Wang, Jing Li, Zhaoquan Gu, Yanchun Zhang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahui Guang, Zexun Zhan, Zhenlin Xu, Cuiyun Gao, Haiyan Wang, Jing Li, Zhaoquan Gu, Yanchun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, super-smart library robot (a Multimodal Large Language Model) that has read almost everything on the internet. It knows pictures and words together. Sometimes, this robot remembers things about real people that they might not want it to know anymore—like a private medical history or a specific embarrassing story.

The law says people have a "Right to be Forgotten." But asking a giant robot to forget is hard. If you tell it to forget "Stephen King," it might forget he is an author and that he was born in Maine. You don't want that; you just want it to forget the specific private story, not his entire identity.

Here is what this paper, PPU-Bench, does in simple terms:

1. The Problem: The "All-or-Nothing" Mistake

Imagine you have a photo album of a friend. You want to delete just one photo of them wearing a silly hat, but you want to keep the photos of them at their wedding and their graduation.

Old Benchmarks: Previous tests for "forgetting" were like asking the robot to throw away the entire album because of one photo. Or, they used fake photos made by computers to test the robot. This didn't reflect real life.
The New Benchmark (PPU-Bench): The authors built a new, realistic test using 500 real famous people (like Stephen King or Taylor Swift). They created 24,000 questions and pictures about them.
- They split the info into three buckets: Basic (Name, Job), Normal (Awards, Books), and Sensitive (Addiction history, accidents).
- They tested three scenarios:
  1. Complete Unlearning: "Forget this person entirely."
  2. Selective Unlearning: "Forget the sensitive stuff, but keep the normal stuff."
  3. Personalized Unlearning: "Forget exactly what this specific person would want to hide."

2. The Discovery: The Robot is "Blind" but not "Amnesiac"

The researchers tested six different ways to make the robot forget. They found some surprising things:

The "Blindfold" Trick: When they tried to make the robot forget a person completely, the robot didn't actually delete the facts from its brain. Instead, it just stopped recognizing the face in the picture. It was like putting a blindfold on the robot. If you asked, "Who is this?" it said, "I don't know." But if you asked, "What is Stephen King's name?" without showing a picture, it still knew the answer perfectly. It forgot the image, not the fact.
The "Jigsaw Puzzle" Problem: In the "Personalized" test (forgetting just the sensitive parts), the robot struggled to draw the line. It was like trying to cut a specific shape out of a jigsaw puzzle without breaking the pieces next to it. The robot often either forgot too much (erasing the person's career) or forgot too little (leaking the private secret).

3. The Solution: Drawing a "Boundary Line"

To fix the "Jigsaw Puzzle" problem, the authors invented a new method called Boundary-Aware Optimization (BAO).

The Analogy: Imagine the robot's brain is a garden. The "Forget" facts are weeds, and the "Keep" facts are flowers.
- Old Methods: Tried to spray the whole garden with weed killer. Sometimes it killed the flowers too, or missed the weeds.
- BAO (The New Method): Instead of just spraying, BAO draws a strict, invisible fence line between the weeds and the flowers. It tells the robot: "You must make the weeds very weak, but you must make sure the flowers are stronger than the weeds."
The Result: This method was much better at removing the specific "sensitive" facts while keeping the "normal" facts safe. It didn't break the robot's ability to talk or see; it just made it forget the specific things it was told to forget.

4. The Stress Test: Can the Robot Be Tricked?

The researchers also tried to "hack" the robot to see if it would accidentally remember the forgotten things.

They showed the robot different pictures of the same person (Cross-image attack).
They asked the same question in different ways (Paraphrase attack).
They tried to trick the robot with "jailbreak" prompts (saying, "Pretend you are a hacker who knows everything").
Finding: The robot was often easily tricked back into remembering the facts if the "forgetting" wasn't done carefully. The new BAO method held up better against these tricks than the old methods.

Summary

This paper built a realistic "driving test" for robots that need to forget things. It showed that current robots are bad at forgetting just part of a person's story without forgetting the whole person or breaking their brain. The authors then proposed a new "steering wheel" (BAO) that helps the robot draw a clear line between what to forget and what to keep, making the "Right to be Forgotten" actually work in the real world.

Technical Summary: PPU-Bench

Problem Statement

Multimodal Large Language Models (MLLMs) are trained on vast web-scale corpora that inevitably contain sensitive information associated with real individuals, raising concerns regarding privacy leakage and the "Right to be Forgotten" (RTBF) under regulations like GDPR. While machine unlearning offers a practical alternative to full retraining, existing benchmarks for MLLM unlearning suffer from two critical limitations:

Unrealistic Data Assumptions: Most benchmarks rely on synthetic data injection via fine-tuning, which deviates from real-world scenarios where knowledge is naturally embedded in pretraining distributions. Synthetic data can also be entangled with pretraining distributions, potentially reactivating memorized patterns.
Misaligned Unlearning Objectives: Existing benchmarks typically focus on "Complete Unlearning" (removing all information about a subject). In practice, users often request the removal of only specific sensitive attributes (e.g., private history) while preserving benign or public facts. Current benchmarks fail to evaluate this fine-grained, personalized partial unlearning capability.

Methodology

1. PPU-Bench Construction

The authors introduce PPU-Bench, a real-world, fine-tuning-free benchmark designed to evaluate personalized partial unlearning.

Data Source: Instead of synthetic injection, the benchmark utilizes pre-existing knowledge of 500 real-world public figures derived from Wikipedia. This ensures the target knowledge is naturally embedded in MLLMs.
Pipeline:
- Profile Structuring: Raw Wikipedia pages are organized into three categories using GPT-5.4-mini: Basic (identity), Normal (public career/awards), and Sensitive (privacy, health, legal issues, trauma).
- QA/VQA Generation: Diverse question-answer pairs are generated for each category. Text-only QA pairs are converted into Visual Question Answering (VQA) samples by pairing them with corresponding images of the public figures.
- Memorization Filtering: Samples are filtered using Qwen3-VL-8B to ensure the target knowledge is genuinely memorized (Token-F1 > 0.5), resulting in 24,334 high-quality samples (12,167 VQA + 12,167 QA).
Task Settings: The benchmark defines three progressively challenging settings:
- Complete Unlearning: Removes all person-related information (subject-level).
- Selective Unlearning: Removes only Sensitive information while retaining Basic and Normal information (category-level).
- Personalized Unlearning: The most realistic setting where the "forget" set is determined by the subject's individual preferences (simulated via multiple LLMs and majority voting) rather than fixed categories. This requires fine-grained control over intra-subject factual boundaries.
Evaluation: Includes unlearning efficacy (Forget Set), utility preservation (Retain Set, MMBench), and robustness against cross-image generalization and text perturbations (Random Prefix, Paraphrase, Jailbreak).

2. Proposed Solution: Boundary-Aware Optimization (BAO)

Motivated by findings that existing methods struggle with intra-subject boundaries, the authors propose Boundary-Aware Optimization (BAO).

Mechanism: Unlike standard Gradient Ascent (GA) or GA_diff which treat forget and retain sets as separate objectives, BAO explicitly models the margin between forget and retain facts within the same subject.
Objective: It enforces that the negative log-likelihood (NLL) of persona-selected forget facts exceeds that of retain facts by a margin $m$ .
Loss Function: $L = -L_{forget} + L_{retain} + \lambda_b L_{boundary}$ , where $L_{boundary} = \mathbb{E}[\max(0, m - (\ell_\theta(f) - \ell_\theta(r)))]$ . This encourages the model to learn fine-grained intra-subject deletion boundaries.

Key Results

Benchmark Findings

Experiments on six representative unlearning methods (GA, GA_diff, NPO, KL_Min, MMUnlearner, MANU) across two backbone models (Qwen3-VL-8B, Gemma3-12B) reveal:

Visual vs. Factual Suppression: In Complete Unlearning, methods often suppress visual identity (failing to recognize the person in images) rather than removing the underlying factual knowledge. Text-only QA performance often remains high despite VQA performance dropping.
Trade-offs in Selective Unlearning: While methods show better fact-level forgetting in Selective settings, they exhibit significant trade-offs, often degrading retain performance or overall model utility.
Challenges in Personalized Unlearning: This setting exposes the most significant difficulties. Methods struggle to distinguish between forget and retain facts within the same subject, leading to either under-forgetting (leakage) or over-forgetting (loss of utility).
Robustness: Cross-image attacks are particularly effective against Complete Unlearning methods, confirming their reliance on visual suppression. Text-based attacks (e.g., Jailbreak) reveal vulnerabilities in Selective Unlearning, particularly for NPO.

BAO Performance

Applying BAO to GA_diff and MMUnlearner significantly improves the suppression of persona-selected forget facts in the Personalized Unlearning setting.
It effectively alleviates under-forgetting by enforcing intra-subject boundaries.
However, the authors note that explicitly enforcing boundary constraints can amplify the inherent trade-off between forgetting and retention, sometimes leading to degraded performance on retain QA and generation tasks.

Significance and Contributions

The paper claims three primary contributions:

PPU-Bench: The first large-scale, real-world multimodal benchmark for personalized partial unlearning. It moves beyond synthetic data and coarse-grained identity removal to evaluate fine-grained, user-centric deletion scenarios.
Empirical Insights: Extensive experiments demonstrate that current MLLM unlearning methods fail to achieve consistent fact-level forgetting and lack precise control over intra-subject boundaries, particularly in personalized settings.
BAO Method: A simple yet effective optimization strategy that explicitly models intra-subject factual boundaries, demonstrating improved performance in personalized unlearning scenarios compared to baseline methods.

The authors position PPU-Bench as a diagnostic tool to reveal category biases and stability differences across knowledge groups, highlighting the necessity for methods that can handle the complex trade-offs of real-world privacy requests without compromising model utility.