💻 computer science

Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models

This paper proposes HFRU, a reinforcement unlearning framework that operates on the vision encoder with a GRPO-based optimization and abstraction reward to achieve deep semantic forgetting of sensitive knowledge in vision-language models while effectively preventing object hallucinations.

Original authors: Kaidi Jia, Yujie Lin, Chengyi Yang, Jiayao Ma, Jinsong Su

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaidi Jia, Yujie Lin, Chengyi Yang, Jiayao Ma, Jinsong Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Vision-Language Model (VLM) as a super-smart, multilingual librarian who has read every book and looked at every picture on the internet. This librarian is incredibly helpful, but sometimes they remember things they shouldn't—like a specific celebrity's face, a private photo, or a copyrighted image.

When we ask the librarian to "forget" these specific things, current methods often act like a clumsy editor. They just cross out the names in the librarian's notebook (the text output) but leave the actual mental image of that person or object intact in the librarian's brain. If you ask a tricky question like, "Is this person in the photo?" the librarian might still recognize them, even if they promised not to say their name. Worse, when forced to forget, the librarian might start making things up, confidently describing a cat when the photo actually shows a dog, just to fill the silence.

The paper introduces a new method called HFRU (Hallucination-Free Reinforcement Unlearning) to fix this. Here is how it works, using simple analogies:

1. The Problem: "Superficial Amnesia"

Most current methods try to make the librarian forget by only editing their speech (the language decoder).

The Analogy: Imagine telling a student, "Don't say the word 'apple'." The student stops saying "apple," but they still have a perfect mental picture of an apple in their head. If you show them a picture and ask, "What fruit is this?" they might still know it's an apple, or they might panic and guess "banana" just to avoid saying "apple." This is called hallucination—making up facts that aren't there.

2. The Solution: Rewiring the "Eyes"

HFRU takes a different approach. Instead of just editing the speech, it goes directly to the part of the librarian's brain that sees and recognizes images (the vision encoder).

The Analogy: Instead of just telling the student not to say "apple," HFRU gently blurs the mental image of the apple in the student's mind. It reshapes how the brain processes that specific visual pattern so it no longer looks like an apple.

3. The Two-Step Process

HFRU uses a two-stage training process to do this safely:

Stage 1: The "Confusion" Phase (Cold Start)
The system shows the librarian pictures of the things to be forgotten (e.g., a specific dog) but tells them to describe them as something else (e.g., "This is a rabbit").
- Goal: This breaks the strong link between the image and its original name. It's like teaching the librarian that when they see this specific dog, they should think "rabbit" instead. This weakens the memory before the real work begins.
Stage 2: The "Smart Reward" Phase (Reinforcement Learning)
The system uses a game-like scoring system (called GRPO) to train the librarian.
- The Penalty: If the librarian mentions the forbidden name (e.g., "dog"), they lose points.
- The Abstraction Reward (The Secret Sauce): This is the paper's big innovation. If the librarian is forced to forget "dog," they might be tempted to guess "cat" (hallucination). HFRU gives a bonus if the librarian uses a safe, general word like "animal" instead.
- The Analogy: Imagine a teacher telling a student, "Don't say 'dog'." If the student says "cat," they get a bad grade (hallucination). But if they say "animal," they get a gold star. This teaches the student to be vague and safe rather than making up a wrong specific answer.

4. The Results

The paper tested this on two types of tasks: recognizing objects (like dogs and elephants) and recognizing faces (like specific famous people).

Forgetting: HFRU successfully made the librarian forget the specific targets about 98-99% of the time.
Remembering: It kept the librarian's ability to recognize everything else (like cats or other people) almost perfectly.
No Hallucinations: Crucially, unlike other methods that made the librarian guess wildly, HFRU almost never made up fake objects. The librarian would either say "I don't know" or use a safe general word like "animal."

Summary

Think of HFRU as a specialized memory surgery. Instead of just putting a gag on the librarian to stop them from speaking certain words, it carefully rewires their visual recognition center. It teaches them to replace specific, sensitive memories with safe, general concepts, ensuring they don't accidentally make things up when they try to forget. The result is a model that truly forgets what it needs to, without losing its mind or making up lies.

Technical Summary: Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models

Problem Definition

Vision-Language Models (VLMs) pose significant risks regarding privacy, copyright, and bias, necessitating machine unlearning to remove specific sensitive knowledge. However, existing unlearning methods primarily rely on fine-tuning the language decoder. The authors identify a critical limitation in this approach: it leads to superficial forgetting. While the model may suppress the generation of specific keywords in open-ended text, it retains the underlying visual representations of the target concept within the vision encoder. Consequently, the model can still recognize or reason about the forgotten concept when prompted discriminatively (e.g., "Is this person X?").

Furthermore, the paper highlights a previously overlooked failure mode: object hallucination. When a model is forced to suppress a concept without a semantic fallback, it may arbitrarily redistribute probability mass to unrelated objects not grounded in the visual input. The paper formally defines the goal of Object Hallucination-Free VLM Unlearning, which requires:

Effective Forgetting: Eliminating the influence of the target concept across both generative and discriminative tasks.
Utility Preservation: Maintaining performance on non-target (retain) data and general benchmarks.
Hallucination-Free Generation: Ensuring that when the model cannot identify a target concept, it does not confidently assert the presence of unrelated objects. Instead, it should either express uncertainty or use semantically valid hypernyms (e.g., "an animal" instead of a specific "dog" or a random "cat").

Methodology: HFRU Framework

The authors propose HFRU (Hallucination-Free Reinforcement Unlearning), a two-stage framework that operates directly on the vision encoder rather than the language decoder. This ensures deep semantic removal at the representation level.

Stage 1: Cold-Start Alignment Disruption

The first stage employs Supervised Fine-Tuning (SFT) to disrupt the alignment between visual features in the forget set and their original semantic labels.

Process: The model generates a caption for an image in the forget set. Keywords related to the target concept are randomly replaced with tokens from the retain set (e.g., replacing "dog" with "rabbit").
Objective: The vision encoder is trained to associate the visual input with the modified, mismatched text. This weakens the model's reliance on the target concept and initializes the encoder in a state where the original association is significantly disrupted.

Stage 2: Encoder-Only Reinforcement Unlearning

The second stage utilizes Group Relative Policy Optimization (GRPO) to refine the vision encoder parameters while keeping the language decoder frozen.

Composite Reward Structure: The core innovation is a reward function designed to penalize target concepts while encouraging semantically valid abstractions.
- Penalty Reward ( $R_{pen}$ ): Penalizes the generation of target keywords and their synonyms.
- Abstraction Reward ( $R_{abs}$ ): Rewards the generation of hypernyms (e.g., "animal" for "dog") instead of unrelated objects. This acts as a "safe semantic fallback," guiding the model to substitute forgotten concepts with valid higher-level categories.
- Retain Reward ( $R_{retain}$ ): Ensures the model retains knowledge of non-target concepts.
Optimization: GRPO computes relative advantages within a group of sampled responses to stabilize training. The objective function maximizes the reward for the forget set (penalizing targets, rewarding hypernyms) and the retain set (rewarding preservation), effectively reshaping the latent representation space.

Theoretical Guarantee

The paper provides a theoretical proof (Lemma 1) demonstrating that the inclusion of the abstraction reward ( $R_{abs}$ ) systematically reduces the expected hallucination rate compared to using a penalty-only reward. By providing a valid semantic path (hypernyms) for the probability mass removed from the target concept, the model is less likely to generate ungrounded, hallucinated objects.

Key Contributions

Problem Definition: The authors formally define the novel problem of object hallucination-free VLM unlearning, addressing the failure mode where concept removal triggers ungrounded content generation.
HFRU Framework: They propose a two-stage framework targeting the vision encoder. This ensures deep semantic removal rather than superficial lexical filtering. Theoretically, this mechanism reduces hallucinations on the forget set.
Novel Visualization: The authors develop a method to reconstruct internal perception from hidden states using a mapping network and a diffusion-based IP-Adapter, allowing for the direct observation of how visual features change during unlearning.
Comprehensive Evaluation: Extensive experiments on object recognition (PACS) and face identity (VGGFace2) tasks demonstrate the framework's efficacy.

Experimental Results

The authors evaluated HFRU on the PACS and VGGFace2 datasets, comparing it against baselines including Gradient Ascent (GA), Gradient Difference (GD), NPO, RMU, SAUCE, and SLUG.

Forgetting and Retention: HFRU achieved an average forgetting rate and retention performance exceeding 98% across various prompt formats (Original, Paraphrased, and Discriminative).
Hallucination Mitigation: Unlike baselines that often trade forgetting for hallucination (e.g., SAUCE showed high forgetting but a 20.67% hallucination rate on VGGFace2), HFRU introduced negligible object hallucination (0.25% on PACS, 1.34% on VGGFace2).
General Utility: HFRU maintained general vision-language capabilities comparable to the base model, whereas many baselines suffered significant degradation on general benchmarks (MMStar, OCRBench, MMMU, RealWorldQA).
Ablation Studies: Removing the abstraction reward led to a massive spike in hallucinations (up to 99.67% on VGGFace2), confirming its critical role. Restricting updates to the vision encoder was shown to be superior to full-parameter fine-tuning, which often resulted in superficial token-level suppression.

Significance and Claims

The paper claims that HFRU establishes a new state-of-the-art for VLM unlearning by solving the dual challenge of effective concept removal and hallucination prevention. The significance lies in:

Shift in Focus: Moving unlearning from the language decoder to the vision encoder to achieve true semantic erasure.
Safety and Reliability: Providing a mechanism that prevents models from "hallucinating" new, potentially harmful or incorrect facts when forced to forget specific data.
Robustness: Demonstrating that the method generalizes to out-of-distribution (OOD) settings (e.g., the Sketch domain in PACS) without overfitting to the training distribution.

The authors conclude that their work highlights the necessity of encoder-level interventions and structured reward design for reliable multimodal unlearning, offering a practical approach for safer, more trustworthy AI systems.